Skip to content

docs: one pitch across every surface, and a shorter first screen - #173

Open
uipreliga wants to merge 1 commit into
mainfrom
docs/conversion-pitch-parity
Open

uipreliga wants to merge 1 commit into
mainfrom
docs/conversion-pitch-parity

Conversation

@uipreliga

Copy link
Copy Markdown
Collaborator

Why

GitHub traffic for the 14 days to 2026-09-13:

Metric Value
Unique visitors 433
Stars gained 10
Visit → star ~2.3%
Share of views on the README ~75%

Two things follow. The README is the entire funnel — /discussions got 49 views against 1,028 for the overview page. And spike traffic converts worst: the Sep 4–5 Hacker News wave brought 179 unique visitors and produced 2 stars, about 1%.

A second problem sat underneath it. Eight surfaces described the project eight different ways, and four were over the 155-character budget where Google and GitHub truncate.

One pitch, layered

Each surface takes as many layers as it has room for.

Layer Text
1 — Hook Playwright for coding agents.
2 — Promise Test that your skills, MCP servers, and CLIs actually work when an agent uses them.
3 — Mechanism Sandboxed YAML suites, weighted scoring, A/B experiments, CI gates.
4 — Roster Claude Code, Codex, Gemini, OpenCode, Pi.

Layer 2 previously existed on exactly one surface — the plugin manifest — and is the only layer that answers why do I care?

What changed

Surface Before After
README.md H1 SEO sentence layers 1+2
README.md intro 249 words 128 words
pyproject.toml 152 ch 114 ch
mkdocs.yml 242 ch (truncated) 145 ch
docs/index.md 311 ch (truncated) 143 ch
docs/llms.txt roster-first layers 1+2 lead
action.yml install-first outcome-first
.github/pages-stub 300+ ch aligned
plugin.json already right untouched

Quick Start moves from line 76 to 63. The duplicate "What you can do with it" section is merged into the feature bullets, and a star request was added after Quick Start.

Three notes on the constraints that shaped this:

  • pyproject.toml drops the harness list because CE047 checks description together with keywords, which already names all five. That frees ~50 characters of the visible PyPI subtitle.
  • mkdocs.yml is the one surface where layer 2 does not fit: CE047 checks site_description in isolation, so ~48 of its 155 characters must name the roster.
  • docs/index.md keeps a keyword-shaped H1. Unlike the README, that page ranks on its own H1; on github.com the page title is UiPath/coder_eval: <repo description>, so the repo description carries the ranking there.

Also in this PR

  • New social preview card (.github/social-preview/) with its HTML source and a regenerate + upload runbook. The old card named three of five harnesses and carried the weaker headline. Already uploaded.
  • Deleted the stray root comparison.md — a design note about the grading-material leak (fix(docker): harness isolation via COPY/PRUNE + GRADE-OUTSIDE. Close the criteria/grader leak #88), unrelated to docs/comparison.md, linked from nowhere.
  • Moved ADOPTERS.md to .github/ — with one entry (the maintainer), a root-level adopters file is negative social proof, and nothing linked to it.

Applied outside the tree

Not visible in the diff, so worth stating:

  • Repo description → 164 chars (was 249, truncated, and missing Pi).
  • Wiki turned off — it was enabled and empty.
  • Topics reworked so all five harnesses appear: long-horizon-agentsmcp, then regression-testing + claude-agent-sdkopencode + pi-coding-agent. Not pi: that topic is roughly half Raspberry Pi content.

⚠️ Nothing enforces the topic roster the way CE047 enforces the seven prose surfaces. A sixth harness needs a topic as well, or the roster-free repo description quietly under-sells it again.

Verification

make lint passes (635), including CE047 on all seven surfaces. tests/test_action_version_pin.py and tests/test_custom_lint.py pass (639).

Not in this PR

Still open from the audit: the YAML example is at 79% depth and belongs on the first screen; the badge row is still 7 wide with no star badge; hero.gif is still 5.9 MB; and coder-eval.com lives outside this repo and still carries the old pitch.

The change will not be measurable for a while. Ten stars in 14 days means 2.3% → 4% is about 7 extra stars — inside the noise at this volume. The honest test is the next traffic spike.

🤖 Generated with Claude Code

https://claude.ai/code/session_016BPXB3v34yHtVV4PFYYLQC

Eight surfaces described the project eight different ways, and four of them
were over the 155-character budget where Google and GitHub truncate. Repo
traffic showed the cost: ~75% of all page views are the README, and the
Sep 4-5 Hacker News wave converted 179 unique visitors into 2 stars (~1%).

One pitch, layered, so each surface takes as many layers as it has room for:

  1. Hook       Playwright for coding agents.
  2. Promise    Test that your skills, MCP servers, and CLIs actually work
                when an agent uses them.
  3. Mechanism  Sandboxed YAML suites, weighted scoring, A/B, CI gates.
  4. Roster     Claude Code, Codex, Gemini, OpenCode, Pi.

Layer 2 previously existed on exactly one surface (the plugin manifest) and
is the only layer that answers "why do I care?".

- README: the H1 now carries layers 1+2; the centred tagline says where a run
  happens; the intro is 249 -> 128 words; the duplicate "What you can do with
  it" section is merged into the feature bullets. Quick Start moves 76 -> 63.
- pyproject: the roster leaves `description` because CE047 checks it together
  with `keywords`, which already names all five harnesses (~50 chars freed on
  the visible PyPI subtitle).
- mkdocs: 242 -> 145 chars, now inside the SERP budget. This is the one surface
  where layer 2 does not fit: CE047 checks `site_description` in isolation, so
  it must spend ~48 chars naming the roster.
- docs/index: 311 -> 143 chars. The H1 stays keyword-shaped; unlike the README,
  that page ranks on its own H1.
- llms.txt: layers 1+2 lead the blockquote, so an LLM quoting one sentence
  quotes the right one.
- action.yml / pages stub: same voice; `plugin.json` is left alone, having got
  this right first.

Also:

- New social preview card (.github/social-preview/), with its HTML source and
  a regenerate + upload runbook. The old card named three of five harnesses and
  carried the weaker headline. It needs a manual upload - GitHub has no API for
  the social preview image.
- Delete the stray root comparison.md: a design note about the grading-material
  leak (PR #88), unrelated to docs/comparison.md, linked from nowhere.
- Move ADOPTERS.md to .github/. With one entry - the maintainer - a root-level
  adopters file is negative social proof, and nothing linked to it.

Applied outside the tree, so not visible in this diff: the repo description
(164 chars, was 249 and missing Pi), the wiki turned off, and topics reworked
so all five harnesses appear - long-horizon-agents -> mcp, then
regression-testing + claude-agent-sdk -> opencode + pi-coding-agent. Not `pi`:
that topic is ~50/50 Raspberry Pi. Note nothing enforces the topic roster the
way CE047 enforces the seven prose surfaces.

Rationale and the open items are recorded in c/2026-09-14-*.md (gitignored).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016BPXB3v34yHtVV4PFYYLQC
@github-actions

github-actions Bot commented Sep 14, 2026

Copy link
Copy Markdown

Claude finished @uipreliga's task in 1m 44s —— View job


Code Review in Progress

Todo List:

  • Read .github/code_review.md for review standards and output format
  • Read CLAUDE.md for project conventions
  • Review the full diff (git diff origin/main...HEAD)
  • Read full context of changed files
  • Perform cross-file consistency checks
  • Analyze "what's missing"
  • Post comprehensive review feedback

Starting review...

Comment thread README.md
@@ -1,4 +1,4 @@
# Coder Eval — evaluate and benchmark AI coding agents and their skills
# Coder Eval — Playwright for coding agents — test that your skills, MCP servers, and CLIs actually work when an agent uses them.

@bai-uipath bai-uipath Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a fan of featuring Playwright (a non-uipath product) so prominently here - how about

Coder Eval - evaluate coding agents, skills, MCP servers, and CLIs on your own tasks

Comment thread README.md
check, an A/B experiment layer, and per-tool cost telemetry, over whatever work *you*
care about. See [How it compares](https://coder-eval.com/docs/comparison).
📚 **Full docs:** **[coder-eval.com/docs](https://coder-eval.com/docs)**.
open-source, **agent-agnostic** framework for evaluating and benchmarking AI coding

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

too much information dump for the the first paragraph, I'd prefer something simple and to the point

Coder Eval runs a real coding agent — Claude Code, Codex, Antigravity (Gemini), OpenCode, or Pi — in a sandbox against declarative YAML tasks, then scores the files and commands it actually produced. You bring the tasks and the scoring, so it measures the work you care about rather than a fixed leaderboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants