Conversation
Eight surfaces described the project eight different ways, and four of them
were over the 155-character budget where Google and GitHub truncate. Repo
traffic showed the cost: ~75% of all page views are the README, and the
Sep 4-5 Hacker News wave converted 179 unique visitors into 2 stars (~1%).
One pitch, layered, so each surface takes as many layers as it has room for:
1. Hook Playwright for coding agents.
2. Promise Test that your skills, MCP servers, and CLIs actually work
when an agent uses them.
3. Mechanism Sandboxed YAML suites, weighted scoring, A/B, CI gates.
4. Roster Claude Code, Codex, Gemini, OpenCode, Pi.
Layer 2 previously existed on exactly one surface (the plugin manifest) and
is the only layer that answers "why do I care?".
- README: the H1 now carries layers 1+2; the centred tagline says where a run
happens; the intro is 249 -> 128 words; the duplicate "What you can do with
it" section is merged into the feature bullets. Quick Start moves 76 -> 63.
- pyproject: the roster leaves `description` because CE047 checks it together
with `keywords`, which already names all five harnesses (~50 chars freed on
the visible PyPI subtitle).
- mkdocs: 242 -> 145 chars, now inside the SERP budget. This is the one surface
where layer 2 does not fit: CE047 checks `site_description` in isolation, so
it must spend ~48 chars naming the roster.
- docs/index: 311 -> 143 chars. The H1 stays keyword-shaped; unlike the README,
that page ranks on its own H1.
- llms.txt: layers 1+2 lead the blockquote, so an LLM quoting one sentence
quotes the right one.
- action.yml / pages stub: same voice; `plugin.json` is left alone, having got
this right first.
Also:
- New social preview card (.github/social-preview/), with its HTML source and
a regenerate + upload runbook. The old card named three of five harnesses and
carried the weaker headline. It needs a manual upload - GitHub has no API for
the social preview image.
- Delete the stray root comparison.md: a design note about the grading-material
leak (PR #88), unrelated to docs/comparison.md, linked from nowhere.
- Move ADOPTERS.md to .github/. With one entry - the maintainer - a root-level
adopters file is negative social proof, and nothing linked to it.
Applied outside the tree, so not visible in this diff: the repo description
(164 chars, was 249 and missing Pi), the wiki turned off, and topics reworked
so all five harnesses appear - long-horizon-agents -> mcp, then
regression-testing + claude-agent-sdk -> opencode + pi-coding-agent. Not `pi`:
that topic is ~50/50 Raspberry Pi. Note nothing enforces the topic roster the
way CE047 enforces the seven prose surfaces.
Rationale and the open items are recorded in c/2026-09-14-*.md (gitignored).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016BPXB3v34yHtVV4PFYYLQC
uipreliga
requested review from
CarlesUIPath,
akshaylive,
bai-uipath and
tmatup
as code owners
September 14, 2026 21:47
|
Claude finished @uipreliga's task in 1m 44s —— View job Code Review in Progress
|
bai-uipath
approved these changes
Sep 14, 2026
| @@ -1,4 +1,4 @@ | |||
| # Coder Eval — evaluate and benchmark AI coding agents and their skills | |||
| # Coder Eval — Playwright for coding agents — test that your skills, MCP servers, and CLIs actually work when an agent uses them. | |||
Collaborator
There was a problem hiding this comment.
not a fan of featuring Playwright (a non-uipath product) so prominently here - how about
Coder Eval - evaluate coding agents, skills, MCP servers, and CLIs on your own tasks
| check, an A/B experiment layer, and per-tool cost telemetry, over whatever work *you* | ||
| care about. See [How it compares](https://coder-eval.com/docs/comparison). | ||
| 📚 **Full docs:** **[coder-eval.com/docs](https://coder-eval.com/docs)**. | ||
| open-source, **agent-agnostic** framework for evaluating and benchmarking AI coding |
Collaborator
There was a problem hiding this comment.
too much information dump for the the first paragraph, I'd prefer something simple and to the point
Coder Eval runs a real coding agent — Claude Code, Codex, Antigravity (Gemini), OpenCode, or Pi — in a sandbox against declarative YAML tasks, then scores the files and commands it actually produced. You bring the tasks and the scoring, so it measures the work you care about rather than a fixed leaderboard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Why
GitHub traffic for the 14 days to 2026-09-13:
Two things follow. The README is the entire funnel —
/discussionsgot 49 views against 1,028 for the overview page. And spike traffic converts worst: the Sep 4–5 Hacker News wave brought 179 unique visitors and produced 2 stars, about 1%.A second problem sat underneath it. Eight surfaces described the project eight different ways, and four were over the 155-character budget where Google and GitHub truncate.
One pitch, layered
Each surface takes as many layers as it has room for.
Layer 2 previously existed on exactly one surface — the plugin manifest — and is the only layer that answers why do I care?
What changed
README.mdH1README.mdintropyproject.tomlmkdocs.ymldocs/index.mddocs/llms.txtaction.yml.github/pages-stubplugin.jsonQuick Start moves from line 76 to 63. The duplicate "What you can do with it" section is merged into the feature bullets, and a star request was added after Quick Start.
Three notes on the constraints that shaped this:
pyproject.tomldrops the harness list because CE047 checksdescriptiontogether withkeywords, which already names all five. That frees ~50 characters of the visible PyPI subtitle.mkdocs.ymlis the one surface where layer 2 does not fit: CE047 checkssite_descriptionin isolation, so ~48 of its 155 characters must name the roster.docs/index.mdkeeps a keyword-shaped H1. Unlike the README, that page ranks on its own H1; on github.com the page title isUiPath/coder_eval: <repo description>, so the repo description carries the ranking there.Also in this PR
.github/social-preview/) with its HTML source and a regenerate + upload runbook. The old card named three of five harnesses and carried the weaker headline. Already uploaded.comparison.md— a design note about the grading-material leak (fix(docker): harness isolation via COPY/PRUNE + GRADE-OUTSIDE. Close the criteria/grader leak #88), unrelated todocs/comparison.md, linked from nowhere.ADOPTERS.mdto.github/— with one entry (the maintainer), a root-level adopters file is negative social proof, and nothing linked to it.Applied outside the tree
Not visible in the diff, so worth stating:
long-horizon-agents→mcp, thenregression-testing+claude-agent-sdk→opencode+pi-coding-agent. Notpi: that topic is roughly half Raspberry Pi content.Verification
make lintpasses (635), including CE047 on all seven surfaces.tests/test_action_version_pin.pyandtests/test_custom_lint.pypass (639).Not in this PR
Still open from the audit: the YAML example is at 79% depth and belongs on the first screen; the badge row is still 7 wide with no star badge;
hero.gifis still 5.9 MB; andcoder-eval.comlives outside this repo and still carries the old pitch.The change will not be measurable for a while. Ten stars in 14 days means 2.3% → 4% is about 7 extra stars — inside the noise at this volume. The honest test is the next traffic spike.
🤖 Generated with Claude Code
https://claude.ai/code/session_016BPXB3v34yHtVV4PFYYLQC