[albinazs]

Follow-up to #486. Adds a promptfoo benchmark that runs Claude Code against the guide with 33 realistic user questions. It records which agent readiness features each run used, and grades the answers.

• Conditions: no tools, the real site, a plain site with every agent feature removed, the real site plus the MCP server, and the real site plus curl.
• Grading: answer checks for facts and the right guide link, a Sonnet judge, trace columns for the surfaces each run reached, and a proxy log of requests and content negotiation.
• Commands: just eval-agents-proxy local, just eval-agents, just eval-agents-view. See README.md.

Full results are in RESULTS.md.

Summary of findings (Haiku, Sonnet and Opus 5.5, 3 repeats)

Condition	Haiku	Sonnet	Opus
No tools	10%	15%	36%
Real site	80%	89%	89%
Plain site	87%	90%	96%
Real site, Markdown from HTML	80%	93%	–
Real site + MCP	97%	100%	100%

• MCP is the one clear win: +11–17 points on every model, and faster and cheaper answers. This assumes the user has already connected the server, since agents don't connect MCP servers themselves.
• Discovery files are only used when a page links to them. On the guide as it is, llms.txt, the agent skill, .well-known files, openapi.json and robots.txt were used in 0 of 564 runs. Agents search the web, land on a deep link, and read that one page.
• Today's Markdown pages hide the navigation. The homepage has 5 links in Markdown against 47 in HTML, and Releases has 0 against 16. That's why the plain site does as well or better.
• Converting Markdown from HTML with Wagtail 8.1's converter (see the feature/markdown-responses PR) takes Sonnet from 89% to 93%. Haiku stays level, because the homepage doesn't link to Releases. Linked llms.txt and llms-full.txt got their first visits.
• <head> links don't reach agents answering users. Given curl, agents used it in 0 of 198 runs.
• Markdown is 4–27× smaller than HTML, but saves Claude Code only 2–11% per fetch, because WebFetch summarizes pages.
• The benchmark cost about $25 at API prices, plus about $8 for the follow-up conditions.

Browsing the traces locally

The run results are attached: agent-readiness-results.zip. The zip holds one promptfoo results file per model, with every answer, tool call and grader reason, plus its summary.

unzip agent-readiness-results.zip && cd agent-readiness-results
npx promptfoo@0.124.1 import --new-id haiku.json   # same for sonnet.json, opus.json
npx promptfoo@0.124.1 view                         # http://localhost:15500

This needs Node ≥ 22.22. In the viewer, pick a model's eval from the dropdown. Each column is a condition and each row a question. Click a cell to see that run's answer, its tool calls and the grader reasons. The trace columns are listed as named scores.

For the summary tables, run this from the repository:

uv run python content/evals/agent-readiness/summarize.py path/to/haiku.json

AI disclosure: AI-assisted design and writing of the eval harness, running the benchmark, and drafting the results.