diff --git a/content/evals/agent-readiness/README.md b/content/evals/agent-readiness/README.md new file mode 100644 index 00000000..01ba8d18 --- /dev/null +++ b/content/evals/agent-readiness/README.md @@ -0,0 +1,82 @@ +# Agent readiness evals + +Which of the guide's agent readiness features do agents actually use, and do they help agents answer users' questions correctly? This eval runs Claude Code against the guide with realistic user questions, records which surfaces each run reached (llms.txt, Markdown pages, content negotiation, MCP, agent skill, and more), and grades the answers. It follows up on [#486](https://github.com/wagtail/guide/issues/486). The latest results are in [RESULTS.md](RESULTS.md). + +The method borrows from the [draftjs_exporter skill evals](https://github.com/wagtail/draftjs_exporter/blob/main/docs/contributing/evals.md) and Anthropic's [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). The trace columns follow the [ora.ai / Vercel agent readiness dataset](https://github.com/agentready-org/standard/blob/main/data/traces.csv), so the results can be compared. + +## Arms + +Each arm is a variant of the same experiment: the same questions, the same model with the default Claude Code system prompt, run in an empty directory outside the repository with no user or project settings. Arms differ in one thing only: + +| Arm | What's different | Measures | +| ----------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------- | +| `baseline` | No tools | What the model knows from memory | +| `web` | `WebSearch` and `WebFetch`, on the guide as it is | What agents do today, and which features they find on their own | +| `web-plain` | Same tools, on a plain site without any agent readiness features (see below) | What the features are worth: `web` minus `web-plain` | +| `web+mcp` | Same tools, plus the guide's MCP server | Whether agents prefer MCP when they have it, and do better | +| `web+curl` | Same tools, plus `curl` through Claude Code's Bash tool | Whether agents read raw HTML, and the links in its `` | + +The plain site is served by the proxy (`proxy.py`), with no changes to the guide. It returns 404 for llms.txt, llms-full.txt, Markdown pages, everything under `/.well-known/` (agent skills, AI catalog, API catalog), `openapi.json` and the MCP server. It ignores requests for Markdown, so pages come back as HTML. From pages, it removes JSON-LD, Markdown and catalog links, and the script bundle that registers WebMCP tools. From robots.txt, it removes the Content Signals. + +## Cases + +`tests.yaml` covers the guide's tasks: information gathering, troubleshooting, and keeping up with releases, plus developer questions. + +- Release cases cover recent releases (Wagtail 7.3 to 8.0), which models know little about from memory. +- Some prompts name the guide, others don't, so the agent has to find it, and some send the agent to the site ("Go to guide.wagtail.org and find…"), like the ora.ai tasks. +- Negative cases are developer questions, which the developer docs answer rather than the guide. +- Bridge cases are developer questions the guide answers itself, with a link to the developer docs. + +## Grading + +- **Answer checks** (pass or fail): key facts, and the URL of the guide page that answers the question. +- **Judge** (pass or fail): an `llm-rubric` for nuance, graded by Claude Sonnet 5.5 with no tools. When it can't tell, it passes the answer with a reason starting `UNKNOWN:`, and `summarize.py` lists those for review by hand. +- **Trace columns** (recorded, never fail a test): `graders/trace_columns.py` classifies each tool call into a surface and reports them as named scores. `answer_grounded` means the answer cites a guide page the run fetched. `versioned_url_reached` flags versioned guide URLs like `/en-6.3.x/` (which redirect to the current page), and `legacy_editor_manual_reached` the old editor manual on docs.wagtail.org. `curl_calls` and `permission_denials` count shell use in the `web+curl` arm. `*_linked` counts fetches of URLs that appeared in an earlier tool output, but `WebFetch` returns a summary of the page rather than the page, so links are undercounted. +- **Proxy log**: the agent's trace doesn't show request headers, so the proxy logs every guide request with its `Accept` header, status, and the content type served. This is how we see Markdown content negotiation. + +`just eval-agents-check` runs before every eval, without calling a model. It checks that every case's reference answer passes its answer checks, that empty, refusing, URL-only and URL-less answers fail them, and that no prompt contains the facts its checks look for. + +## Running the evals + +Requires Node 26 (see `.nvmrc`), `uv`, and a local Claude Code login or `ANTHROPIC_API_KEY`. With a Claude subscription login, runs count towards your plan's usage limits instead of being billed per token. The costs promptfoo reports are what the same tokens would cost through the API. + +```sh +just eval-agents-init # promptfoo + Claude Agent SDK, once + +# In one terminal: the proxy, forwarding to `just run` (local) or the live site (prod). +just eval-agents-proxy local + +# In another terminal: +just eval-agents # Claude Sonnet 5.5 +just eval-agents claude-haiku-5-5 --repeat 3 --output run.json +just eval-agents-view # promptfoo viewer + +uv run python content/evals/agent-readiness/summarize.py run.json \ + --proxy-log ~/.cache/guide-agent-eval/requests.jsonl +``` + +Local mode needs the development server running with the guide's real content, for example from a restored production database dump: the fixtures from `just buildfixtures` don't have the pages the cases ask about. + +The first run creates a mitmproxy certificate authority in `~/.mitmproxy`. Only the agent trusts it, through `NODE_EXTRA_CA_CERTS`, and the proxy only intercepts `guide.wagtail.org`. In local mode, those requests go to the development server, so unreleased features like the MCP server can be tested with the real domain in every URL. `WebSearch` results still come from the live search index. + +`just eval-agents` also builds a CA bundle for the `web+curl` arm, as curl doesn't read `NODE_EXTRA_CA_CERTS`. + +promptfoo exits with code 100 when any test fails, which is expected for the `baseline` arm. + +### Running one arm, or testing a site change + +To rerun one arm, filter providers by label, like `--filter-providers '^web$'`, and `--filter-pattern '^start-'` to run some cases only. To measure a change to the site, rerun the `web` arm against the changed local site, relabel its results (`jq '.results.results[].provider.label = "web-md"' run.json > web-md.json`), and pass them to `summarize.py` next to the earlier runs, which reports each label as its own arm. `--override` replaces an arm's results with a rerun of it: + +```sh +uv run python content/evals/agent-readiness/summarize.py run-1.json run-2.json \ + --override baseline-rerun.json +``` + +## Reading results with care + +- **One repeat is not enough to act on.** Run with `--repeat 3` before comparing arms, and compare arms at the same repeat count. +- **Grade what the agent produced, not the path.** Trace columns show which features were used, and a run that skips llms.txt but answers correctly isn't a failure. +- **Read the transcripts.** Before trusting a score, check failing runs in `just eval-agents-view`: if a correct answer failed, fix the case, not the agent. +- **The proxy log is per eval run, not per test.** Clear `requests.jsonl` before a run to attribute its requests. +- **Cost is what Claude Code reports per run, at API list prices.** It includes the model, the Haiku calls `WebFetch` uses to summarize pages, and web search fees, but not the judge. With a subscription login, it's what the run would cost through the API, not what you pay. +- **Local mode is not production.** Production adds Cloudflare caching and the live content. diff --git a/content/evals/agent-readiness/RESULTS.md b/content/evals/agent-readiness/RESULTS.md new file mode 100644 index 00000000..e4da1fce --- /dev/null +++ b/content/evals/agent-readiness/RESULTS.md @@ -0,0 +1,303 @@ +# Agent readiness results: guide.wagtail.org + +Which agent readiness techniques do AI agents actually use on the guide, and do they help agents answer users' questions? Results from October 2026, for [#486](https://github.com/wagtail/guide/issues/486). See [README.md](README.md) for the method and how to rerun it. + +## In short + +- **The MCP server is the one technique that clearly pays off.** With it, agents answered 97–100% of questions correctly on every model, against 80–89% when browsing the site. They were also up to twice as fast, and up to four times cheaper. This assumes the user has connected the server: agents don't find or connect MCP servers on their own, so users need to know it exists. +- **Agents only use discovery files that pages link to.** On the guide as it is, llms.txt, the agent skill, `.well-known` files, openapi.json and robots.txt were used in **0 of 564 runs**, even when we sent agents to the site to look something up: agents search the web, get a deep link, and read that one page. Once our Markdown pages linked to llms.txt and llms-full.txt, Haiku started using them in a few runs. Files that no page links to stayed unused. +- **Markdown is used on every page fetch, but our Markdown pages hide the navigation**, including the link to llms.txt. Agents saw fewer links in Markdown than in HTML, so they sometimes couldn't find the right page. The same site with every agent feature switched off did as well or better (+7 points for Haiku and Opus, level for Sonnet). +- **Converting pages to Markdown from their HTML, with Wagtail 8.1's converter, fixes the problem for Sonnet but not yet for Haiku.** Section pages now list their pages and every page links to llms.txt. Sonnet improved in every repeat (89% → 93%), now level with or slightly above the plain site (90%), and llms-full.txt and llms.txt got their first visits. Haiku is still 7 points behind the plain site (80% against 87%): the homepage doesn't link to Releases, because the converter leaves out the menu. +- **Links in the page's `` (AI catalog, API catalog, openapi.json) don't reach agents answering users.** Claude Code reads pages as text, without ``. Given curl to read raw HTML, agents never used it (0 of 198 runs). These links serve crawlers and scanners like ora.ai. +- **Cheap fixes:** ship the MCP server, convert Markdown from HTML (with a link to every section, Releases included, on the homepage), and redirect `/en/latest/*` to `/en/*`. + +## What we tested + +We asked Claude Code 33 realistic questions about using Wagtail, from "How do I schedule a page?" to "Why aren't my changes showing on the site?", and checked whether each answer had the right facts and linked to the right guide page. Each question ran 3 times on each model, under 4 conditions: + +| Condition | What the agent has | What it tells us | +| --------------- | ------------------------------------------------------- | --------------------------------- | +| No tools | Nothing, memory only | How much the model already knows | +| Real site | Web search and page fetching, on the guide as it is | What happens today | +| Plain site | The same, on the guide with every agent feature removed | What our agent features are worth | +| Real site + MCP | The same as real site, plus the guide's MCP server | What MCP adds | + +Two follow-up conditions ran on Haiku and Sonnet: [Markdown converted from HTML](#real-site-with-markdown-from-html), and the same with [curl to read raw HTML](#do-agents-read-raw-html-when-they-can). + +The plain site removes llms.txt, llms-full.txt, Markdown pages and content negotiation, the agent skill, the AI and API catalogs, openapi.json, the MCP server, JSON-LD, WebMCP, Markdown 404s and Content Signals. + +The questions cover information gathering (15), troubleshooting (8), release notes (6), and developer questions (4): 3 that the developer docs answer, and 1 that the guide answers with a link to them. They're asked in three ways: + +- **Naming the guide** (19): "Using the user guide on guide.wagtail.org, how do I…?" +- **Not naming it** (9): "How do I…? Link to official documentation." The agent has to find the guide. +- **Sending the agent to the site** (5): "Go to guide.wagtail.org and find out…", like the tasks in the [ora.ai traces](https://github.com/agentready-org/standard/blob/main/data/traces.csv). Haiku and Sonnet only. + +## How AI agents read web pages + +Different kinds of AI agents see different parts of a page, which explains most of our results. Our benchmark uses Claude Code, a coding agent with a page-fetching tool. + +| Kind of agent | Examples | How it reads a page | Sees `` links (AI catalog, openapi.json)? | +| -------------------------------------------------------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | +| Assistants and coding agents with a fetch tool that summarizes | Claude Code (WebFetch), Gemini CLI | Asks for the page (Claude Code asks for Markdown first), converts it to text, and has a small model summarize it. The main model only reads the summary. | No | +| Agents with a fetch tool that passes the page on | OpenCode | Gives the page's Markdown, text or HTML straight to the model | Only when asking for HTML | +| Agents with a shell | Claude Code, Codex CLI, with `curl` allowed | Reads the raw HTML, if the agent chooses to | Yes, but agents rarely choose it (see below) | +| Browser agents | Claude in Chrome, ChatGPT agent and Atlas, Perplexity Comet | Reads what's shown on screen (accessibility tree and screenshots), and runs JavaScript, so it can use WebMCP tools | No | +| Search and AI crawlers | GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot | Reads raw HTML to build search indexes, which agents then search | Yes, but they don't answer users directly | +| Readiness scanners | ora.ai, is-agentic | Checks which files and links a site has, and scores them | Yes | + +What this means for our results: + +- **Agents answering users mostly see text and links in the page itself.** Files and tags only machines read (`` links, JSON-LD, `.well-known` files) reach crawlers and scanners, which is why ora.ai scores the guide 20/20 on discovery while our agents never used those files. +- **Agents follow links, they don't guess.** A May 2026 [Ahrefs study](https://ahrefs.com/blog/llmstxt-study/) of 137,000 sites found 97% of llms.txt files got no requests, and that "AI tools fetch llms.txt when a link, an index, or a user instruction tells them it exists". Claude Code was the second biggest named reader of llms.txt, after GPTBot. +- **Results for other kinds of agents may differ.** Browser agents can use WebMCP, and agents that read raw HTML can see `` links. We only tested Claude Code. + +## Results at a glance + +Share of answers with the right facts and the right guide link, mean ± variation across 3 repeats: + +| Condition | Haiku 5.5 | Sonnet 5.5 | Opus 5.5\* | +| ----------------------------------------- | ----------- | ------------ | ------------ | +| No tools | 10% ± 2 | 15% ± 3 | 36% ± 4 | +| Real site | 80% ± 2 | 89% ± 2 | 89% ± 4 | +| Plain site | 87% ± 5 | 90% ± 2 | 96% ± 0 | +| Real site, Markdown from HTML† | 80% ± 2 | 93% ± 2 | not run | +| Real site, Markdown from HTML, with curl‡ | 76% ± 5 | 85% ± 3 | not run | +| **Real site + MCP** | **97% ± 3** | **100% ± 0** | **100% ± 0** | + +\* Opus ran the first 28 questions, without the "go to the site" ones. +† The guide with page Markdown converted from the rendered HTML. See [Real site with Markdown from HTML](#real-site-with-markdown-from-html). +‡ The same, with curl available to read raw HTML. See [Do agents read raw HTML when they can?](#do-agents-read-raw-html-when-they-can) + +### MCP versus the real site + +| Model | Correct answers | Time per answer | Cost per answer | +| ---------- | --------------- | --------------- | --------------- | +| Haiku 5.5 | +17 pts | 22s → 11s | 1.6¢ → 0.4¢ | +| Sonnet 5.5 | +11 pts | 17s → 10s | 2.8¢ → 2.5¢ | +| Opus 5.5 | +11 pts | 28s → 16s | 4.6¢ → 4.0¢ | + +### By type of question (real site → with MCP) + +| Question type | Haiku | Sonnet | Opus | +| --------------------- | ----------- | ----------- | ----------- | +| Information gathering | 84% → 96% | 91% → 100% | 91% → 100% | +| Troubleshooting | 67% → 100% | 79% → 100% | 88% → 100% | +| Release notes | 72% → 94% | 89% → 100% | 80% → 100% | +| Developer questions | 100% → 100% | 100% → 100% | 100% → 100% | + +### By how the question is asked (real site → with MCP) + +| Framing | Haiku | Sonnet | Opus | +| --------------------------- | ---------- | ---------- | ---------- | +| Names the guide | 81% → 98% | 95% → 100% | 95% → 100% | +| Doesn't name the guide | 74% → 100% | 81% → 100% | 78% → 100% | +| Sends the agent to the site | 87% → 87% | 80% → 100% | not run | + +Troubleshooting and questions that don't name the guide gain the most: with MCP, the agent goes straight to the guide instead of relying on web search. + +### Without the site + +With no tools, models know many of the facts (68–86%) but rarely the guide page to cite (13–39%). Opus knows the most. Questions about recent releases (Wagtail 7.3 to 8.0) were answered correctly 0–13% of the time from memory, so for recent changes, agents need the site. + +## Which techniques agents actually use + +Share of runs that used each technique, across all models and conditions with web access: + +| Technique | Used? | Evidence | Verdict | +| ------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | +| MCP server | **Yes, ~87% of runs that had it** | Agents chose it over web search almost every time | Keep, and promote it | +| Markdown content negotiation | **Yes, every fetch** | Claude Code always asks for Markdown first, and gets it | Keep, but fix the navigation (below) | +| Guide pages (docs) | Yes, 54–87% of runs | The main source of answers | Keep accurate | +| Web search | Yes, 57–92% of runs without MCP | The usual first step, to find a deep link, even when sent to the site | Make sure pages rank and stay current | +| Homepage | Sometimes: Haiku ~43%, Sonnet ~8%, Opus ~1% | Mostly when sent to the site; stronger models go straight to a page | Markdown version needs the full table of contents | +| Markdown mirror URLs (`/markdown/`) | Rarely, 0–12% | Mostly Opus, after seeing the link | Fine as is | +| llms-full.txt, sitemap.xml | Almost never: 1–2% of Opus runs. llms-full.txt: 5% of Haiku runs once linked | Reached through links, never by guessing | Link from pages agents read | +| llms.txt | **Never** on the guide as it is; 1% of Haiku runs once linked | Today's Markdown homepage doesn't link to it | Link it from Markdown pages (tested with Markdown from HTML) | +| Agent skill (`.well-known/agent-skills`) | **Never** | No page links to it | Link it from the homepage, then retest | +| AI catalog, API catalog, openapi.json, robots.txt | **Never** | Linked only from ``, which Claude Code drops; given curl, agents never read raw HTML | Keep for crawlers and scanners | +| Agent-friendly 404s | **Never seen** | Claude Code doesn't pass the body of a 404 to the model | No impact with Claude Code | +| JSON-LD, WebMCP, Content Signals | Can't be used by Claude Code | It converts pages to text, without scripts, and doesn't run JavaScript | Not measurable with this agent | + +### Why the ora.ai study saw agents use discovery files + +The [ora.ai traces](https://github.com/agentready-org/standard/blob/main/data/traces.csv) show llms.txt used in 23–77% of runs, depending on the model. Their data points to three differences with our setup, not to the agent harness (both harnesses they tested used llms.txt): + +- **Where the agent starts.** Their agents started on the site's homepage in 91–96% of runs, for tasks like "find the pricing". Ours mostly start from a web search, which lands on the right page directly. +- **Links.** 86% of their llms.txt fetches followed a link the agent had seen. Agents almost never guess the path. Our Markdown pages, which are what Claude Code reads, don't link to llms.txt. +- **The model.** Claude Fable 5 used llms.txt in 54–77% of their runs, and almost never searched the web. Other models used it in 23–37%. We haven't tested Fable yet. + +To check the first point, we added 5 questions that send the agent to the site ("Go to guide.wagtail.org and find out…"). Agents then reached the homepage much more often (Haiku 86%, Sonnet 46%), but still searched first in most runs, and **still never opened llms.txt**. This fits the links explanation: from today's Markdown homepage, there's no link to follow. With [Markdown from HTML](#real-site-with-markdown-from-html), which links llms.txt from every page, Haiku started opening it. + +### Why the ora.ai study doesn't report on MCP + +The ora.ai study measured which files agents reached and where their answers came from, not whether the answers were right, so it couldn't show what MCP adds. Its traces have no MCP column, and only 20 of 1,033 runs got their answer from an MCP server. Agents don't connect MCP servers mid-task: the user connects them before the session starts. An agent browsing a site barely meets MCP. + +Our "real site + MCP" condition connects the guide's MCP server before every run, like a user who has installed it. So it measures what MCP is worth once installed, not whether agents find it. That makes promoting the server part of shipping it. + +## What a typical agent does + +The most common paths through the site, per model: + +| Model | Real site | With MCP | +| ------ | ----------------------------------------------------------------------------- | ---------------------------- | +| Haiku | search → guide page (×1–2), often via the homepage | MCP search (×1–2) → MCP read | +| Sonnet | search → guide page (38%), or **search only, without reading any page (30%)** | MCP search → MCP read | +| Opus | search → guide page (32%), or straight to a guide page from memory (12%) | MCP search → MCP read | + +When sent to the site, the most common path is **search → guide homepage → guide page** (Haiku) or **search → guide page** (Sonnet). + +- **Agents take 2–3 steps.** They don't explore: one search, one or two pages, then the answer. +- **Sonnet often answers from search snippets alone.** Only just over half its real-site answers cite a page it actually read, against 88% with MCP. +- **Search sends agents to old guide URLs** (`/en-6.3.x/…`, `/en-latest/…`) in 23–64% of runs. These redirect to the current page, so the answers stay right. +- **Developer questions go to docs.wagtail.org**, as they should, in every condition with web access. + +## Where agents fail + +Outside the no-tools condition, failures fall into four kinds: + +| Kind of failure | Example | Cause | +| ---------------------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------- | +| Couldn't find the right guide page | Cites the developer docs or a section page instead of "Manage images" | Markdown section pages don't list their pages (fixed by Markdown from HTML) | +| Missed part of a multi-page answer | "What changed from 7.2 to 8.0?" covers 8.0 only | Same: the Markdown Releases page lists no releases (fixed by Markdown from HTML) | +| Used an outdated source | Cites the old editor manual on docs.wagtail.org | Old pages still rank in search | +| Wrong or incomplete answer | Mixes up multi-site and multi-tenancy | The page answers it, but the agent read it too quickly | + +Questions that failed most often on the real site: + +- **"What changed between 7.2 and 8.0?"**: Opus failed 3 out of 3 on the real site and passed 3 out of 3 on the plain site. On the real site, it never found the 7.4 release notes. +- **"Go to guide.wagtail.org and find the latest release"**: Haiku and Sonnet failed 2 out of 3 on the real site. The guide gave them no list of releases to follow (the homepage doesn't link to Releases, and the Markdown Releases page listed none), so they answered from the developer docs instead. +- **"How do I see which pages are locked?"**: agents go to the old editor manual on docs.wagtail.org. +- **"What's the difference between multi-site and multi-tenancy?"**: answers mix up multi-tenancy and multi-instance. +- **"How do I get a deleted document back?"**: some answers miss that deleted documents can't be recovered. + +### Why the real site sometimes does worse than the plain site + +Claude Code asks for Markdown, and our Markdown pages leave out the navigation: + +| Page | Links in HTML | Links in Markdown | +| -------- | --------------------------------- | ----------------- | +| Homepage | 47 (every page, through the menu) | 5 (four sections) | +| Releases | 16 (every release) | 0 | + +With the plain site, agents get HTML with the full menu, see "Manage images" or "New in Wagtail 7.4", and go there. With the real site, they see only the section names, and either guess or give up on the guide. + +## Real site with Markdown from HTML + +Our Markdown pages are rendered from page content only, so they leave out what visitors see around it: the "In this section" lists of section pages, and links to llms.txt. We tested the alternative: converting each page's rendered HTML to Markdown, with the `HTMLToMarkdownConverter` coming in Wagtail 8.1. + +### How we tested it + +- **Prototype on the guide:** the guide still runs Wagtail 8.0, so we tested a copy of the converter from Wagtail's `main` branch, adapted to 8.0 (with `draftjs_exporter` 7.3.1, and entity map keys as strings, as in 8.1). The change itself, using Wagtail 8.1's converter, is on the `feature/markdown-responses` branch. +- **What agents get:** page Markdown responses (content negotiation and `/markdown/` URLs) are converted from the page's full HTML (`content_selector="body"`), after the same frontmatter and `Page URL` line as before. The converter leaves out the menu (`