Make MCP tool errors actionable and tolerate common agent arguments

[NielsRogge]

Why

A 48-hour audit of Papers With Code production chat (2026-09-16 to 09-18) showed the MCP tools failing in ways the model could not act on, and burning its 12-call tool budget on retries. Per-tool failure rates: read_paper 51 of 63 calls, get_paper_info 74 errors (48 of them generic), list_benchmarks 25 argument rejections, get_benchmark 13 metric-name rejections.

Replaying the failing production inputs through build_server(CatalogClient()) in-process pinned each cause.

Fixes

Catalog client (pwc_mcp/catalog.py)

• _rows used payload.get("results") or payload.get("items"), so an empty results list read as a missing list and every title not in the catalog became upstream_error: the Papers With Code catalog request failed. Empty results now resolve to not_found: Paper title not found: ….
• DOI, publisher, and venue URLs fail fast with not_found: Paper URL not supported … instead of paging through empty searches.
• Numeric catalog IDs (what list tools return) canonicalise to the paper's arXiv ID via one cached papers/{id} call, because the Markdown route stores arXiv papers under the arXiv ID only. External papers keep their numeric ID.
• A Markdown 404 for a paper that does not exist reports not_found: Paper not found: … instead of no_markdown.

Transport (pwc_cli/transport.py)

• Read timeouts surfaced as a bare TimeoutError and other socket failures as OSError, which the MCP server treated as unexpected exceptions. Both are TransportError now; timeouts map to upstream_timeout.

Server (pwc_mcp/server.py)

• 400/422 → invalid_argument: <bounded, control-char-stripped API detail>; 429 → rate_limited; "timed out" recognised alongside "timeout". The remaining generic path logs only the exception class name.
• list_benchmarks: order_by=trending without task falls back to name ordering; area combined with task/search/ordering filters is dropped in favour of the filtered flat listing; grouped listings no longer forward page/limit. Each fallback is stated in the text summary. Descriptions updated.
• read_paper identity mismatch says "paper changed; restart reading from the beginning" instead of the generic message.

CLI (pwc_cli/cli.py)

• Metric names for --sort, --min, --max, --require-metrics, and --pareto resolve case-insensitively, by normalised spelling (Top-1 Accuracy = Top 1 Accuracy), and through a small alias table (AP↔mAP, top1↔Accuracy, AUROC↔AUC, F1↔F1 score, …) before the unknown-metric error. The MCP inherits this.
• Not-found hints rank the closest results by shared words with the reference instead of raw search order.

Versions: pwc-cli 0.4.1 → 0.4.2, pwc-mcp 0.2.1 → 0.2.2. standalone_cli/SKILL.md regenerated; it had drifted from the parser (page-size flag names, two --search flags).

Verification

uv run --project standalone_cli --with pytest pytest standalone_cli/tests   # 86 passed
uv run --project mcp_server pytest mcp_server/tests                        # 54 passed
uv run python standalone_cli/scripts/check_contract.py                      # ok

Replay of the production inputs against the live API after the change: DOI/ACL URLs → not_found; 2308.10195 → not_found for both info and read; read_paper("86122") resolves to 2505.18132; get_benchmark("coco-val2017", sort_metric="AP:desc") and ("kinetics-400", "top1:desc") succeed; list_benchmarks(order_by="trending") and (area, search) succeed with notes.

Not fixed here (backend follow-up)

GET /api/v1/research/papers/2505.18132/read returns 200, but the same route with ?offset=0&limit=65536 returns 404: chunk mode requires a _content/manifests/… file that older Markdown uploads never received. read_paper will keep reporting no_markdown for those papers until the backend either backfills manifests or lets chunk mode fall back to the plain object. Tracked in the paperswithcode repo.

Release

sudo pwc-deploy staging <sha> --component mcp, exercise the four tools from a staging chat, then sudo pwc-deploy production <sha> --component mcp --confirm DEPLOY_PRODUCTION.

🤖 Generated with Claude Code