Resolve titles, metrics, and entity names the way chat agents spell them - #21
Merged
Merged
Conversation
Production chat traces (36 h, 2026-09-18/19) show the remaining MCP tool
failures that are not catalog gaps:
- `get_paper_info("Attention Is All You Need")` failed with "Too many results
to resolve paper title safely": the exact title was the first keyword hit,
but resolution paged through ten pages of common-word matches and gave up.
Titles are now looked up as one quoted phrase first (websearch_to_tsquery
phrase semantics, one small page), falling back to the plain query, and an
exact match found before the page budget runs out is used.
- `get_benchmark` rejected `sort_metric` spellings such as "Overall" for
"GenEval Score", "D4RL Normalized Score" for "Normalized Score", "Pass@1"
for "Accuracy", "Pass Rate" for "Success Rate", and "image-level AUROC" for
"Detection AUROC" (12 rejections after the alias table shipped). Metric
resolution now also accepts the one leaderboard metric that contains the
request or one of its aliases, and the alias table covers Pass@1, pass and
success rate, overall, F-score, image/pixel-level AUROC, WER, CER, and EM.
- `get_method("mixture of experts")` reported "not found" although the only
candidate was "Mixture-of-Experts (MoE)". Method, task, conference,
organization, and framework lookups now fall back to a unique match on
letters and digits alone, ignoring a trailing acronym but not a qualifier
such as "(Video)".
- `read_paper` rejected a valid continuation cursor when the agent continued
with another reference to the same paper (the numeric catalog ID after
starting from the arXiv ID). The cursor is honoured when the new reference
resolves to the cursor's paper, and the invalid-cursor error now says what
to pass instead.
Bumps pwc-cli to 0.4.3 and pwc-mcp to 0.2.4.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Production chat traces from 2026-09-18/19 (after the alias table from #18 and the hybrid default from #19 were deployed at 14:10 UTC on 09-18) still show MCP tool failures that are not catalog coverage gaps:
get_paper_info("Attention Is All You Need")→ "Too many results to resolve paper title safely"get_benchmark"unknown metric(s)" forOverall,Score,D4RL Normalized Score,Pass@1,Pass Rate,image-level AUROCget_method("mixture of experts")not found; only candidateMixture-of-Experts (MoE)read_paper"invalid continuation cursor" when continuing with the numeric catalog ID after starting from the arXiv IDWhat changes
catalog.py): quoted-phrase keyword query first (websearch_to_tsqueryphrase semantics, one small page), plain query as fallback; an exact match found before the page budget runs out is used instead of raising.cli.py): after exact/normalized/alias lookup, accept the one leaderboard metric containing the request or an alias of it (or contained in it). New aliases: Pass@1, pass/success rate, overall, F-score/F-measure, image/pixel-level AUROC, WER, CER, EM. Ambiguous requests (e.g.AUROCon MVTec with Detection and Segmentation AUROC,F-scoreon Tanks and Temples) still error with the leaderboard's names.cli.py): methods, tasks, conferences, organizations, frameworks fall back to a unique letters-and-digits match, ignoring a trailing acronym like(MoE)/(ViT)/(PINNs)but not a qualifier like(Video).cursors.py,server.py): a valid cursor supplied with a different reference is honoured when that reference resolves to the cursor's paper; otherwise an explicit "different paper" error. Garbage cursors get an actionable message.Verification
uv run --project standalone_cli --with pytest pytest standalone_cli/tests: 88 passeduv run --project mcp_server pytest mcp_server/tests: 60 passed (parity test included)PWC_API_URL=http://127.0.0.1:8010/api/v1):CatalogClient().resolve_paper("Attention Is All You Need")→1706.03762in 0.09 s (previously 10 pages then error); LoRA and the"Why Should I Trust You?"title also resolve; a missing title still reportsPaper title not found.pwc benchmark --name geneval --sort Overall:desc,--name "MVTec AD (Zero-Shot Cross-Domain)" --sort "image-level AUROC:desc",--name "D4RL AntMaze-medium-play-v2" --sort "D4RL Normalized Score:desc",--name swe-bench-pro --sort Pass@1:descall render;--name coco-test-dev --sort latency:descstill errors with the available metrics.pwc method --name "mixture of experts"→ Mixture-of-Experts (MoE);pwc task --name "time-series forecasting"→ Time-series forecasting;pwc task --name "Person Re-Identification"still reports not found.ruff checkintroduces no new findings on the touched files.Not addressed here (catalog coverage, not lookup bugs): AlexNet, Random Forest, federated learning, Occupancy Network as methods; aspect-based sentiment analysis, handwritten text recognition, time-series anomaly detection as tasks.
🤖 Generated with Claude Code