Repository navigation
test(multi-agent): routing eval cases for the retrieval role #10009
Description
Refs: #6901
What to do
Part of the paired model-routing eval (#6901): the same tasks from this repository run on Haiku 5.5 at high and xhigh, Haiku 5.5 with an Opus 5.5 advisor (Anthropic's Haiku coding prompt), Sonnet 5.5 at low, medium and high, and Opus 5.5 at low and medium. Graders score pass rate or verified findings; the judge is outside the candidate set, or Opus with its self-preference noted. State no prices. This slice writes the retrieval cases (codebase retrieval: locate code, trace callers, answer where-does-X-happen questions) under plugins/multi-agent/evals/routing/retrieval/, with the arm configuration each setup needs and validates them statically; the paid runs belong to the run slice.
Done when
-
python3 plugins/evals/skills/validate/scripts/validate-cases.py plugins/multi-agent/evals/routing/retrievalexits 0. - Each case's grader is deterministic, or an
llmgrader calibrated against labeled samples, with the calibration in the case directory. -
ci-statusis green on the PR.
Split by the owner's HITL session (2026-10-24, owner decisions C12 and Q93-Q95). Intake triage sets its fields.
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Understood: this slice writes the retrieval-role routing eval cases (locate code, trace callers, where-does-X-happen) under
plugins/multi-agent/evals/routing/retrieval/. It includes the per-arm configuration for each model setup and static validation withvalidate-cases.py. The paid runs belong to a separate slice. The done-when items are specific and checkable, and nothing is missing. No likely duplicates found among open issues; it is part of #6901.Automated triage, applied by the lanes App.