Repository navigation
perf(multi-agent): run the paired routing eval and confirm or change the role defaults #10022
Description
Refs: #6901
What to do
Part of the paired model-routing eval (#6901): the same tasks from this repository run on Haiku 5.5 at high and xhigh, Haiku 5.5 with an Opus 5.5 advisor (Anthropic's Haiku coding prompt), Sonnet 5.5 at low, medium and high, and Opus 5.5 at low and medium. Graders score pass rate or verified findings; the judge is outside the candidate set, or Opus with its self-preference noted. State no prices. Run the three role suites across all setups with the same --runs, judge and threshold. Spend is pre-approved within prepaid credit: estimate first, run under --max-cost-usd, and stop with needs-human and the estimate if it would exceed the prepaid balance. Record pass rate or verified findings, tokens per completed task (input, output, thinking), advisor call rate and wall time per setup, then confirm or change plugins/multi-agent/reference/defaults.yaml. A role moves to Haiku only when it matches the current model's accuracy at no higher token cost per completed task.
Done when
- The PR body holds the per-role, per-setup table of pass rate, tokens per completed task, advisor call rate and wall time, with no prices.
-
git diff origin/main -- plugins/multi-agent/reference/defaults.yamlchanges a role only where the table shows the move rule holds, or the PR states that every default is confirmed. - The multi-agent plugin's existing tests under
plugins/multi-agent/exit 0. -
ci-statusis green on the PR.
Split by the owner's HITL session (2026-10-24, owner decisions C12 and Q93-Q95). Intake triage sets its fields.
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: run the paired routing eval from #6901 across the three role suites and the listed model setups, then record the per-role, per-setup table in the PR. Confirm or change the role defaults in plugins/multi-agent/reference/defaults.yaml according to the stated move rule. The spend is pre-approved from prepaid credit. The run should stop with an estimate if that would exceed the balance. Nothing appears to be missing. The eval run itself may need an attended or paid-eval environment, so only the lane-checkable done-when items can be judged here. No likely duplicates found.
Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 25, 2026
Needs a run of
claude(eval, probe or /usage), which the issue lane cannot do (pr-pipeline.yaml:289-314). The owner's HITL session runs it on the operator machine; no owner action needed.