Repository navigation
test(multi-agent): routing eval cases for the review role #10012
Description
Refs: #6901
What to do
Part of the paired model-routing eval (#6901): the same tasks from this repository run on Haiku 5.5 at high and xhigh, Haiku 5.5 with an Opus 5.5 advisor (Anthropic's Haiku coding prompt), Sonnet 5.5 at low, medium and high, and Opus 5.5 at low and medium. Graders score pass rate or verified findings; the judge is outside the candidate set, or Opus with its self-preference noted. State no prices. This slice writes the review cases (code review and verification: find seeded defects in a diff and refute false findings) under plugins/multi-agent/evals/routing/review/, with the arm configuration each setup needs and validates them statically; the paid runs belong to the run slice.
Done when
-
python3 plugins/evals/skills/validate/scripts/validate-cases.py plugins/multi-agent/evals/routing/reviewexits 0. - Each case's grader is deterministic, or an
llmgrader calibrated against labeled samples, with the calibration in the case directory. -
ci-statusis green on the PR.
Split by the owner's HITL session (2026-10-24, owner decisions C12 and Q93-Q95). Intake triage sets its fields.
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: add routing eval cases for the review role (find seeded defects in a diff, refute false findings) under plugins/multi-agent/evals/routing/review/, with the arm configuration each setup needs. The cases must pass the static validator, use deterministic graders or calibrated llm graders, and CI must be green. The paid runs are out of scope. Nothing is missing that blocks starting. No likely duplicates found among the open issues; the parent is #6901.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 25, 2026
The token broker refused the work-resolve-issue run that
agent-readystarted because the lanes kill switch was on, so it did no work. work-unblock removed and addedagent-readyback to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last addedagent-readyor a run last reported, work-unblock waits longer before each one, up to 24 hours.melodic-automation-lanes commentedon Oct 25, 2026
The work-resolve-issue lane stopped on its funding (
funding: no credit or a disabled key), not on this issue. It keepsagent-ready; once the lane is funded again, a person removes and addsagent-readyto restart it.Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40232710829