Repository navigation
test(multi-agent): routing eval cases for the mechanical-edit role #10017
Description
Refs: #6901
What to do
Part of the paired model-routing eval (#6901): the same tasks from this repository run on Haiku 5.5 at high and xhigh, Haiku 5.5 with an Opus 5.5 advisor (Anthropic's Haiku coding prompt), Sonnet 5.5 at low, medium and high, and Opus 5.5 at low and medium. Graders score pass rate or verified findings; the judge is outside the candidate set, or Opus with its self-preference noted. State no prices. This slice writes the mechanical-edit cases (bounded mechanical edits: a scoped rename, pointer repoint or format change with a checkable result) under plugins/multi-agent/evals/routing/mechanical-edit/, with the arm configuration each setup needs and validates them statically; the paid runs belong to the run slice.
Done when
-
python3 plugins/evals/skills/validate/scripts/validate-cases.py plugins/multi-agent/evals/routing/mechanical-editexits 0. - Each case's grader is deterministic, or an
llmgrader calibrated against labeled samples, with the calibration in the case directory. -
ci-statusis green on the PR.
Split by the owner's HITL session (2026-10-24, owner decisions C12 and Q93-Q95). Intake triage sets its fields.
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Understood: add static routing-eval cases for the mechanical-edit role (scoped rename, pointer repoint, format change) under
plugins/multi-agent/evals/routing/mechanical-edit/. Each case gets its arm configuration and a deterministic grader, or a calibratedllmgrader. Validation runs throughvalidate-cases.py. The paid runs belong to a separate slice. Nothing is missing. Related sibling slices: #10012 (review role) and #10009 (retrieval role). They look like parallel work, not duplicates.Automated triage, applied by the lanes App.