Repository navigation
feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879
Description
Problem
The evals plugin's eval suite reached a VALID run with the plugin scoring 0.99 and 0.31 without (delta +0.68, 95% interval +0.52 to +0.83; 29 of 31 cases at 1.00). The work left gaps in the suite's judge calibration, and a wider conversion task, none of them blocking.
Follow-ups
- Two judges fail correct answers that their calibration samples don't cover.
noise-report-before-posting/gate-before-numbersfailed correct answers in four runs. Every one used the shape "run run-validity.py first; on INVALID, post the INVALID line and its reasons". Candidate causes: an absolute script path, a second gate on the noise report, or a trailingpartialcheck.noise-before-gain/not-establishedfailed an answer that opens with "the numbers point the right way", which its rubric names as allowed.- Add must-pass samples of these shapes, then recalibrate.
- Three must-fail samples are never tested, because the reply-only agent will not reproduce them word for word:
measurable-criterionfour-properties sample 3: the model drops its "launch a medical bot with 'answers are safe'" paragraph as unsafe advice.worktree-guard-stopno-way-around sample 4, andzeros-after-usage-limitcheck-and-rerun sample 3: both get paraphrased.- Rewrite each so it still fails for one reason, with wording the model will repeat, for example a retraction that isn't safety-sensitive.
measurable-criterionmust-fail sample 2 now misses two properties. It was built to miss only the trial set. Since the Achievable rule changed, its invented "2% measured last month" baseline fails too, so it no longer tests one gap on its own.- Convert the other plugins' evals (V7). Split to feat(evals): convert other plugins' evals.json files to claude plugin eval suites #8635.
This issue carries follow-ups 1-3 as free sample edits in one plugin's suite. The paid calibration and suite runs are split to the needs-human child (Attended Build).
Acceptance criteria
-
noise-report-before-postinggate-before-numbers andnoise-before-gainnot-established each gain must-pass samples inplugins/evals/evals/<case>/samples/<grader>.json, covering the shapes listed in follow-up 1. -
The three named must-fail samples are rewritten so each misses exactly one rubric property, with wording that is not safety-sensitive and that the model reproduces:
measurable-criterionsample 3;pe-worktree-guard-stopsample 4;- the check-and-rerun sample 3 of
zeros-after-usage-limit. No case by that name appears inplugins/evalson main, so the PR names the case it was renamed to, or records that the sample is gone.
The PR states each fix's one missing property.
-
measurable-criterionmust-fail sample 2 again misses only the trial-set property. -
The PR's CI runs the evals plugin tests and they pass.
-
python3 plugins/evals/skills/plugin-eval/scripts/calibrate-judge.py build --suite plugins/evals/evals --out <tmp>succeeds, andplugins/evals/skills/validate/scripts/validate-cases.pyreports no FAIL. CI runs neither and neither is a lane check, so the merging reviewer runs both. -
An evals fragment exists (
scripts/new-changelog-fragment.sh evals patch), andscripts/check-changelog-fragments.sh --checkpasses.
Context
The loop that produced these, and the reasoning behind each, is in the pull request that adds the evals suite and validity gate (link to follow).
Activity
feat: adopt eval-design and hillclimbing guidance across six plugins (#…
kyle-sexton commentedon Oct 4, 2026
This was generated by AI during triage.
Agent Brief
Type: Feature
Summary: Close the evals plugin suite's judge-calibration gaps, and convert other plugins' skill-creator evals to
claude plugin evalsuites.Current behavior:
The evals plugin's own
claude plugin evalsuite passes its validity gate, but somellmjudges have uncovered shapes: thenoise-report-before-postinggate-before-numbers judge and thenoise-before-gainnot-established judge fail correct answers their must-pass samples do not cover; three must-fail samples (measurable-criterionfour-properties sample 3,worktree-guard-stopno-way-around sample 4,zeros-after-usage-limitcheck-and-rerun sample 3) are never exercised because the model will not repeat them word for word; andmeasurable-criterionmust-fail sample 2 now misses two properties instead of one. Most plugins still carry only skill-creatorevals.jsonfiles.Desired behavior:
Acceptance criteria:
llmgrader in the evals suite, including the new samplesOut of scope:
Routing: human-gated (needs decision)
The issue holds two different sizes of work. Follow-ups 1 to 3 are scoped calibration fixes inside one plugin's suite (C3). Follow-up 4, converting other plugins' evals, is a cross-plugin migration across dozens of plugins (C4), and its numbers are stale: code-tidying, discovery, education, improvement and planning now also have
case.yamlsuites, not only source-control, and the repository holds 341evals.jsonfiles.Question for the owner: split follow-up 4 into its own item (decomposed per plugin), and let this issue cover follow-ups 1 to 3 only? Recommended: yes. Once split, this issue can be routed agent-ready as
work-class: scoped; until then it carries the structural class of its largest part.Basis: verified, PR #5895 (merged 2026-10-03) names this issue as a follow-up; the named cases and samples exist under plugins/evals/evals/ unchanged since #5895;
find plugins -path '*/evals/*' -name case.yamllists suites in six plugins;find plugins -name evals.jsoncounts 341. Searcheddocs/out-of-scope/: no matching concept.kyle-sexton commentedon Oct 13, 2026
Decision recorded. The owner approved applying independently verified recommendations on 2026-10-12.
Basis: feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879 triage comment 2026-10-04 (six plugins have case.yaml suites, 341 evals.json files); gh issue search this run found no existing split item
Follow-ups 1-3 are judge-calibration fixes in the evals plugin's own suite; follow-up 4 converts other plugins' evals.json files, a cross-plugin migration. Triage recommends splitting 4 out so 1-3 can go agent-ready as scoped work; no split issue exists yet.
Verified by an independent fresh-context check (confirmed, risk low): gh issue view 5879: triage recommends the split, no owner answer; git show --stat for #7000 and #6269: neither touched the named cases (noise-report-before-posting, noise-before-gain, measurable-criterion, worktree-guard-stop, zeros-after-usage-limit); assumption true; gh issue list search: no split item exists (#6405 is a CI lane, not the conversion)
Admission to an agent lane stays with the work-class rules and owner ratification.
kyle-sexton commentedon Oct 13, 2026
Owner batch-ratified this admission in an attended session on 2026-10-12: work-class: scoped. Rule: class C1-C3, no trust surface, the independent check confirmed the recorded decision, and no owner step remains.
kyle-sexton commentedon Oct 13, 2026
Correction: this issue already carried work-class: structural, which batch ratification never admits. The admission above is reverted; it stays escalated for an attended decision.