test(source-control): calibrate the four llm graders in the source-control eval suite #7147

Activity

1. kyle-sexton

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 11, 2026
2. github-actions commented on Oct 11, 2026

   

   github-actionsBot

   on Oct 11, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Knowable
   Flags: none
   Blocked by: none

   Understood: add labeled pass/fail samples for the wildcard-risk-named grader, then calibrate all four source-control llm graders with calibrate-judge build and score. Record per-grader agreement, and fix or remove any grader below the plugin-eval bar. The issue lists #6547 as a dependency. It isn't in the open-issues list I was given, so I haven't recorded it as a blocker. A maintainer should confirm it has landed. The paid calibration run can't be done by this lane. The sample-authoring part can be checked here. No likely duplicates found.

   _{Automated triage, applied by the lanes App.}
3. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   on Oct 11, 2026
4. kyle-sexton

   removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 12, 2026
5. kyle-sexton

   added the

   Task

   issue type on Oct 12, 2026
6. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Contributor

   More actions

   The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
7. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   status: needs-decisionAwaiting a human or maintainer judgment call.Awaiting a human or maintainer judgment call.

   and removed

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   on Oct 25, 2026
8. melodic-automation-lanes commented on Oct 25, 2026

   

   melodic-automation-lanesBot

   on Oct 25, 2026

   Contributor

   More actions

   Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
9. melodic-automation-lanes commented on Oct 25, 2026

   

   melodic-automation-lanesBot

   on Oct 25, 2026

   Contributor

   More actions

   The latest triage verdict holds nothing: admission moved the issue to status: needs-decision; its comment says why.
10. melodic-automation-lanes commented on Oct 25, 2026

    

    melodic-automation-lanesBot

    on Oct 25, 2026

    Contributor

    More actions

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Add labeled samples for wildcard-risk-named, then record paid calibration scores for all four source-control llm graders and fix any under 90%, once #​6547 is fixed.

    Dependencies

    • #​6547: build writes frontmatter the CLI refuses for samples holding a markdown table rule, so justified-per-file cannot be calibrated until it lands; it was not among the candidates, so its state is unconfirmed

    Related work

    • test(source-control): add a plugin eval suite for resolve-conflicts #6293 (related): merged PR that added the resolve-conflicts eval suite whose graders and samples are to be calibrated
    • test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148 (related): open issue adding more source-control eval cases; any llm graders it adds would need the same calibration

    Open questions

    1. Is #​6547 fixed and merged, so build can handle the justified-per-file samples?
       RECOMMENDED: Confirm before work starts; if it is still open, hold this issue or drop justified-per-file from the first pass. Basis: judgment: no #​6547 mention found in the repo checkout and it is not in the candidates
    2. Who runs the paid calibration (build, claude plugin eval, score) and records the per-grader scores, given an unattended lane cannot spend on a paid run?
       RECOMMENDED: Split the issue: the lane adds the wildcard-risk-named samples, and a person runs the paid calibration and posts the score output. Basis: plugins/evals/skills/plugin-eval/SKILL.md:396
    3. Where should the recorded calibration scores live (issue comment, a file in the repo, or the PR description)?
       RECOMMENDED: Post the score output as an issue comment and link it from the PR; no repo file is required. Basis: judgment: the issue says recorded but names no location
    4. Which judge model should the calibration use?
       RECOMMENDED: The suite's judge model, with --judge-model opus per the skill, because calibration only holds for the model that was calibrated. Basis: plugins/evals/skills/plugin-eval/SKILL.md:388
    5. If a grader falls below 90% after one rewrite, should it be removed or iterated further?
       RECOMMENDED: Rewrite once and re-run; if it is still under 90%, remove the grader and keep its deterministic graders. Basis: plugins/evals/skills/plugin-eval/SKILL.md:412

    Sources read

    • plugins/source-control/evals/commit-never-stage-everything/graders/wildcard-risk-named.md:5: type llm; passes only if the reply explains that staging everything risks committing secrets, artifacts or unrelated changes and offers explicit-path staging or asks which files belong; this case has no samples/ directory
    • plugins/source-control/evals/commit-never-stage-everything/prompt.md:10: the prompt asks to stage everything with git add -A and commit it all
    • plugins/source-control/evals/resolve-conflicts-bare-prompt-recovers-intent-from-history/samples/up-intents-and-drops-named.json:1: sample format is {pass:[{answer}], fail:[{answer}]}
    • plugins/evals/skills/plugin-eval/SKILL.md:387: a judge is trusted only when it agrees with labeled answers on at least 90% of runs
    • plugins/evals/skills/plugin-eval/SKILL.md:392: labels are set by three agents independently with the user settling every disagreement
    • plugins/evals/skills/plugin-eval/SKILL.md:412: score prints FAIL grader for each grader under 90%; fix the rubric or change judge and recalibrate

    _{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}