test(developer-experience): port eval cases T3 and T4 #9993

Activity

1. kyle-sexton

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   admission-deferredReady for the issue lane; work-unblock admits it when queued work is under the queued capReady for the issue lane; work-unblock admits it when queued work is under the queued cap

   on Oct 25, 2026
2. kyle-sexton

   added the

   Task

   issue type on Oct 25, 2026
3. kyle-sexton

   added a parent issue on Oct 25, 2026

   • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649
4. kyle-sexton

   marked this as blocked by 2 issues on Oct 25, 2026

   • chore(developer-experience): commit the hand-rolled eval scenario sources #9936
   • test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9943
5. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Knowable
   Flags: none
   Blocked by: #9936, #9943

   Understood: port the hand-rolled eval scenarios T3 and T4 into plugins/developer-experience/evals/ as one case each on the shared fixtures. Use deterministic graders where possible and check each llm grader against the labeled samples. The T4 #2 part waits on a person relabeling its samples. The rest does not wait.

   The scenario sources (#9936) and the shared fixtures (#9943) are both still open, so this work depends on them. Sibling port issues, such as the T1/T2 and S-series ones, cover other cases and don't look like duplicates.

   _{Automated triage, applied by the lanes App.}
6. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   on Oct 25, 2026
7. melodic-automation-lanes commented on Oct 25, 2026

   

   melodic-automation-lanesBot

   on Oct 25, 2026

   Contributor

   More actions

   The triage verdict left this issue at status: needs-decision:

   • held by needs-human
   • no lanes App verdict comment shows whether an earlier verdict was flagged
8. melodic-automation-lanes

   added

   status: needs-decisionAwaiting a human or maintainer judgment call.Awaiting a human or maintainer judgment call.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   and removed

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   on Oct 25, 2026
9. kyle-sexton

   marked this as blocking on Oct 25, 2026

   • test(developer-experience): run the plugin-eval suite on the eval dispatch lane and record the delta #10005
10. kyle-sexton

    removed

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    on Oct 25, 2026
11. kyle-sexton commented on Oct 25, 2026

    

    kyle-sexton

    on Oct 25, 2026

    MemberAuthor

    More actions

    This was generated by AI.

    Owner decision Q114 (2026-10-25, "go w/ recommendations"): T4 #2 no longer waits on a person relabeling.

    1. When porting T4 fix: correct plugin-system inaccuracies and harden references #2, first try replacing the judge item with a deterministic grader (for example a regex that the reply reports the old .claude/ location).
    2. If the item needs judgment, three blind agents relabel the 3 samples, given the rubric and the owner's 2026-10-09 comment on feat(developer-experience): claude plugin eval suite in GitHub Actions #6649, not the old labels or judge verdicts. A label is accepted only when all three agree and it differs from the old literal label. Any other sample goes to the owner as one intent question.
    3. Then re-calibrate to >= 90% on the subscription login (never an API key).

    Basis: the three-agent rule at plugins/evals/skills/plugin-eval/SKILL.md:391-393; the relabel precedent in docs/adr/0084. The owner accepts the validity risk against Anthropic's eval-audit guidance, which calls for human-labelled samples.

    Dependency: this runs once #9936 and #9943 land. The issue body is updated to match; needs-human is removed.
12. melodic-automation-lanes

    marked this as blocking on Oct 26, 2026

    • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649