test(developer-experience): port eval cases S2 and S5 #9997

Activity

1. kyle-sexton

   added the

   Task

   issue type on Oct 25, 2026
2. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
3. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Knowable
   Flags: none
   Blocked by: none

   Understood: port eval scenarios S2 and S5 into plugins/developer-experience/evals// (prompt.md, case.yaml, graders/), using deterministic graders where possible and calibrated llm graders, including rm and compound-shell checks where needed. The done-when items are the validate-cases.py run, a case.yaml for each of S2 and S5, and green ci-status. The issue refers to the "committed sources" without a path, so the implementer will need to find them (see #6649). I found no duplicates among the open issues.

   _{Automated triage, applied by the lanes App.}
4. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
5. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   admission-deferred

   on Oct 25, 2026
6. kyle-sexton

   added a parent issue on Oct 25, 2026

   • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649
7. kyle-sexton

   marked this as blocked on Oct 25, 2026

   • test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9984
8. melodic-automation-lanes

   added

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   and removed

   admission-deferred

   on Oct 25, 2026
9. kyle-sexton commented on Oct 25, 2026

   

   kyle-sexton

   on Oct 25, 2026

   MemberAuthor

   More actions

   Duplicate split of #6649 by a second Backlog HITL writer. To reopen: reopen this issue and say what the kept child misses.

   Kept child: #9958.
10. kyle-sexton

    closed this as not plannedon Oct 25, 2026