test(developer-experience): port eval cases S3 and S6 #10004

Activity

1. kyle-sexton

   added the

   Task

   issue type on Oct 25, 2026
2. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
3. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Knowable
   Flags: none
   Blocked by: none

   Understood: port eval scenarios S3 and S6 into plugins/developer-experience/evals// (prompt.md, case.yaml, graders/), using deterministic graders where possible and calibrated llm graders, including the rm and compound-shell checks where needed. Done-when is clear and checkable. The "committed sources" for S3 and S6 are not linked in the issue, so the implementer will need to locate them (see #6649). No duplicates found among the open issues.

   _{Automated triage, applied by the lanes App.}
4. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
5. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   admission-deferred

   on Oct 25, 2026
6. kyle-sexton

   added a parent issue on Oct 25, 2026

   • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649
7. kyle-sexton

   marked this as blocked on Oct 25, 2026

   • test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9984
8. melodic-automation-lanes

   added

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   and removed

   admission-deferred

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   on Oct 25, 2026
9. melodic-automation-lanes commented on Oct 25, 2026

   

   melodic-automation-lanesBot

   on Oct 25, 2026

   Contributor

   More actions

   The work-resolve-issue lane stopped partway through this issue and needs a person.

   Why, in the run's own words:

   The source material for S3 and S6 is not in the repository. The issue says to port them 'from the committed sources', but those sources were never committed. Child issue ＃9936 ('commit the hand-rolled eval scenario sources', meant for plugins/developer-experience/evals/sources/) is still open with needs-human and no PR. Per ＃6649 and ＃9936, the only copy is in PR ＃6668's gitignored .work/devx/evals/ tree on the maintainer's host (melo-desk-001): harness/scenarios.sh, the fixture generator, checklists, labeled judge samples, and the spec .work/devx/eval-scenarios-draft.md. Writing S3 and S6 prompts, checklists, and calibrated llm graders from scratch would mean inventing the scenarios and labeled samples. The issue's timeline also shows it blocked by ＃9984 (shared .NET and Python eval fixtures), which may be the fixtures these cases need. Unblock by landing ＃9936, and ＃9984 if these cases need its fixtures, then re-add agent-ready.

   What the run says it did not do:

   • Port S3 and S6 into plugins/developer-experience/evals/<case>/ (prompt.md, case.yaml, graders/): blocked because the scenario sources are not committed (＃9936)
   • validate-cases.py on plugins/developer-experience/evals: not run because there are no cases to validate
   • Local fresh-context review of the diff: skipped because the diff is empty (review-venue.txt absent)
   • ci-status green on the PR: left to the PR's CI; no PR is warranted with no changes

   Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40209625731

   To retry, remove needs-human and add agent-ready again.
10. kyle-sexton commented on Oct 25, 2026

    

    kyle-sexton

    on Oct 25, 2026

    MemberAuthor

    More actions

    Duplicate split of #6649 by a second Backlog HITL writer. To reopen: reopen this issue and say what the kept child misses.

    Kept child: #9966.
11. kyle-sexton

    closed this as not plannedon Oct 25, 2026