chore(developer-experience): commit the hand-rolled eval scenario sources #9936

Activity

1. kyle-sexton

   added the

   Task

   issue type on Oct 25, 2026
2. kyle-sexton

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   admission-deferredReady for the issue lane; work-unblock admits it when queued work is under the queued capReady for the issue lane; work-unblock admits it when queued work is under the queued cap

   on Oct 25, 2026
3. kyle-sexton

   added a parent issue on Oct 25, 2026

   • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649
4. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
5. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Knowable
   Flags: none
   Blocked by: none

   Understood: commit the hand-rolled eval scenario sources (scenario definitions, fixture generator, checklists, labeled samples) from the PR #6668 harness into main. That means scrubbing the 11 files that hold a home path or email, leaving out results and judge output, and recording an identifying-details review in the PR.

   The sources currently live only on one host, so the copy step needs a run there. That is a constraint on the run, not a gap in the request. No likely duplicates found in the open issues.

   _{Automated triage, applied by the lanes App.}
6. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
7. melodic-automation-lanes

   added

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   on Oct 25, 2026
8. kyle-sexton

   marked this as blocking 8 issues on Oct 25, 2026

   • test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9943
   • test(developer-experience): port eval cases S1 and S4 #9953
   • test(developer-experience): port eval cases S2 and S5 #9958
   • test(developer-experience): port eval cases S3 and S6 #9966
   • test(developer-experience): port eval cases S7 and N1 #9979
   • test(developer-experience): port eval cases T1 and T2 #9988
   • test(developer-experience): port eval cases T3 and T4 #9993
   • test(developer-experience): port the stays-quiet eval case #9998
9. melodic-automation-lanes

   added

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   and removed

   admission-deferredReady for the issue lane; work-unblock admits it when queued work is under the queued capReady for the issue lane; work-unblock admits it when queued work is under the queued cap

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   on Oct 25, 2026
10. melodic-automation-lanes commented on Oct 25, 2026

    

    melodic-automation-lanesBot

    on Oct 25, 2026

    Contributor

    More actions

    The work-resolve-issue lane stopped partway through this issue and needs a person.

    Why, in the run's own words:

    The files this issue asks to commit are not on this runner. The issue says the hand-rolled harness (harness/, fixtures/, labels/, lists/, held-out/, judge/) and the spec .work/devx/eval-scenarios-draft.md exist only on melo-desk-001, in the gitignored .work/devx/evals/ of PR ＃6668's worktree, and that the item has to run on that host through /fleet:reach. A search of this checkout found none of it: no scenarios.sh, build-fixtures.py, eval-scenarios-draft.md, or any tarnquill/brackenfold reference, and plugins/developer-experience/evals/ does not exist. Writing these files from scratch would be inventing the source content, and the scrub and identifying-details review need the real files. Run this item on melo-desk-001, or copy the sources somewhere this lane can reach, then re-run.

    What the run says it did not do:

    • Commit the scenario sources under plugins/developer-experience/evals/sources/: the source files are only on melo-desk-001 and this runner can't get to them
    • Scrub home paths and email addresses from the 11 affected files, and do the identifying-details review under .claude/rules/eval-case-transcripts.md: both need the real files
    • Changelog fragment for developer-experience: not written because nothing changed in the plugin
    • PR prep (review, verify, lint, class audits): skipped because there is no diff

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40208853249

    To retry, remove needs-human and add agent-ready again.
11. melodic-automation-lanes

    marked this as blocking on Oct 26, 2026

    • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649