test(developer-experience): port eval cases S7 and N1 #10006

Activity

1. kyle-sexton

   added the

   Task

   issue type on Oct 25, 2026
2. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
3. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Small
   Complexity: Medium
   Uncertainty: Knowable
   Flags: none
   Blocked by: none

   Understood: port eval scenarios S7 and N1 into plugins/developer-experience/evals/ as case directories (prompt.md, case.yaml, graders/), using deterministic graders where possible and calibrated llm graders otherwise, including the rm and compound-shell checks. Done when validate-cases.py exits 0, both case.yaml files are tracked, and ci-status is green. The scope is clear. The only open point is locating the committed S7 and N1 sources, referenced via #6649. I found no likely duplicates.

   _{Automated triage, applied by the lanes App.}
4. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
5. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   admission-deferred

   on Oct 25, 2026
6. kyle-sexton

   added a parent issue on Oct 25, 2026

   • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649
7. kyle-sexton

   marked this as blocked on Oct 25, 2026

   • test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9984
8. melodic-automation-lanes

   added

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   and removed

   admission-deferred

   on Oct 25, 2026
9. melodic-automation-lanes commented on Oct 25, 2026

   

   melodic-automation-lanesBot

   on Oct 25, 2026

   Contributor

   More actions

   The token broker refused the work-resolve-issue run that agent-ready started because the lanes kill switch was on, so it did no work. work-unblock removed and added agent-ready back to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last added agent-ready or a run last reported, work-unblock waits longer before each one, up to 24 hours.
10. melodic-automation-lanes

    added

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    and removed

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    on Oct 25, 2026
11. melodic-automation-lanes commented on Oct 25, 2026

    

    melodic-automation-lanesBot

    on Oct 25, 2026

    Contributor

    More actions

    The work-resolve-issue lane did not start this issue: it needs a person to make it clearer first.

    Why, in the run's own words:

    The scenario definitions for S7 and N1 are not in the repository, so there is nothing to port and no way to tell what either case should test. The issue says to port them 'from the committed sources', but plugins/developer-experience/evals/ does not exist on main (no sources/ directory, no S7 or N1 text anywhere in the checkout). Parent ＃6649 says the scenarios live only in the gitignored .work/devx/evals/ harness on the maintainer's host (PR ＃6668's worktree). Committing them to plugins/developer-experience/evals/sources/ is ＃9936, which is open and labeled needs-human. The sibling issue ＃9979 (the same S7/N1 port) lists ＃9936 and ＃9943 (shared fixtures) as blockers, and this issue does not. Before this can run: a person lands ＃9936 (and ＃9943 if the cases need the shared fixtures), or pastes the S7 and N1 scenario definitions, checklists and labeled samples into this issue. Also confirm whether ＃10006 duplicates ＃9979 and which one should stay open.

    What the run says it did not do:

    • Port S7 and N1 into plugins/developer-experience/evals/<case>/: blocked because their scenario sources are not committed (＃9936 open)
    • python3 plugins/evals/skills/validate/scripts/validate-cases.py plugins/developer-experience/evals: not run, because no cases were written and this lane cannot run an interpreter
    • ci-status green on the PR: left to the PR's CI, and no PR results from a pushback

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40223407414

    To retry, answer the question here, remove needs-human and add agent-ready again.
12. kyle-sexton commented on Oct 25, 2026

    

    kyle-sexton

    on Oct 25, 2026

    MemberAuthor

    More actions

    Duplicate split of #6649 by a second Backlog HITL writer. To reopen: reopen this issue and say what the kept child misses.

    Kept child: #9979.
13. kyle-sexton

    closed this as not plannedon Oct 25, 2026