test(developer-experience): port eval cases T1 and T2 #10007

Activity

1. kyle-sexton

   added the

   Task

   issue type on Oct 25, 2026
2. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
3. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Knowable
   Flags: none
   Blocked by: none

   Understood: port eval scenarios T1 and T2 from the committed sources into plugins/developer-experience/evals/ as cases, each with prompt.md, case.yaml and graders. Graders should be deterministic where possible, with llm graders calibrated on labeled samples. The rm and compound-shell checks are included where a case needs them. The done-when items are checkable. The main thing to confirm is where the committed T1/T2 sources live. No duplicates found among the listed open issues.

   _{Automated triage, applied by the lanes App.}
4. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
5. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   admission-deferredReady for the issue lane; work-unblock admits it when queued work is under the queued capReady for the issue lane; work-unblock admits it when queued work is under the queued cap

   on Oct 25, 2026
6. kyle-sexton

   added a parent issue on Oct 25, 2026

   • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649
7. kyle-sexton

   marked this as blocked on Oct 25, 2026

   • test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9984
8. melodic-automation-lanes

   added

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   and removed

   admission-deferredReady for the issue lane; work-unblock admits it when queued work is under the queued capReady for the issue lane; work-unblock admits it when queued work is under the queued cap

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   on Oct 25, 2026
9. melodic-automation-lanes commented on Oct 25, 2026

   

   melodic-automation-lanesBot

   on Oct 25, 2026

   Contributor

   More actions

   The work-resolve-issue lane did not start this issue: it needs a person to make it clearer first.

   Why, in the run's own words:

   This run can't port T1 and T2 because their scenario text isn't anywhere it can read. The issue says to port them 'from the committed sources', but nothing is committed. plugins/developer-experience/evals/ (and evals/sources/) does not exist on main. No file in the repository defines a developer-experience T1 or T2. The issue that was supposed to commit the sources, ＃9936, is still open with no PR; its lane comment says the sources exist only in the gitignored .work/devx/evals/ folder of PR ＃6668's worktree on melo-desk-001. Neither ＃6649 nor duplicate ＃9988 contains the scenario text either. The question for a person: will you land ＃9936 (commit the hand-rolled sources to plugins/developer-experience/evals/sources/) or paste the T1 and T2 prompts, expected behavior and labeled samples into this issue? Also consider recording ＃9936 as a blocker of this issue, as ＃9988 does, so the lane does not admit it again before the sources land.

   What the run says it did not do:

   • No eval cases written: the T1 and T2 sources are not in the repository (blocked on ＃9936)
   • validate-cases.py was not run: no plugins/developer-experience/evals directory exists to validate
   • ci-status on the PR is left to CI; no PR is opened for a pushback

   Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40209631918

   To retry, answer the question here, remove needs-human and add agent-ready again.
10. kyle-sexton commented on Oct 25, 2026

    

    kyle-sexton

    on Oct 25, 2026

    MemberAuthor

    More actions

    Duplicate split of #6649 by a second Backlog HITL writer. To reopen: reopen this issue and say what the kept child misses.

    Kept child: #9988.
11. kyle-sexton

    closed this as not plannedon Oct 25, 2026