test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9984

Activity

1. kyle-sexton

   added the

   Task

   issue type on Oct 25, 2026
2. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
3. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: scoped
   Basis: automated intake triage; the lanes App applied this class.

   Size: Medium
   Complexity: Medium
   Uncertainty: Known
   Flags: none
   Blocked by: none

   Understood: build the shared .NET and Python fixture repositories under plugins/developer-experience/evals/fixtures/ from committed sources, with every fixture file named by a scaffold or seed script. The done-when items are validate-cases.py exiting 0, the orphaned-fixtures check passing, and ci-status green. The scope looks clear from the issue and no duplicates turned up among the open issues. Refs #6649.

   _{Automated triage, applied by the lanes App.}
4. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
5. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   admission-deferredReady for the issue lane; work-unblock admits it when queued work is under the queued capReady for the issue lane; work-unblock admits it when queued work is under the queued cap

   on Oct 25, 2026
6. kyle-sexton

   added a parent issue on Oct 25, 2026

   • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649
7. kyle-sexton

   marked this as blocked on Oct 25, 2026

   • chore(developer-experience): commit the hand-rolled eval scenario sources #9977
8. kyle-sexton

   marked this as blocking on Oct 25, 2026

   • test(developer-experience): port eval cases S1 and S4 #9992
9. melodic-automation-lanes

   removed

   admission-deferredReady for the issue lane; work-unblock admits it when queued work is under the queued capReady for the issue lane; work-unblock admits it when queued work is under the queued cap

   on Oct 25, 2026
10. kyle-sexton

    marked this as blocking 4 issues on Oct 25, 2026

    • test(developer-experience): port eval cases S2 and S5 #9997
    • test(developer-experience): port eval cases S3 and S6 #10004
    • test(developer-experience): port eval cases S7 and N1 #10006
    • test(developer-experience): port eval cases T1 and T2 #10007
11. melodic-automation-lanes

    added

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    and removed

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    on Oct 25, 2026
12. melodic-automation-lanes commented on Oct 25, 2026

    

    melodic-automation-lanesBot

    on Oct 25, 2026

    Contributor

    More actions

    The work-resolve-issue lane did not start this issue: it needs a person to make it clearer first.

    Why, in the run's own words:

    The issue says to build the .NET and Python fixture repositories 'from the committed sources', but those sources are not in the repository. plugins/developer-experience/evals/ does not exist, and nothing in the checkout names a developer-experience .NET or Python fixture. The issue that commits them, ＃9936 ('commit the hand-rolled eval scenario sources' under plugins/developer-experience/evals/sources/), is still open, has no linked PR, and says the sources live only on the host that ran the PR ＃6668 eval. Its duplicate, ＃9977, was closed as not planned. There is a second blocker: check-orphaned-fixtures.sh counts a fixture as consumed only when some evals/*/scaffold.sh (or a seed script a scaffold runs) names it, and no developer-experience case exists yet, because the porting children ＃9992, ＃9997, ＃10004, ＃10006, ＃10007, ＃9993 and ＃9998 are still to land. So the 'orphaned-fixtures passes' criterion cannot be met by fixtures alone. A person needs to decide: (a) block ＃9984 on ＃9936 (add 'Depends on: ＃9936') and requeue it once the sources are committed, or (b) say what the .NET and Python fixture repositories should contain without those sources (files, project layout, the states each case needs), and which case scaffolds should name them in this PR so the orphaned-fixtures check passes.

    What the run says it did not do:

    • No fixtures built: the committed sources the issue names do not exist yet (＃9936 is open)
    • validate-cases.py and the orphaned-fixtures check were not run on a developer-experience evals tree, because none exists
    • ci-status green on the PR: left to the PR's CI; no PR results from this run

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40207234292

    To retry, answer the question here, remove needs-human and add agent-ready again.
13. kyle-sexton commented on Oct 25, 2026

    

    kyle-sexton

    on Oct 25, 2026

    MemberAuthor

    More actions

    Duplicate split of #6649 by a second Backlog HITL writer. To reopen: reopen this issue and say what the kept child misses.

    Kept child: #9943.
14. kyle-sexton

    closed this as not plannedon Oct 25, 2026