Repository navigation
test(developer-experience): port eval cases S7 and N1 #9979
Description
Refs: #6649
Parent
Depends on
- chore(developer-experience): commit the hand-rolled eval scenario sources #9936
- test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9943
Problem
#6649 moves the developer-experience measurement onto claude plugin eval. Cases S7 and N1 exist only
as hand-rolled scenarios.
Fix
Port S7 and N1 from the committed scenario sources into plugins/developer-experience/evals/, one case
each, on the shared fixtures. Use deterministic graders where the scenario's D and T items allow; check
each llm grader against the committed labeled samples.
Also
- Case layout:
plugins/developer-experience/evals/<case>/(prompt.md,case.yaml,graders/). - Include the
rmand compound-shell checks where a case needs them.
Done when
- Each case loads and passes
/evals:validate(no paid run). -
scripts/check-changelog-fragments.sh --check-required origin/mainexits 0.
Filed by AI from the owner-approved split of #6649 (2026-10-18), after the scenario sources were found on melo-desk-001 (2026-10-25).
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: #9936, #9943
Understood: port the hand-rolled scenarios S7 and N1 into
plugins/developer-experience/evals/as oneclaude plugin evalcase each, on the shared fixtures. Prefer deterministic graders, and check anyllmgrader against the committed labeled samples. Done when both cases pass/evals:validateand the changelog-fragment check exits 0.The issue lists #9936 (commit the scenario sources) and #9943 (shared fixtures) as dependencies, and both are still open. Sibling ports are #9953, #9958 and #9966. #9977 has the same title as #9936 and may be a duplicate of it.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 25, 2026
Not admitted to the issue lane yet: blocker #9936 is not closed by a listed human or a merge; blocker #9943 is not closed by a listed human or a merge.