Repository navigation
test(developer-experience): port eval cases S3 and S6 #9966
Description
Refs: #6649
Parent
Depends on
- chore(developer-experience): commit the hand-rolled eval scenario sources #9936
- test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9943
Problem
#6649 moves the developer-experience measurement onto claude plugin eval. Cases S3 and S6 exist only
as hand-rolled scenarios.
Fix
Port S3 and S6 from the committed scenario sources into plugins/developer-experience/evals/, one case
each, on the shared fixtures. Use deterministic graders where the scenario's D and T items allow; check
each llm grader against the committed labeled samples.
Also
- Case layout:
plugins/developer-experience/evals/<case>/(prompt.md,case.yaml,graders/). - Include the
rmand compound-shell checks where a case needs them.
Done when
- Each case loads and passes
/evals:validate(no paid run). -
scripts/check-changelog-fragments.sh --check-required origin/mainexits 0.
Filed by AI from the owner-approved split of #6649 (2026-10-18), after the scenario sources were found on melo-desk-001 (2026-10-25).
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: #9936, #9943
Understood: port the hand-rolled eval scenarios S3 and S6 into
plugins/developer-experience/evals/as oneclaude plugin evalcase each. The cases use the shared fixtures and deterministic graders where possible. Eachllmgrader is checked against the committed labeled samples. Done when each case passes/evals:validateand the changelog-fragment check exits 0.The scenario sources (#9936) and the shared fixtures (#9943) are both still open, so this waits on them. No duplicates found.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 25, 2026
Not admitted to the issue lane yet: blocker #9936 is not closed by a listed human or a merge; blocker #9943 is not closed by a listed human or a merge.