Repository navigation
test(developer-experience): port eval cases S2 and S5 #9958
Description
Refs: #6649
Parent
Depends on
- chore(developer-experience): commit the hand-rolled eval scenario sources #9936
- test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9943
Problem
#6649 moves the developer-experience measurement onto claude plugin eval. Cases S2 and S5 exist only
as hand-rolled scenarios.
Fix
Port S2 and S5 from the committed scenario sources into plugins/developer-experience/evals/, one case
each, on the shared fixtures. Use deterministic graders where the scenario's D and T items allow; check
each llm grader against the committed labeled samples.
Also
- Case layout:
plugins/developer-experience/evals/<case>/(prompt.md,case.yaml,graders/). - Include the
rmand compound-shell checks where a case needs them.
Done when
- Each case loads and passes
/evals:validate(no paid run). -
scripts/check-changelog-fragments.sh --check-required origin/mainexits 0.
Filed by AI from the owner-approved split of #6649 (2026-10-18), after the scenario sources were found on melo-desk-001 (2026-10-25).
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: #9936, #9943
Understood: port hand-rolled scenarios S2 and S5 into
plugins/developer-experience/evals/as oneclaude plugin evalcase each. They use the shared fixtures and deterministic graders where possible. Anyllmgrader gets checked against the committed labeled samples. Done when both cases pass/evals:validateand the changelog-fragment check exits 0.This depends on #9936 (the scenario sources) and #9943 (the shared fixtures), and both are still open. No duplicates found.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 25, 2026
Not admitted to the issue lane yet: blocker #9936 is not closed by a listed human or a merge; blocker #9943 is not closed by a listed human or a merge.