Repository navigation
test(developer-experience): port eval cases S1 and S4 #9953
Description
Refs: #6649
Parent
Depends on
- chore(developer-experience): commit the hand-rolled eval scenario sources #9936
- test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9943
Problem
#6649 moves the developer-experience measurement onto claude plugin eval. Cases S1 and S4 exist only
as hand-rolled scenarios.
Fix
Port S1 and S4 from the committed scenario sources into plugins/developer-experience/evals/, one case
each, on the shared fixtures. Use deterministic graders where the scenario's D and T items allow; check
each llm grader against the committed labeled samples.
Also
- Case layout:
plugins/developer-experience/evals/<case>/(prompt.md,case.yaml,graders/). - Include the
rmand compound-shell checks where a case needs them. - Carry the S4 build(deps): bump ci-workflows pin to b6431a1 (PSSA RULE_ERROR retry fix) #10 withhold rule check.
Done when
- Each case loads and passes
/evals:validate(no paid run). -
scripts/check-changelog-fragments.sh --check-required origin/mainexits 0.
Filed by AI from the owner-approved split of #6649 (2026-10-18), after the scenario sources were found on melo-desk-001 (2026-10-25).
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: #9936, #9943
Understood: port the hand-rolled eval scenarios S1 and S4 into
plugins/developer-experience/evals/as one case each on the shared fixtures. Use deterministic graders where possible, and check eachllmgrader against the labeled samples. The done-when items are that each case passes/evals:validateand that the changelog-fragment check exits 0.The scenario sources (#9936) and the shared fixtures (#9943) are both still open, and this work needs both. No duplicates found among the open issues.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 25, 2026
Not admitted to the issue lane yet: blocker #9936 is not closed by a listed human or a merge; blocker #9943 is not closed by a listed human or a merge.