Repository navigation
test(developer-experience): port eval cases T1 and T2 #9988
Description
Refs: #6649
Parent
Depends on
- chore(developer-experience): commit the hand-rolled eval scenario sources #9936
- test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9943
Problem
#6649 moves the developer-experience measurement onto claude plugin eval. Cases T1 and T2 exist only
as hand-rolled scenarios.
Fix
Port T1 and T2 from the committed scenario sources into plugins/developer-experience/evals/, one case
each, on the shared fixtures. Use deterministic graders where the scenario's D and T items allow; check
each llm grader against the committed labeled samples.
Also
- Case layout:
plugins/developer-experience/evals/<case>/(prompt.md,case.yaml,graders/). - Include the
rmand compound-shell checks where a case needs them.
Done when
- Each case loads and passes
/evals:validate(no paid run). -
scripts/check-changelog-fragments.sh --check-required origin/mainexits 0.
Filed by AI from the owner-approved split of #6649 (2026-10-18), after the scenario sources were found on melo-desk-001 (2026-10-25).
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: #9936, #9943
Understood: port the hand-rolled developer-experience scenarios T1 and T2 into two
claude plugin evalcases underplugins/developer-experience/evals/. They use the shared fixtures and deterministic graders where possible. Eachllmgrader is checked against the labeled samples. Done when each case passes/evals:validateand the changelog-fragment check exits 0.The scenario sources (#9936) and the shared fixtures (#9943) are both still open and are listed as dependencies, so this work waits on them. I found no likely duplicates.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 25, 2026
Not admitted to the issue lane yet: blocker #9936 is not closed by a listed human or a merge; blocker #9943 is not closed by a listed human or a merge.