Repository navigation
test(developer-experience): run the plugin-eval suite on the eval dispatch lane and record the delta #10005
Description
Refs: #6649
Parent
Depends on
- test(developer-experience): port eval cases S1 and S4 #9953
- test(developer-experience): port eval cases S2 and S5 #9958
- test(developer-experience): port eval cases S3 and S6 #9966
- test(developer-experience): port eval cases S7 and N1 #9979
- test(developer-experience): port eval cases T1 and T2 #9988
- test(developer-experience): port eval cases T3 and T4 #9993
- test(developer-experience): port the stays-quiet eval case #9998
Problem
#6649 is done when a dispatch runs the suite and posts the with/without delta per case.
Owner decision needed
The approved split runs this on the shared CLAUDE_CODE_OAUTH_TOKEN, the subscription login. On
2026-10-24 the owner ruled that eval and claude runs use an API-key credential, never the subscription
login. Which credential this run uses is the owner's call:
ANTHROPIC_API_KEYin the eval environment (recommended: matches the 2026-10-24 rule, and the
plugin-evals page's CI example names only that variable, per feat(developer-experience): claude plugin eval suite in GitHub Actions #6649's body).CLAUDE_CODE_OAUTH_TOKENas planned, with an explicit exception to the rule.
Fix
Dispatch the eval workflow after the cases merge, and record the per-case delta and whether the chosen
credential reached the eval run.
Done when
- The run's per-case delta is posted on feat(developer-experience): claude plugin eval suite in GitHub Actions #6649.
- The credential question in feat(developer-experience): claude plugin eval suite in GitHub Actions #6649's body is answered with the run's evidence.
Filed by AI from the owner-approved split of #6649 (2026-10-18), after the scenario sources were found on melo-desk-001 (2026-10-25).
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: none
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: intent
Blocked by: #9953, #9958, #9966, #9979, #9988, #9993, #9998
I understood this as: dispatch the plugin-eval workflow once the cases merge, post the per-case with/without delta on #6649, and record whether the chosen credential reached the eval run. It is blocked until the seven listed dependencies are done. No likely duplicates found.
One decision is open:
ANTHROPIC_API_KEYin the eval environment. Basis: it matches the 2026-10-24 owner rule that eval andclauderuns use an API-key credential, and the plugin-evals CI example names only that variable. The alternative isCLAUDE_CODE_OAUTH_TOKENwith an explicit exception to that rule.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Oct 25, 2026
The triage verdict left this issue at
status: needs-decision:kyle-sexton commentedon Oct 25, 2026
This was generated by AI.
Credential update (owner, 2026-10-25): there is no API-key credential for eval or
clauderuns. They run on the subscription login or not at all. This run is parked until subscription usage allows it, and then it runs on the subscription. That supersedes the API-key wording in my earlier comment.