Repository navigation
chore(developer-experience): commit the hand-rolled eval scenario sources #9977
Description
Needs a person: the scenario sources live only on the machine that ran the PR #6668 eval, and 3 judge samples need a person to relabel them.
Refs: #6649
What to do
The S1-S7, T1-T4, N1 and stays-quiet scenario prompts, the two invented fixtures (.NET and Python), the checklists and the labeled judge samples from the PR #6668 eval exist only outside this repository, on the machine that ran it. Commit them under plugins/developer-experience/evals/sources/ (no transcripts, per .claude/rules/eval-case-transcripts.md). In the same step, relabel the 3 T4 #2 judge samples that read the rubric literally (#6649 comment of 2026-10-09T16:15).
Done when
-
git ls-files plugins/developer-experience/evals/sources | wc -lprints a number above 0, and the tree holds a prompt for each of S1-S7, T1-T4, N1 and stays-quiet. - The T4 fix: correct plugin-system inaccuracies and harden references #2 samples carry person-made labels.
-
ci-statusis green on the PR.
Split by the owner's HITL session (2026-10-24, owner decisions C12 and Q93-Q95). Intake triage sets its fields.
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: none
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
I understood this as a request to commit the PR #6668 eval scenario sources (S1-S7, T1-T4, N1 and stays-quiet prompts, the two fixtures, the checklists and the labeled judge samples) under
plugins/developer-experience/evals/sources/, without transcripts. It also asks for the 3 T4 #2 judge samples to be relabeled by a person. The sources currently exist only on the machine that ran the eval, so someone with access to that machine has to supply them. The issue names no likely duplicates. It is related to #6649.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Oct 25, 2026
The triage verdict left this issue at
status: needs-decision:kyle-sexton commentedon Oct 25, 2026
Duplicate split of #6649 by a second Backlog HITL writer. To reopen: reopen this issue and say what the kept child misses.
Kept child: #9936.