fix(evals): run scaffold scripts in plugin-eval and fail validity when a case was never scaffolded

Problem

claude plugin eval runs a case's scaffold_script only when --scaffold is passed. claude plugin eval --help on 2.1.311 says: "--scaffold Run each case's scaffold_script (runs author-supplied bash as you; off by default)".

The Run command in plugins/evals/skills/plugin-eval/SKILL.md:237 omits --scaffold. Every case that builds its workspace in scaffold.sh therefore runs in an empty directory.

run-validity.py still returns verdict: VALID on such a pass, so the empty runs look like real results.

Evidence

On 2026-10-27, a 27-run pass over plugins/review/evals/security-review-* without --scaffold behaved as follows:

Every run answered "the diff isn't in the workspace".
Every positive case scored 0, and every negative case passed vacuously.
run-validity.py printed verdict: VALID, warning only about permission denials.
The pass cost about 3.4 USD.

The same pass with --scaffold produced the expected results.

Fix

Add --scaffold to the Run command when any case sets context.scaffold_script, naming that it runs repository-authored bash as the operator.
Have run-validity.py fail a run whose case declares a scaffold script but whose workspace was never scaffolded.

Understood: the Run-command half is already done (#10438), so what's left is a scaffold check in run-validity.py. A pass where a case sets context.scaffold_script but shows no sign of scaffolding should get a non-VALID verdict naming the case. An unreadable case.yaml should report UNCHECKED, and cases without a scaffold script should be unaffected. The earlier refinement already gives the acceptance criteria. Pull request #10919 was closed unmerged, so the work is open again. Which scaffold signal to use is still to be found by comparing a --scaffold run with a plain run. No duplicates found among open issues.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

The latest triage verdict holds nothing: the issue is at status: ready.

Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

Intended outcome: run-validity.py reports INVALID for any run whose case sets context.scaffold_script but whose workspace was never scaffolded; the SKILL.md --scaffold half is already in place.

Related work

fix(evals): plugin-eval sealed-sandbox cleanup and --scaffold condition in the run step #10438 (fixed-by): closed PR titled as the --scaffold condition in the run step; SKILL.md:242-243 already says to add --scaffold when a case sets context.scaffold_script; merge state not shown
test(heuristic-evals): fix the scaffold and skill-fired defects and add behavior-present variant cases #10184 (related): scaffold defects in heuristic evals

Open questions

Is the Run-command half already done? SKILL.md:242-243 already says to read each scaffold script and add --scaffold, and evals.json has a scaffold-case-gets-the-flag case. Should this issue shrink to the run-validity.py check only?
RECOMMENDED: Yes: shrink to the run-validity.py check; keep the command at SKILL.md:237 as is since --scaffold is conditional. Basis: plugins/evals/skills/plugin-eval/SKILL.md:242
What in the result JSON or trace shows a workspace was scaffolded? run-validity.py reads no scaffold field today, so the check has nothing to key on.
RECOMMENDED: Compare a kept trace and result from a run with --scaffold and one without, key on the field or line that differs, and fail when it is absent; if none differs, have the operator pass a flag such as --scaffolded to run-validity.py. Basis: plugins/evals/skills/plugin-eval/scripts/run-validity.py:65
How should the validator learn which cases declare a scaffold script, and what if case.yaml is unreadable?
RECOMMENDED: Read context.scaffold_script from <suite.root>/<case dir>/case.yaml like prompt.md is read at line 294; report UNCHECKED, not PASS, when the file or the scaffold signal is unreadable. Basis: plugins/evals/skills/plugin-eval/scripts/run-validity.py:294
Should the failure apply to both arms or only the with-arm, and should a run intentionally made without --scaffold be allowed?
RECOMMENDED: Fail both arms, since the empty workspace invalidates the delta; no opt-out. Basis: judgment

Sources read

plugins/evals/skills/plugin-eval/SKILL.md:237: the Run command lacks --scaffold in its literal text
plugins/evals/skills/plugin-eval/SKILL.md:242: prose already says read each scaffold script then add --scaffold
plugins/evals/skills/plugin-eval/scripts/run-validity.py:843: checks list has no scaffold check
plugins/evals/skills/plugin-eval/evals/evals.json:171: case scaffold-case-gets-the-flag exists

To answer, post a comment that starts with this line:

**Decision recorded.** <answer>

_{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}

Re-triage. No owner stop applies. This is a validator bug fix. The refinement's open questions are decided below on the code at origin/main 1ac46f037a.

Is the Run-command half done?
Decided: yes. fix(evals): plugin-eval sealed-sandbox cleanup and --scaffold condition in the run step #10438 (merge 0de10be6e3) added the condition at plugins/evals/skills/plugin-eval/SKILL.md:242-243: "When a selected case's case.yaml sets context.scaffold_script, read each such script first ... then add --scaffold." The scope shrinks to run-validity.py.
What shows that a workspace was scaffolded?
Decided: compare the result JSON and a kept trace from a pass run with --scaffold against one run without it, and key on the first field or line that differs. If nothing differs, run-validity.py takes an operator flag (for example --scaffolded) recording that the pass used --scaffold. Basis: run-validity.py reads no scaffold field today (checks list at :843).
How does the validator learn which cases declare a scaffold script?
Decided: it reads context.scaffold_script from each case's case.yaml under the suite root, the same way it reads prompt.md (run-validity.py:294). If the file or the scaffold signal cannot be read, it reports UNCHECKED, never PASS.
Which arms fail?
Decided: both arms, with no opt-out, since an empty workspace invalidates the delta. Basis: the issue's evidence of 27 runs, each answering "the diff isn't in the workspace".

Acceptance criteria:

run-validity.py has a scaffold check. A pass where a case sets context.scaffold_script and the run carries no scaffold evidence (the signal from decision 2) gives a non-VALID verdict that names the case.
A test fixture for that case, without the scaffold signal, fails the check. The same fixture with the signal passes it.
A case whose case.yaml cannot be read reports the check as UNCHECKED, not PASS, and a test covers it.
A case with no context.scaffold_script is unaffected, and a test covers it.
SKILL.md names the new check where it lists what run-validity.py judges.
scripts/check-changelog-fragments.sh --check-required origin/main exits 0.

The lanes App's pull request #10919 for this issue was closed without merging. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage, which decides what work is left and admits it again.

The lanes App's pull request #11291 for this issue was closed without merging. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage, which decides what work is left and admits it again.

Navigation Menu

Repository navigation

fix(evals): run scaffold scripts in plugin-eval and fail validity when a case was never scaffolded #10504

Problem

Evidence

Fix

Related

1 remaining item

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

fix(evals): run scaffold scripts in plugin-eval and fail validity when a case was never scaffolded #10504

Description

Problem

Evidence

Fix

Related

Activity

github-actions commented on Oct 27, 2026

melodic-automation-lanes commented on Oct 27, 2026

melodic-automation-lanes commented on Oct 27, 2026

melodic-automation-lanes commented on Oct 27, 2026

kyle-sexton commented on Oct 29, 2026

1 remaining item

melodic-automation-lanes commented on Oct 30, 2026

melodic-automation-lanes commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions