Add STAMPED skills development plan

[yarikoptic]

Summary

A development plan for a collection of STAMPED skills, from stamped-principles/stamped-agent-skills#3. It builds on #23 (assess-stamped, align-stamped, still open), which lands first on its own, and on #16 (merged): its report conventions and the "Skill versions" rule.

The skills are developed against concrete use cases. Agents run them on a ladder of ds007116 tasks in fresh containers, and the resulting projects are graded outside the sandbox, by scripted probes, private reference tables and fresh-clone reruns.

Docs only; there are no skills or code yet.

Files

• docs/stamped/PLAN.md: the skills and their names, design principles, layering and layout, answer format, roadmap with per-PR obligations, risks, and decisions and open questions.
• docs/stamped/evals.md: use cases R0–R2 (with R1, brain extraction on 10 named subjects, as the main target), pilots, arms, runners and sandbox rules, grading (external vs self-assessed), calibration, iteration protocol, publishing, budget and done criteria.
• docs/stamped/prior-work.md: sources.
• docs/stamped/stack-repronim.md: draft ReproNim stack reference.
• docs/stamped/talks.md: how this feeds the MIT and US-RSE talks (WiP: 2026 USRSE CON and MIT ReproNIM STAMPED talks talks#33).

Status

Draft for discussion. Four review rounds and @asmacdo's notes are addressed (f25257e, 4a18324, 3f5a560, f351632), and master was merged in after #16 landed; open decisions are in PLAN §8. The plan is to squash before this leaves draft, or to squash-merge. con/skills has no PR template or contributing guide. The validator workflow does not run on docs-only changes; ci/validate_skills.py passes locally (0 errors).

Handoff: starting phase 1b/2 on a controlled host

For a Claude Code session continuing there:

1. Work from branch claude/ecstatic-cerf-nh4j6w, or branch off it.
2. Check the host: Apptainer, scratch space (R1 needs a few GB per image), the Claude CLI, and uv.
3. Install the pinned runner tooling and record it under evals/stamped/:
   • datalad (a release with the gh-7899 fix);
   • datalad-container;
   • git-annex (the PyPI wheel works);
   • con-duct.
4. Bootstrap the results dataset (datalad create -c text2git), add con/skills as a subdataset, and add the hub siblings, private first.
5. Follow PLAN §6 from the current phase.

🤖 Generated with Claude Code

https://claude.ai/code/session_018TvTLThBRhJrk8ZZtGVFNb

[yarikoptic]

Review of the draft plan by two independent reviewers (eval/reproducibility, and skill authoring/repo conventions), who agreed on the list below after one round of corrections. Line numbers refer to docs/stamped/PLAN.md at ea48c88. No CI checks have run on this PR.

Overlap with open work

1. #23 is not mentioned. It already adds assess-stamped / align-stamped, while §4 creates assess-alignment-with-stamped-principles / align-with-stamped-principles from scratch, and L4 settles the names differently from #23's reviewers. Suggest: working names follow #23, the long issue names stay as trigger phrases in description, and Phase 1 builds on #23 (rebased onto stamped-conventions) instead of forking. Please also add #23 to the "con/skills itself" list (L190-199). (We could only see #23's file list, not its contents.)
2. Reuse #16's code-quality-report-conventions for report location, provenance stamp and evidence rules, so stamped-conventions adds only the STAMPED-specific parts (checklist pin, verdict mapping). Skip response-format.md / verdicts.md unless #23 already defines them.
3. Avoid new tooling where something exists. ci/validate_skills.py already has W003 (>600 lines) and analyze-duplicates covers clones, so a new size/duplicate lint looks unnecessary. The harness/{run,grade,aggregate}.py at L312 contradicts L203-204 (reuse skill-creator); drop it or say why skill-creator is insufficient.
4. Validator fit. The validator only globs *.py directly in a skill directory (ci/validate_skills.py:228), so scripts/** is not checked. E003 would flag /home/<user>/ literals in SKILL.md, so keep the no-abs-paths pattern table in references/. L159 says metadata.version, while L194/L254 use top-level version: (as #16): please use top-level.

Eval design
5. R0 oracle (L554) is "a deterministic rerun gives identical outputs", which a wrong column also passes. Suggest *Oracle:* values match a reference table computed by hand and committed under oracle/; determinism is checked separately by fresh-rerun. Please also confirm that *_desc-FreeSurfer_qc.tsv actually holds the GM/WM volumes R0 asks for (L550-551), and add the oracle-not-copied probe that L883 claims exists but §6.5 lacks.
6. Arms and grader. pointer vs skill (L605-609) differ by more than the skill, so define skill as pointer + skills (or add a skills-only arm). Inner-loop runs share the host filesystem with the plan and oracles (L621-623): state that they are for debugging only and never for reported numbers or the pointer/skill comparison. T2 (L672-675) is the skill under test; have it see a stripped copy (no transcript, AGENTS.md or skill stamps), use a different model family from the runner, and never gate a release alone. Also say how hold-out tasks stay clean of the inner loop.
7. Probes and oracle. large-in-annex has no threshold and no-abs-paths would flag legitimate datalad run records, so consider about six probes for v0.1 and compare plots through their TSV, never pixels. "Brain size" is undefined (L565); for example Spearman >= 0.8 between mask volume and FreeSurfer BrainSegVol over the same sessions (threshold set from a pilot). Subject/session selection needs a deterministic rule so repeats are comparable.

Scope and safety
8. Scope. The summary promises "conventions + assess v0.1", but Phase 1 (L805-813) adds an evals skeleton, hub bootstrap, mutants, a lint and upstream PRs. Suggest Phase 1 = stamped-conventions v0.1 + #23's assess on the pin, with mutants as 1b. Consider moving §2 detail, §6.8, §7.1, §8 and the repronim stack detail (L384-404) to follow-up issues. Add a per-iteration budget cap (sessions, USD, wall time) to §6.9, since a full benchmark is roughly hundreds of sessions, and mark R3-R5 out of scope for v0.1. State in §10 that with 3 repeats the "≥ 2 of 3" and 90%/80% criteria (L784-790) are descriptive, not significance tests, and whether they are placeholders.
9. Publishing (L752-766). Please state the token's scope and whether the hub repo is private (an Apache egress allow-list is not authentication). We suggest agent-generated repos (transcripts, absolute paths) land in a private namespace and are promoted to public by a human after a secret/path scan. L762 "verified" is only a dry-run push.

Consistency
10. Assess is "read-only" (L42, L342) but writes into the target (L338), and fresh-rerun (L663) clones and reruns; suggest "writes only its report; fresh-rerun is opt-in, in a scratch clone". L355's introduce-reuse-compliance is renamed setup-* in #21. Open question 2 (L903) is already treated as decided in §5. These probes belong to the assess skill, so points 7 and 10 interact. Optional: note that any phase PR follows the repo's PR conventions.

This is from reading the plan and the repo checkout; I did not run anything, and could not read #16's or #23's file contents.

---

Generated by Claude Code

[yarikoptic]

Thanks for the review. All ten points are addressed in f25257e. I read #23's and #16's contents, which the review could not see; what follows takes them into account.

1. Add assess-stamped and align-stamped skills #23. The working names follow Add assess-stamped and align-stamped skills #23 (assess-stamped, align-stamped, create-stamped-project, create-repronim-stamped-project, do-full-research-loop, self-assess-stamped). The issue's long names become trigger phrases. Phase 1 builds on Add assess-stamped and align-stamped skills #23, and Add assess-stamped and align-stamped skills #23 now appears in §2 and in prior-work.md.
2. Add health report skills #16. Location, provenance stamp and evidence rules now come from code-quality-report-conventions. response-format.md and verdicts.md are dropped: Add assess-stamped and align-stamped skills #23 already defines yes/no/unknown with inspection/execution evidence. Add assess-stamped and align-stamped skills #23 writes the report outside the target, while Add health report skills #16 defaults to docs/ inside it. For assess, the outside-the-target default stays, with Add health report skills #16's dated name and its never-overwrite rule.
3. No new tooling. The size/duplicate lint is dropped. The harness is reduced to a single outer-loop runner that writes skill-creator's layout.
4. Validator fit. Scripts sit directly in the skill directory. The abs-path pattern table goes in references/. Versions use top-level version:.
5. R0 oracle. It is now a hand-computed reference table, with determinism checked by fresh-rerun. The shipped QC tables have the needed columns: 222 sessions, in plain git. oracle-not-copied is added.
6. Arms and grader. skill = pointer + skills. The inner loop is for debugging only. The blind assessor gets a stripped copy. Hold-outs are kept out of the repo.
7. Probes. Six for v0.1, with thresholds and exclusions. Plots are compared through their TSV. Brain size is defined. Subject selection is deterministic.
8. Scope. Phase 1 is narrowed. R3–R5 are out of v0.1. Budget caps are added. The criteria are marked as descriptive placeholders.
9. Publishing. Runs are private first and promoted by a human after a secret/path scan. Access details are kept out of this public PR (edited 2026-10-13 after the second review).
10. Consistency. Assess writes only its report and never reruns. setup-reuse-compliance after Rename introduce-* skills to setup-* #21. The answer vocabulary is open (Q2).

---

Generated by Claude Code

[yarikoptic]

Re-review of f25257e (delta since ea48c88), by two independent reviewers (reproducibility/eval design, and skill authoring/repo conventions). They also looked for bloat and for places where existing repo functionality should be reused. Both agree the ten earlier points are mostly resolved: names follow #23, #16 is reused, scope is R0–R2, the R0 oracle and oracle-not-copied are fixed, and hold-outs are kept out of the repo. The remaining points below are where the two reviewers' findings overlapped or did not conflict. No CI has run on this PR. I did not run anything.

Eval design (worth fixing before phase 1b)

1. Grader independence (PLAN.md ~L293, 478, 529, 594). probes.py ships in assess-stamped, which the skill arm installs, and do-full-research-loop step 8 runs assess-stamped on its own output and fixes the MUST nos. The skill arm therefore optimises against the probes that then grade it. Suggest grading with a frozen, version-pinned copy of the probes plus a few probes kept out of the skill, and reporting "self-assessed" and "externally graded" separately.
2. Oracle leakage (~L429-466). "Installs the skills at a pinned commit" is ambiguous: a clone of con/skills would expose evals/stamped/use-cases/*/oracle/, rubric.yaml and the plan to the agent. Say that only the skill directories are installed, and that oracles and rubrics are mounted into the grader only.
3. Budget arithmetic (~L504, 566, 582). The 40-session cap counts the 28 runner sessions only. It omits one T2 assessor session per run, the skill, explicit and skill@vN-1 arms that §6.7 step 7 requires, and pilot runs. Restate the cap as "runner + assessor sessions" and list the exact v0.1 arm set.
4. fresh-rerun vs promotion (~L496-501; infrastructure.md:25-29). It is defined on the published object, but promotion to public is a human step, so it cannot run per run. Run it from the private namespace with a read token, and block datalad push of runs to the public repo until the private org exists (open question 13). Referencing the private hold-out from the public .gitmodules leaks that it exists.
5. R1 oracle (~L413). Spearman ≥ 0.8 at n = 10 is weak and passes a wrong-unit result (e.g. voxel counts). Add a magnitude check (mask volume / brainseg_brainsegvol within a pilot-derived band, in mm³) and fix the pilot protocol in the use case.
6. Statistics and provenance (~L448, 471, 594). "Clearly better" and "≥ 2 of 3" are undefined at n = 3. Say per-use-case paired pass counts, with raw tallies and no pooled significance claim. The run record should also hold the Claude CLI version, container image digest and dataset commit.

Repo conventions (please check against master)
7. PR rule misapplied (PLAN.md:40, :603; prior-work.md:115). The AGENTS.md PR rule binds skills that prepare PRs for other projects. con/skills has no PR template or CONTRIBUTING, so "phase PRs follow the repository's PR conventions" has no content. State that, and list the real obligations: ci/validate_skills.py (via tox), a README table row per skill, and #16's "Skill versions" rule.
8. version: is not yet a con/skills convention (PLAN.md:102, :145). Nothing on master has top-level version:; it comes only from open #16 (while #23 uses metadata:). Mark it "from #16, not yet merged" and add that to §9 as a phase 1 dependency, together with code-quality-report-conventions and #23 (both unmerged). The "changelog line per bump" has no repo rule behind it; drop it or say where it lives.
9. README rows. Every phase adds skills (including non-invocable stamped-conventions) and none mentions the README table row. #16 is the precedent for non-invocable skills.
10. Validator and #13 (PLAN.md:176, :647). find_skill_dirs globs */SKILL.md and the workflow path filters are */SKILL.md and */**.py, so a later move under skills/ (#13) silently stops validating; add "update the validator and workflow paths" to that move. run_outer.py and mutants/*.py are outside the checked paths. W003 counts only SKILL.md, so reword the bloat mitigation to "W003 on SKILL.md only".

Reuse and size
11. run_outer.py as "the only new code" (~L458, 292). Please re-check whether skill-creator's scripts/run_eval.py / run_loop.py already shell out to claude -p; if so, wrap them and add only the container and datalad layer. The long names kept as trigger phrases also have no trigger test: skill-creator's description-optimisation loop would cover it.
12. Bloat. About 1,000 lines overlap across the five docs: PLAN §2 vs prior-work.md; prior-work.md:113-143 vs PLAN §1/§3/§4.2/§5; the "Decided so far" header vs §10 (with struck-through history); the "Revised 2026-10-13" notes; infrastructure.md "Starting on typhon" vs PLAN §7. Suggest PLAN.md keeps naming, scope, phases, layering and open questions, §6 moves to an evals doc, and prior-work.md holds sources only.
13. Public operational detail (infrastructure.md:26-41). The pushing account, HUB_TOKEN secret and its missing scope, and the egress allow-list are in a public repo. A one-line "results hub: con/stamped-skills-evals, private-first" suffices; keep the rest in a private note.

Optional

• The reviewers differ on stamped-conventions: one finds the lock.json + answer-mapping layering fine, the other would keep the pin inside assess-stamped until a third consumer exists, since #23 already pins by tagged URL and a new skill forces a rebase of #23. Your call; if kept, defer the sha256 and converter until the converter exists.
• Verb consistency after #21: create-stamped-project, create-repronim-stamped-project and do-full-research-loop break the setup-/assess-/align-/analyze-/review- pattern; setup-* names are free to adopt before any skill exists, with the issue's names as trigger phrases.
• Timeline: talks.md offers assess-stamped on t261 and an "R0 in 3 conditions" stretch before calibration (§6.6) and run_outer.py (phase 2) exist; label these "uncalibrated demo, hand-run" or drop the stretch.

Confirmed fine: the validator claims (*.py only directly in the skill directory, W003 > 600 lines, E003 scanning only SKILL.md so references/abs-path-patterns.md works), the #16/#21/#23 cross-references, phase scoping consistency, and the assess-writes-outside-target decision.

---

Generated by Claude Code

[yarikoptic]

The re-review (5503593745) is addressed in 4a18324. The five docs total 714 lines, down from 1,042.

1. Grader independence. Grading uses a frozen copy of probes.py, pinned at the evaluation harness's commit rather than the copy installed with the skill, plus hidden probes stored with the private oracles. "Self-assessed" and "externally graded" are reported separately (evals.md §6).
2. Leakage. Only the skill directories are copied into the sandbox: no evals/, docs/ or .git. Oracles, rubrics and grading probes are mounted into the grader only, after the agent has finished. They are also kept out of public repositories, because the sandbox has network access (§5, §9).
3. Budget. The cap is now 40 sessions counting runners, assessors and pilots. The v0.1 arm set is listed exactly: pointer, skill, skill@vN-1 × R0, R1 × 1 tier × 3 repeats = 18 + 18 + ≤4 pilots. bare, the implicit arm and the second tier are release-only (§10).
4. fresh-rerun. It clones from the private namespace with a read-only token. Nothing goes to the public repo until the private namespace exists. The hold-out dataset is not referenced from any public .gitmodules.
5. R1 oracle. A magnitude band (mask volume / brainseg_brainsegvol) is added to Spearman. The pilot protocol is written into the use case: a hand-run SynthStrip reference sets the band and threshold, and both are frozen before any reported run.
6. Statistics and provenance. Results are per-use-case paired pass tallies with raw counts and no pooled claims. "Clearly better" becomes "never below pointer". Each run records the CLI version, model, image digest, con/skills commit and dataset commit.
7. PR rule. Corrected: the AGENTS.md rule is for skills that prepare PRs elsewhere. The real obligations are now listed: tox -e validate, a README row per skill, and Add health report skills #16's version rule (PLAN §6).
8. version:. Marked as coming from Add health report skills #16, not yet merged. Add health report skills #16 and Add assess-stamped and align-stamped skills #23 are named as phase 1 dependencies in §6 and §7. "Changelog line per bump" is dropped.
9. README rows. Included in the per-PR obligations.
10. Validator. The feat: distribute and organize skills through APM #13 move now carries the validator and workflow path updates. Phase 2 extends the Python checks and path filter to evals/. The bloat mitigation now reads "W003 on SKILL.md only".
11. Reuse. Confirmed: skill-creator's run_eval.py already calls claude -p, but only for trigger tests. It is reused unchanged for the trigger phrases, including the long issue names. The new runner adds only the container and the datalad run layer for full task runs.
12. Bloat.
    • PLAN.md keeps names, scope, layering, the answer format, phases, risks, and decisions/open questions.
    • §6 moved to evals.md.
    • prior-work.md is sources only.
    • infrastructure.md is removed; its typhon steps are in evals.md §11.
    • The struck-through history and "Revised" notes are gone.
13. Operational detail. Removed from the docs. My earlier comment on this PR is edited to drop the same details. They remain in the earlier commits on this branch (e9839d3, 90f4153); a squash merge keeps them out of master.

Optional points:

• stamped-conventions: dropped for now. The pin stays in assess-stamped, where Add assess-stamped and align-stamped skills #23 has it, until a third skill needs it directly.
• Names: kept create-* and do-full-research-loop. They are verb-first, and setup-* in Rename introduce-* skills to setup-* #21 means adding tooling to an existing project, which these do not do. Happy to switch if you prefer.
• Talks: the t261 demo is labelled "uncalibrated demo, hand-run", and the R0 stretch is dropped.

---

Generated by Claude Code

[asmacdo]

One philosophical difference I think we should discuss: what are these skills for?

• Hands-on with the user (my preference): the skills provide patterns and context, and the user drives their agent the way they prefer.
  • Pros: the user stays involved and understands the what and the why; minimal required trust is a design goal. You trust your agent when you control it, but when it executes shared instructions, you can't trust it without reading the whole skill. Keeping the user in the loop for anything consequential means they don't have to.
  • Cons: less deterministic; harder to compare runs on one project or across projects; harder to benchmark.
• A packaged executable that performs the task (where I think Add STAMPED skills development plan #24 leans):
  • Pros: works out of the box; automatic, one-shot; easier to compare.
  • Cons: requires trusting instructions the user likely hasn't read; does not respect the user's preferred workflow by default; more complex, with more surface area to maintain.

#23 tries to serve both: level 1 is the comparable one-shot, and level 2 is hands-on. The open question is which one is the default and the design center.

---

Generated by Claude Code

[asmacdo]

Thanks, this builds on #23 nicely. On the points addressed to me:

• Q7 (who rebases onto whom): Add assess-stamped and align-stamped skills #23 lands first, standalone, with the review fixes; stamped-conventions follows as a small PR on top once there is a second consumer of the pin.
• Verbatim JSON copy: not for now. Tag-pinned raw URLs are enough for v0.1, and a vendored copy is one more thing to drift.
• probes.py: I'd keep it out of assess-stamped and under evals/. Several probes assume a stack (annex, .datalad/, run records), which conflicts with the skill's tool-agnostic stance, and shipped inside the skill they would grade the arm that installed them. I'd like the skill itself to start very minimal.
• Assessment levels: Add assess-stamped and align-stamped skills #23 now splits assessment in two. Level 1 is inspection only and never runs target code, so it is safe on any target and comparable across runs; evals would use it. Level 2 is guided execution: the agent proposes commands to settle the unknowns and the user approves each one.
• allowed-tools: Add assess-stamped and align-stamped skills #23 omits it in both skills. The field grants pre-approval and restricts nothing, so a shared skill that sets it widens the installing user's permissions; the skills defer to the user's setup instead. The separator question for skills that do set it is allowed-tools: comma-separated or space-separated? #25.

---

Generated by Claude Code

[yarikoptic]

Re-review of 4a18324 (delta since f25257e; net about -330 lines). No CI runs on this docs-only PR. Two independent reviewers (eval design, and repo conventions/packaging) found all 13 earlier points addressed; nothing is blocking. Confirmed: independent grading with a frozen probes.py and hidden probes, skill-directories-only install with oracles mounted into the grader only, the PR-rule correction (real obligations: tox -e validate, README row, #16's version rule), version: marked as pending #16, #16/#23 listed as phase-1 dependencies, validator/#13 path note, infrastructure.md removed, prior-work.md reduced to sources.

Should fix

1. R2 vs the budget (evals.md §3 vs §10). v0.1 covers R0–R2 but the 40-session arithmetic counts only R0 and R1, and R2's CPU-hour pilot is not in it. Add R2 to the arithmetic or mark it "not in the per-iteration budget".
2. Cap headroom (§10). 18 + 18 + 4 = 40 exactly, so one harness-fault rerun breaks the cap, and 18 sessions go to an assessor the plan calls advisory. Suggest counting reruns in the cap, and running assessors on a sample of about 3 objects per iteration until the §7 gold-set calibration passes.
3. Statistics wording (§10). "Paired" is a misnomer (repeats are not paired across arms). At 3 repeats even 3/3 vs 0/3 is only p ≈ 0.1 two-sided (Fisher), so the "no regression" and "never below pointer" v1.0 gates cannot be judged. Suggest "side-by-side pass counts (k of 3 per arm), raw counts, no significance claims", and either raising repeats (about 10 per arm on the main use case) before v1.0 or labelling the gates smoke thresholds.
4. oracle-not-copied (§6). "No run record reads derivatives/freesurfer-post/" fails a legitimate sanity check. Suggest: no output reproduces the shipped values (or a constant multiple), and no output depends via run records on that directory as an input; read-only inspection is logged as a process metric.
5. fresh-rerun token (§6). datalad rerun executes agent-authored code, so a read token in the rerun container's environment or .git/config is exposed to it, along with the private namespace (oracles, hold-outs). Suggest a token scoped to the runs namespace, kept outside the container, clone mounted read-only, and egress limited to data and container hosts.
6. Sandbox can still fetch grader material (§5). con/skills is public on github.com, which also hosts the datasets, so a domain allow-list cannot block it. State that public grader material is assumed readable (only oracle values and hidden probes carry secrecy), and add "fetches of con/skills or the evals repo" to the process metrics, flagging such runs as contaminated.
7. R1 band (§3). A band calibrated on one tool (SynthStrip, ±10%) would fail a legitimate container with a different mask definition. Suggest the union of ratio ranges from two containers, widened by 10%, and say that the frozen values live in the private oracle (only the protocol is public).
8. Session-handoff content in a durable doc. evals.md §11 ("Starting on typhon") hard-codes the branch claude/ecstatic-cerf-nh4j6w and host checks, and is linked from L100 and PLAN.md L234. Move it to the PR description or a phase-2 issue and repoint the links to §5. The docs also address one reviewer directly ("your review", "yours to set", "you sign off", "my recommendation"); use "the maintainers" / "recommendation".
9. History. Commits 90f4153 and e9839d3 still name the HUB_TOKEN network secret, the egress admission and the pushing account, even though the files are clean. Squash-merge so none of it reaches master, and consider rewording or squashing the branch before it leaves draft.

Optional

• Remaining bloat: PLAN §5 restates evals §4–6 and §10; PLAN §7 repeats dependency, publishing and cap text; prior-work.md "Hence:" lines repeat PLAN §2; the -n repronim-mriqc slide note belongs in con/talks#33.
• Say in evals §2 that run results and reviews reuse the stamped-skills-assessment record types and skill-creator's grading.json (no new schema).
• Hidden probes need private mutants; define "declared output" for derived-have-records (every tracked file under results/ or derivatives/ is an output of a datalad run record); large-in-annex exempts text, so add an upper cap (e.g. 5 MiB).
• Names: a one-line note that create-stamped-project, create-repronim-stamped-project, do-full-research-loop and self-assess-stamped deliberately keep the issue's wording until the setup-* family from #21 settles.
• typhon appears throughout public text; "the lab compute host" would do.

I did not run anything; this is from the diff and the five docs at head.

---

Generated by Claude Code

[yarikoptic]

3f5a560 addresses the third review (5508263110) and @asmacdo's two comments.

@asmacdo

• Q7: adopted. Add assess-stamped and align-stamped skills #23 lands first, on its own. A shared conventions skill comes later as a small PR on top, once a second consumer of the pin exists. No vendored JSON; tag-pinned URLs only.
• probes.py: moved to evals/stamped/, never into a skill, for both reasons you gave. assess-stamped stays Add assess-stamped and align-stamped skills #23 as it is.
• Levels: the evals use Add assess-stamped and align-stamped skills #23's level 1. The blind assessor runs level 1 on a stripped copy, and level 2 stays user-approved.
• allowed-tools: the skills omit it, as Add assess-stamped and align-stamped skills #23 does, with a pointer to allowed-tools: comma-separated or space-separated? #25.
• Hands-on vs packaged: recorded as PLAN §8 open question 1, with a recommendation for maintainers to decide:
  • hands-on is the default, and the user approves anything consequential;
  • unattended behaviour exists only where it is safe by construction: Add assess-stamped and align-stamped skills #23's level 1, and the eval sandbox, which stands in for the user's approval;
  • do-full-research-loop asks for one approval of its plan, then asks again for anything outside the project.

Third review

1. R2: its own iterations, outside the per-iteration budget.
2. Cap: reruns now count. The blind assessor samples about 3 objects per iteration until §7 calibration passes. That makes 18 + ~3 + ≤4 = 25 of 40, leaving room for reruns.
3. Statistics: "side-by-side pass counts (k of 3 per arm), raw, no significance claims", with the Fisher caveat stated. Per-iteration gates are smoke thresholds; v1.0 uses a release benchmark with about 10 repeats per arm on R1.
4. oracle-not-copied: fails only when an output reproduces the shipped values (or a constant multiple), or depends on that directory through its run records. Read-only inspection is a logged process metric.
5. fresh-rerun: the grader host clones with a token scoped to the runs namespace that never enters the container. The clone is mounted read-only, and egress is limited to data and container hosts.
6. Public grader material: assumed readable. Only oracle values and hidden probes are secret. Fetches of con/skills or the evals repo are a process metric that marks a run contaminated.
7. R1 band: the union of the SynthStrip and FSL bet ranges, widened by 10 %. Only the protocol is public; the values are in the private oracle.
8. Session handoff: moved out of the docs and into the PR description. Direct address and the host name are gone.
9. History: agreed to squash. My suggestion is to squash-merge, or squash the branch just before it leaves draft, rather than now, while reviews still cite f25257e and 4a18324.

Optional points: all taken.

• PLAN §5 is now one line.
• The "Hence:" line is dropped.
• The -n repronim-mriqc slide note is out of talks.md, for WiP: 2026 USRSE CON and MIT ReproNIM STAMPED talks talks#33.
• evals §2 names the reused record formats.
• Hidden probes get private mutants.
• "Declared output" is defined.
• large-in-annex has a 5 MiB cap for text.
• The names note is added.

---

Generated by Claude Code

[yarikoptic]

Re-review of 3f5a560 (delta since 4a18324). No CI runs on this docs-only PR. Two independent reviewers (eval design and reproducibility, and skill authoring and repo conventions) found nothing blocking and their lists did not conflict. Verified: the 18 + ~3 + ≤4 = 25 of 40 arithmetic, the Fisher figure (3/3 vs 0/3 gives p = 0.1), the R1 band definition, and the evals.md §5/§6/§7/§10 cross-references. The session-handoff text is gone, and probes now live under evals/ as agreed. I did not run anything.

Should fix

1. #16 is merged (5ae48e9 on master, with code-quality-report-conventions at version: 0.1.0 and the "Skill versions" rule in AGENTS.md). PLAN header, §6 ("#23 and #16, both unmerged"), §7 ("Unmerged dependencies") and prior-work.md still treat it as open. Rebase on master, drop "once #16 is merged", and say new skills declare version: 0.1.0 while their PR is open.
2. W002 is expected. ci/validate_skills.py warns on a missing allowed-tools and still exits 0. Say in PLAN §2.4 and §6 that the warning is expected and tied to #25, so it is not read as a miss.
3. Pilot sessions (evals.md §3 vs §10). §3 freezes the pilots before any reported run, but §10 charges "≤4 pilots" to every iteration. Make them a one-off outside the per-iteration cap, or charge them to the first iteration, and say whether the 4 are 2 containers × 2 rungs.
4. "Reruns" in §10 means re-running agent sessions after harness faults, which fresh-rerun in §6 is not (it is a grader-side datalad rerun). Reword to "repeat runs after harness faults or contamination".
5. Contaminated runs (§5) have no stated fate. Suggest they are excluded from the k-of-3 count and rerun from the spare sessions.
6. R1 ordering threshold (§3). "At least the pilot threshold" for Spearman is undefined. Give one rule, for example the lower of the two pilot values minus a margin.
7. v1.0 gate scope (§10). The ~10-repeat benchmark is "on R1", but the criterion says "every v0.1 use case". Narrow one of them.
8. fresh-rerun mounts the clone read-only (§6), but datalad rerun must write. Say "read-only source clone, rerun in a throwaway copy". The "egress limited to data and container hosts" claim should also admit that GitHub hosts the datasets.
9. Operational detail in public docs. evals.md §9 says it is kept out, yet PLAN §8 ("token scopes"), evals.md §5 ("lab's compute host") and §6 (token and network layout in fresh-rerun) still carry some. Trim to "runs in a fresh container on a controlled host".
10. Wording in evals.md §5: "They share a filesystem with this plan" reads as leftover handoff text. Use "with the grader material".
11. Validator extension (Phase 2). The validator walks only */SKILL.md and skill_dir/*.py, so extending it to evals/ means special-casing a non-skill directory. Present it as a choice, with ruff in its own tox env or only a workflow path-filter entry as the simpler option.

Optional

• oracle-not-copied catches only exact copies and constant multiples. Say the run-record dependency clause is the primary check.
• The v1.0 gate "pass count ≥ pointer's" can pass on noise at ~10 repeats; consider a stated margin, still as raw counts.
• Say how an unattended claude -p run gives do-full-research-loop its "one approval" (a pre-approving prompt or a permissions flag), or it may stall.
• Phase 1b needs a hand-built known-good R0 object before the Phase 2 runner exists, and should state that it creates evals/stamped/.
• "Probes live under evals/, never in a skill" and "omit allowed-tools" are each stated in 3–5 places. Keep one statement and make §8 "Decided" links. Run analyze-duplicates over the five docs.

---

Generated by Claude Code

[yarikoptic]

f351632 addresses the fourth review (5509631912). master was merged in as 8430544 rather than rebased, so the hashes cited in the earlier reviews stay valid; the branch gets squashed before it leaves draft anyway.

1. Add health report skills #16 merged. The header, §6, §7 and prior-work.md now treat it as merged. New skills stay at version: 0.1.0 while their PR is open. Add assess-stamped and align-stamped skills #23 is the only remaining dependency.
2. W002 is stated as expected, once, in PLAN §2 item 4, and referenced from §6.
3. Pilots are a one-off outside the per-iteration cap. They are hand-run compute, not agent sessions: 2 containers × R1/R2, so 4 runs.
4. "Reruns" now reads "repeat runs after harness faults or contamination". The arithmetic is 18 + ~3 = 21 of 40.
5. Contaminated runs are excluded from the counts and repeated from the spare sessions.
6. R1 ordering threshold: the lower of the two pilot Spearman values minus 0.1.
7. v1.0 gate is scoped to R1, with a stated margin (2 of about 10, placeholder, still raw counts). R0 and R2 stay at smoke level.
8. fresh-rerun: the rerun happens in a throwaway copy of a read-only source clone, in a container without credentials. It now says GitHub stays reachable.
9. Operational detail: "token scopes", "lab's compute host" and the token/network layout are trimmed; a controlled host is all the docs say now.
10. evals §5 now reads "share a filesystem with the grader material".
11. Python checks for evals/ are presented as a choice: a workflow path-filter entry alone, or ruff in its own tox env. validate_skills.py stays skill-only.

Optional:

• oracle-not-copied now treats the run-record clause as primary.
• §5 says the eval prompt supplies do-full-research-loop's one approval, with permissions pre-granted only inside the container.
• Phase 1b creates evals/stamped/ with a hand-built R0 object.
• The repeated statements are reduced to one each, with §8 linking to them. jscpd (what analyze-duplicates uses) finds 0 clones across the five docs.

ci/validate_skills.py reports 0 errors on the merged tree (18 skills). The 10 warnings are on master's existing skills.

---

Generated by Claude Code