Sample goldens are regenerated by an undocumented manual procedure held outside the repository

[pengfei-threemoonslab]

Delivered — 2026-09-09

Implemented in #605, merged 826557678271b5fdabb0727f2103b5ed1a00ce0c. scripts/regenerate_goldens.py regenerates all 24 tracked golden artifacts in 7 sample directories from the current source tree, with per-sample selection and --check; the historical 23-directory estimate below is not the current inventory. Disposable copies use manifest-relative output paths, LF inputs/outputs, isolated Git state, no ambient plugins, fixed packet time and normalization before control-pointer hashing. Missing/drifted goldens and machine-local paths fail explicitly. Existing CI executes the command through its regression test, including an invocation outside the repository directory. Contributor, schema/contract and sample guidance now use this recipe.

Two independent GitHub coding-agent review/address rounds: round 1 / address, round 2 / address. Final full suite: 9,291 passed, 5 skipped; final-head CI, aggregate coverage and current-base verification passed. The first round's repeated-CRLF conversion defect was reproduced, fixed and independently rechecked.

This closes the reproducible-current-goldens work. #569 still owns the report 1.0 freeze and any qualification/migration fixtures introduced with that contract; these sample tests are not release qualification.

v1.0 release disposition — 2026-09-08

v1.0 freeze slice (#569/#572): commit a reproducible recipe for the report 1.0 fixtures actually used for qualification/migration. Broad historical-golden cleanup and a new automation framework are not prerequisites for the tag.

Summary

Sample goldens are byte-compared in CI, and regenerating them is an undocumented manual procedure held outside the repository. There is no Makefile, no justfile, and no regeneration script under scripts/ (which has 17 scripts, including generate_schemas.py for the JSON schemas — the goldens got no equivalent).

The scale of what is regenerated by hand: 23 sample directories, each with expected/ artifacts (report.md, report.json, suggested-declarations.yaml, cold-report.md, packet files). The procedure carries at least two traps that are known only because someone hit them:

• a relative output_dir resolves under the manifest directory, not the working directory, so the naive command writes the files somewhere unexpected;
• the shipped control-pointer digest must be computed after repository-path normalization, not before — a real defect that shipped and had to be fixed (30aed75e, "the shipped control pointer hashes the bytes committed beside it").

Both facts currently live in a private assistant memory rather than in the repository, which is the same continuity problem as #494 and directly blocks anyone else from touching a golden.

Proposal

A regeneration script, in the repository, with a check mode — the shape scripts/generate_schemas.py already established for schemas:

python scripts/regenerate_goldens.py            # rewrite all sample goldens
python scripts/regenerate_goldens.py --check    # verify no drift; exit 1 on diff
python scripts/regenerate_goldens.py <sample>   # one sample

It owns the traps: correct output_dir resolution, normalization before hashing, deterministic ordering, and any temp-path scrubbing (a generated_reports temp path leaked into a golden once already). CI calls --check, which replaces the current byte-comparison tests' regeneration story with a single source of truth.

Acceptance

• The script exists, regenerates every sample golden, and --check reports drift with the offending file named.
• Running it on a clean tree produces no diff — the committed goldens are exactly what it generates.
• The known traps are encoded in the script, not in a comment: normalization happens before hashing, and relative output paths resolve as the manifest expects.
• CONTRIBUTING.md § Useful Commands names it; the schema-bump and contract-bump checklists point at it instead of describing the manual steps.
• Negative control: introducing a deliberate one-byte change in a golden makes --check fail and names that file; introducing a temp path into a report makes it fail rather than committing it.

Surface discipline

Developer tooling; no product surface, no engine behavior. Headline metric: none directly — it removes a manual, trap-laden procedure from the critical path of every schema and contract change, and it is a prerequisite for anyone other than the current maintainer editing a sample.

Related

#494 (the continuity issue — this is the highest-value of the privately-held recipes) · #425 / #458 (the samples whose goldens this regenerates) · 30aed75e (the digest-ordering defect this encodes against) · scripts/generate_schemas.py (the pattern to follow).