Repository navigation
Fix the probe's dead columns and the audit's ignored failures and redundant repeats - #109
Conversation
The headless summary has printed Grazers, Hunters and Omnivores since the diet axis landed, so the probe's Foragers and Predators fields never matched and both columns always read "?". Parse the three current labels and widen the step-summary headers in probe.yml and the description in QUALITY.md to match.
lefthook.yml runs scripts/pre-push-test.sh at pre-push, which skips scripts/test.sh on docs-only pushes; setup.sh still told the developer the hook was scripts/test.sh.
Headless runs have been deterministic since #31, so the second run of each seed reproduced the first and every audit since has said "one run per seed". REPEATS is now 1. The seed-42 determinism probe pair stays. run_one was backgrounded and a bare wait returned 0, so a crashed or truncated run passed silently and showed up only as "?" in the summary table. Each batch is now waited on PID by PID; a run that exits non-zero or ends without a final summary is reported as it happens, the rest of the audit still runs, and the script exits 1 naming the failed runs. Extinction is still a result, not a failure. The probe pair now goes through the same batching, so PARALLEL=1 runs it sequentially.
The weekly schedule and the workflow_dispatch default ran each seed twice on separate runners, and nothing compared the pair: headless runs have been deterministic since #31, so the second run reproduced the first. Both now default to one run per seed, matching scripts/attractor_audit.sh; the runs input still accepts 2, and the determinism job remains the same-seed comparison. Triggers, schedule and permissions are unchanged.
The audit launches the probe pair together only when PARALLEL is above 1.
…design doc The Measurement section still said every phase audit runs each seed twice and carries same-seed divergence as a caveat. Headless runs have been deterministic since #31 and scripts/attractor_audit.sh now runs each seed once.
ReviewIndependent review by a separate Claude Opus 5.5 reviewer agent at Verified: probe.sh's field patterns match the headless summary's Findings: 2.
Not checked: the GitHub Actions Probe run itself (the ten-column step Post-rebase reviewIndependent review by a separate Claude Opus 5.5 reviewer agent at Verified: the diff covers only the four current findings, removes only Findings: 3.
Not checked: the Probe workflow on a real GitHub runner; extinction |
Why
Four small tooling defects from the 2026-10-09 full review, all in the
scripts that watch the simulation rather than in the simulation itself.
probe-summary-forager-predator-columns-dead:scripts/probe.shparsed
Foragers:andPredators:from the headless summary, whichhas printed
Grazers:,Hunters:andOmnivores:since the dietaxis landed (Add the diet trait and four-way strategy instruments with no behaviour change #18). Two of the probe's six numbers were always "?", so
the run page of every probe job hid the strategy mix it was meant to
show.
audit-repeats-predate-determinism:scripts/attractor_audit.shandthe weekly probe ran every seed twice, a habit from when same-seed
runs diverged. Headless runs have been deterministic since Make headless runs deterministic: fixed frame clock and one ordered tick #31
("Headless runs are deterministic" in
docs/DECISIONS.md), everyaudit since has been read as one run per seed, and nothing compared
the pairs, so half of each audit and eight of the sixteen weekly jobs
produced output nobody read.
attractor-audit-ignores-run-failures: the audit backgrounded eachrun and used a bare
wait, which returns 0, so a crashed run passedsilently and surfaced only as "?" cells in the summary table.
setup-echo-pre-push-stale:scripts/setup.shtold the developerthe pre-push hook runs
scripts/test.sh;lefthook.ymlrunsscripts/pre-push-test.sh, which skips docs-only pushes.What changed
scripts/probe.shparsesGrazers,HuntersandOmnivores. Thesummary row, and the two table headers in
.github/workflows/probe.yml(matrix job and determinism job), nowhave ten columns: label, seed, ticks, population, plants, grazers,
hunters, omnivores, species, verdict.
docs/QUALITY.mdnames thecolumns and the summary lines they come from.
scripts/attractor_audit.sh:REPEATS=1), not two compared runs.docs/DECISIONS.mdsays a headless run is a function of the seedand flags, and assigns the ongoing same-seed check to the CI
determinism probe. The audit's own seed-42 probe pair stays, and
scripts/attractor_audit_summary.pystill compares it.whose summary has no
Total organisms (final):line is logged asFAIL <label> (...)when it ends; the rest of the audit stillruns; the script then exits 1 and names every failed run on stderr.
An extinct population is still a result, not a failure.
launched together at the default
PARALLEL=4and runs sequentiallyat
PARALLEL=1. File names (seed<S>-run<R>,seed42-runprobe-a/b) are unchanged..github/workflows/probe.yml, in its own commit: the weekly scheduleand the
workflow_dispatchdefault run each seed once (count=1,eight jobs instead of sixteen), matching the audit. The
runsinputstill accepts 2. Triggers, schedule, permissions and the determinism
job are unchanged.
docs/QUALITY.mdis updated to match.scripts/setup.shnamesscripts/pre-push-test.shas the pre-pushhook and says it skips docs-only pushes.
README.md: the audit command's comment reads "8 seeds x 1 run x 15kticks". The Measurement section of
docs/design/simulation-rules.mdnames the two audits that ran each seed twice (2026-09-18 and the
phase 1 audit) and says the audit now runs each seed once.
scripts/attractor_audit_summary.py: the probe line reads"Determinism probe (two runs of the same seed)", since the pair is
launched together only when
PARALLELis above 1.review/BACKLOG.md: theprobe-and-audit-scriptsentry is removed.Markers
Type: tooling
Resolves review backlog: probe-and-audit-scripts
Resolves review finding: probe-summary-forager-predator-columns-dead
Resolves review finding: audit-repeats-predate-determinism
Resolves review finding: attractor-audit-ignores-run-failures
Resolves review finding: setup-echo-pre-push-stale
Validation
None of these change the simulation, so the checks are command runs
and inspections. Every run below is short and headless; nothing opens
a window. Run from the repository root after
cargo build --release.Probe columns (
probe-summary-forager-predator-columns-dead):Before and after, the row reads:
The three new numbers match the
Grazers:,Hunters:andOmnivores:lines under "Final strategy breakdown" in/tmp/probe-check/seed42.txt. The Probe workflow runs on thispull request because it changes
probe.yml; each matrix job's stepsummary should show a ten-column table with a number in every cell.
Remove
/tmp/probe-checkafterwards.Audit failures (
attractor-audit-ignores-run-failures). Make everyrun fail fast with a flag the binary rejects:
Before, this printed
audit completeandexit 0. Now each runlogs
FAIL seed<S>-run<R> (exit 2)and the script ends withaudit FAILED: 10 run(s) failed: ...andexit 1.For a crash in the middle of an otherwise healthy audit, start a
background killer that waits for seed 2's run and kills it, then the
audit at 1500 ticks (each run takes a few seconds, too short to kill
by hand):
The audit should log
FAIL seed2-run1 (exit 137), carry on throughthe remaining seeds and the probe pair, and end with
audit FAILED: 1 run(s) failed: seed2-run1andexit 1.One run per seed (
audit-repeats-predate-determinism):Expect
exit 0,README.txtsaying1 run(s) each, oneseed<S>-run1.txtand.csvper seed and norun2files, plus theseed42-runprobe-a/bpair. The summary prints one row per seed andends with
Determinism probe (two runs of the same seed): identical histories.The trailing
1(the script's PARALLEL argument) runs the pair oneafter the other, which keeps this check to one process at a time.
In
.github/workflows/probe.yml, theplanjob'sscheduleand
workflow_dispatchbranches setcount=1.Remove
/tmp/audit-*afterwards.setup.sh hook list (
setup-echo-pre-push-stale). Comparetail -4 scripts/setup.shwith thepre-pushblock inlefthook.yml: both namescripts/pre-push-test.sh.Supporting checks:
shellcheckis clean onscripts/probe.sh,scripts/attractor_audit.shandscripts/setup.sh;actionlint .github/workflows/probe.ymlis clean;/bin/bash -n(bash 3.2) parses the audit script;
bash scripts/check.shpasses.