Skip to content

Fix the probe's dead columns and the audit's ignored failures and redundant repeats - #109

Merged
vhata merged 9 commits into
mainfrom
review/probe-and-audit-scripts
Nov 9, 2026
Merged

vhata merged 9 commits into
mainfrom
review/probe-and-audit-scripts

Conversation

@vhata

@vhata vhata commented Nov 9, 2026

Copy link
Copy Markdown
Owner

Why

Four small tooling defects from the 2026-10-09 full review, all in the
scripts that watch the simulation rather than in the simulation itself.

  • probe-summary-forager-predator-columns-dead: scripts/probe.sh
    parsed Foragers: and Predators: from the headless summary, which
    has printed Grazers:, Hunters: and Omnivores: since the diet
    axis landed (Add the diet trait and four-way strategy instruments with no behaviour change #18). Two of the probe's six numbers were always "?", so
    the run page of every probe job hid the strategy mix it was meant to
    show.
  • audit-repeats-predate-determinism: scripts/attractor_audit.sh and
    the weekly probe ran every seed twice, a habit from when same-seed
    runs diverged. Headless runs have been deterministic since Make headless runs deterministic: fixed frame clock and one ordered tick #31
    ("Headless runs are deterministic" in docs/DECISIONS.md), every
    audit since has been read as one run per seed, and nothing compared
    the pairs, so half of each audit and eight of the sixteen weekly jobs
    produced output nobody read.
  • attractor-audit-ignores-run-failures: the audit backgrounded each
    run and used a bare wait, which returns 0, so a crashed run passed
    silently and surfaced only as "?" cells in the summary table.
  • setup-echo-pre-push-stale: scripts/setup.sh told the developer
    the pre-push hook runs scripts/test.sh; lefthook.yml runs
    scripts/pre-push-test.sh, which skips docs-only pushes.

What changed

  • scripts/probe.sh parses Grazers, Hunters and Omnivores. The
    summary row, and the two table headers in
    .github/workflows/probe.yml (matrix job and determinism job), now
    have ten columns: label, seed, ticks, population, plants, grazers,
    hunters, omnivores, species, verdict. docs/QUALITY.md names the
    columns and the summary lines they come from.
  • scripts/attractor_audit.sh:
    • One run per seed (REPEATS=1), not two compared runs.
      docs/DECISIONS.md says a headless run is a function of the seed
      and flags, and assigns the ongoing same-seed check to the CI
      determinism probe. The audit's own seed-42 probe pair stays, and
      scripts/attractor_audit_summary.py still compares it.
    • Each batch is waited on PID by PID. A run that exits non-zero or
      whose summary has no Total organisms (final): line is logged as
      FAIL <label> (...) when it ends; the rest of the audit still
      runs; the script then exits 1 and names every failed run on stderr.
      An extinct population is still a result, not a failure.
    • The probe pair goes through the same batching, so it is still
      launched together at the default PARALLEL=4 and runs sequentially
      at PARALLEL=1. File names (seed<S>-run<R>,
      seed42-runprobe-a/b) are unchanged.
  • .github/workflows/probe.yml, in its own commit: the weekly schedule
    and the workflow_dispatch default run each seed once (count=1,
    eight jobs instead of sixteen), matching the audit. The runs input
    still accepts 2. Triggers, schedule, permissions and the determinism
    job are unchanged. docs/QUALITY.md is updated to match.
  • scripts/setup.sh names scripts/pre-push-test.sh as the pre-push
    hook and says it skips docs-only pushes.
  • README.md: the audit command's comment reads "8 seeds x 1 run x 15k
    ticks". The Measurement section of docs/design/simulation-rules.md
    names the two audits that ran each seed twice (2026-09-18 and the
    phase 1 audit) and says the audit now runs each seed once.
  • scripts/attractor_audit_summary.py: the probe line reads
    "Determinism probe (two runs of the same seed)", since the pair is
    launched together only when PARALLEL is above 1.
  • review/BACKLOG.md: the probe-and-audit-scripts entry is removed.

Markers

Type: tooling
Resolves review backlog: probe-and-audit-scripts
Resolves review finding: probe-summary-forager-predator-columns-dead
Resolves review finding: audit-repeats-predate-determinism
Resolves review finding: attractor-audit-ignores-run-failures
Resolves review finding: setup-echo-pre-push-stale

Validation

None of these change the simulation, so the checks are command runs
and inspections. Every run below is short and headless; nothing opens
a window. Run from the repository root after cargo build --release.

  1. Probe columns (probe-summary-forager-predator-columns-dead):

    scripts/probe.sh 42 200 /tmp/probe-check && cat /tmp/probe-check/summary-row.md
    

    Before and after, the row reads:

    before: | seed42 | 42 | 200 | 932 | 196 | ? | ? | 55 | ok |
    after:  | seed42 | 42 | 200 | 932 | 196 | 655 | 20 | 61 | 55 | ok |
    

    The three new numbers match the Grazers:, Hunters: and
    Omnivores: lines under "Final strategy breakdown" in
    /tmp/probe-check/seed42.txt. The Probe workflow runs on this
    pull request because it changes probe.yml; each matrix job's step
    summary should show a ten-column table with a number in every cell.
    Remove /tmp/probe-check afterwards.

  2. Audit failures (attractor-audit-ignores-run-failures). Make every
    run fail fast with a flag the binary rejects:

    EXTRA_ARGS=--no-such-flag scripts/attractor_audit.sh /tmp/audit-fail 10 1; echo "exit $?"
    

    Before, this printed audit complete and exit 0. Now each run
    logs FAIL seed<S>-run<R> (exit 2) and the script ends with
    audit FAILED: 10 run(s) failed: ... and exit 1.

    For a crash in the middle of an otherwise healthy audit, start a
    background killer that waits for seed 2's run and kills it, then the
    audit at 1500 ticks (each run takes a few seconds, too short to kill
    by hand):

    (for i in $(seq 600); do pkill -9 -f '/tmp/audit-kill/seed2-run1\.csv' && break; sleep 0.1; done) &
    scripts/attractor_audit.sh /tmp/audit-kill 1500 1; echo "exit $?"
    

    The audit should log FAIL seed2-run1 (exit 137), carry on through
    the remaining seeds and the probe pair, and end with
    audit FAILED: 1 run(s) failed: seed2-run1 and exit 1.

  3. One run per seed (audit-repeats-predate-determinism):

    scripts/attractor_audit.sh /tmp/audit-ok 300 1; echo "exit $?"
    scripts/attractor_audit_summary.py /tmp/audit-ok
    

    Expect exit 0, README.txt saying 1 run(s) each, one
    seed<S>-run1.txt and .csv per seed and no run2 files, plus the
    seed42-runprobe-a/b pair. The summary prints one row per seed and
    ends with
    Determinism probe (two runs of the same seed): identical histories.
    The trailing 1 (the script's PARALLEL argument) runs the pair one
    after the other, which keeps this check to one process at a time.
    In .github/workflows/probe.yml, the plan job's schedule
    and workflow_dispatch branches set count=1.
    Remove /tmp/audit-* afterwards.

  4. setup.sh hook list (setup-echo-pre-push-stale). Compare
    tail -4 scripts/setup.sh with the pre-push block in
    lefthook.yml: both name scripts/pre-push-test.sh.

Supporting checks: shellcheck is clean on scripts/probe.sh,
scripts/attractor_audit.sh and scripts/setup.sh;
actionlint .github/workflows/probe.yml is clean; /bin/bash -n
(bash 3.2) parses the audit script; bash scripts/check.sh passes.

vhata added 9 commits November 8, 2026 17:30
The headless summary has printed Grazers, Hunters and Omnivores since the
diet axis landed, so the probe's Foragers and Predators fields never matched
and both columns always read "?". Parse the three current labels and widen
the step-summary headers in probe.yml and the description in QUALITY.md to
match.
lefthook.yml runs scripts/pre-push-test.sh at pre-push, which skips
scripts/test.sh on docs-only pushes; setup.sh still told the developer the
hook was scripts/test.sh.
Headless runs have been deterministic since #31, so the second run of each
seed reproduced the first and every audit since has said "one run per
seed". REPEATS is now 1. The seed-42 determinism probe pair stays.

run_one was backgrounded and a bare wait returned 0, so a crashed or
truncated run passed silently and showed up only as "?" in the summary
table. Each batch is now waited on PID by PID; a run that exits non-zero or
ends without a final summary is reported as it happens, the rest of the
audit still runs, and the script exits 1 naming the failed runs. Extinction
is still a result, not a failure. The probe pair now goes through the same
batching, so PARALLEL=1 runs it sequentially.
The weekly schedule and the workflow_dispatch default ran each seed twice on
separate runners, and nothing compared the pair: headless runs have been
deterministic since #31, so the second run reproduced the first. Both now
default to one run per seed, matching scripts/attractor_audit.sh; the runs
input still accepts 2, and the determinism job remains the same-seed
comparison. Triggers, schedule and permissions are unchanged.
The audit launches the probe pair together only when PARALLEL is above 1.
…design doc

The Measurement section still said every phase audit runs each seed
twice and carries same-seed divergence as a caveat. Headless runs have
been deterministic since #31 and scripts/attractor_audit.sh now runs
each seed once.
@vhata

vhata commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

Review

Independent review by a separate Claude Opus 5.5 reviewer agent at
1551bde, the commit now at 86fda0a: approve with nits.

Verified: probe.sh's field patterns match the headless summary's
Grazers:, Hunters: and Omnivores: lines exactly once each (seed
5, 150 ticks, row matches the summary) and the description's seed-42
numbers reproduced; the row's ten columns line up with both probe.yml
headers, separators and the fallback row; actionlint is clean;
shellcheck is clean on the changed scripts. One run per seed agrees
with the DECISIONS determinism entry, and the CI determinism job is
unchanged. Under /bin/bash 3.2: a rejected flag at PARALLEL=2 fails all
ten runs and exits 1; a wrapper that exits 101 for one run and 0 with
no summary for another reports exactly those two; at most two headless
processes ran at once; a real kill -9 at 1500 ticks fails only that
run; a clean PARALLEL=1 run exits 0 with no run2 files. The previous
script exits 0 on the killed run, confirming the original bug.
setup.sh matches lefthook.yml; README and QUALITY.md match the new
behaviour. scripts/check.sh and check-pr-markers pass.

Findings: 2.

  1. Validation step 2's mid-run kill could not be done by hand: a
    300-tick run ends in under a second. Fixed in this description: the
    step uses 1500 ticks and a background polling killer, rerun by the
    coordinator (FAIL seed2-run1 (exit 137), audit FAILED, exit 1).
  2. scripts/attractor_audit_summary.py crashes with FileNotFoundError
    when a probe run failed before writing its CSV. Pre-existing and out
    of scope; more likely to be hit now that failures are reported.
    Filed in TODO.md as audit-summary-crashes-on-missing-csv.

Not checked: the GitHub Actions Probe run itself (the ten-column step
summary on a real runner); extinction handling (code reading only);
9e198bc, the one-line design-doc correction in
docs/design/simulation-rules.md, which was written after the review.

Post-rebase review

Independent review by a separate Claude Opus 5.5 reviewer agent at
9e198bc (rebased onto origin/main 6fef5a1, after the docket vendoring
and the October 24 review): approve with nits; fixes confirmed on the
head rebased onto 2a2dafb, approve.

Verified: the diff covers only the four current findings, removes only
this backlog entry and touches nothing under docket/; probe.sh matches
a real seed 5 summary and probe.yml's ten-column tables; a rejected
flag under /bin/bash 3.2 fails all ten runs with exit 1 (origin/main
exits 0); shellcheck and actionlint are clean; scripts/check.sh and
check-pr-markers --strict pass.

Findings: 3.

  1. docs/design/simulation-rules.md said the audits "through phase 1"
    ran each seed twice; only the 2026-09-18 and phase 1 audits did.
    Fixed in c1e0769.
  2. Validation steps set PARALLEL as an environment variable, which the
    script does not read. Fixed in the description.
  3. The previous review record listed 9e198bc as unchecked; superseded
    by this review.

Not checked: the Probe workflow on a real GitHub runner; extinction
handling (code reading only).

@vhata vhata added docket Made under the Docket workflow type:tooling Docket change type: tooling queue:review Docket: claims or resolves review backlog work area:tooling Docket queue area [TOOLING] priority:p3 Docket queue priority P3 labels Nov 9, 2026
@vhata
vhata merged commit 24d7a7c into main Nov 9, 2026
14 checks passed
@vhata
vhata deleted the review/probe-and-audit-scripts branch November 9, 2026 01:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tooling Docket queue area [TOOLING] docket Made under the Docket workflow priority:p3 Docket queue priority P3 queue:review Docket: claims or resolves review backlog work type:tooling Docket change type: tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant