the nightly benchmark stand: one box, every [benchmark] in the tree, plotted over commits

[aleksisch]

Why. Nothing measures how daslang's performance moves over commits. The tree holds 69 [benchmark] files and no rail that runs them on one machine every night.

What changes.

• A new tool runs every benchmark under benchmarks/ in the interpreter, JIT and AOT lanes, one file per process under a wall-clock and memory cap.
• A second verb turns the records into a static viewer and a markdown summary, and returns the night's verdict as its exit code.
• A regression needs a move past both 10% and 3x the baseline's own noise.
• A box-side driver builds the ref and publishes at /bench/; a nightly workflow drives it over ssh and posts the summary.
• The AOT lane measures native code: the benchmark bodies join DAS_AOT_SUITES, so their stubs land in test_aot, and the night builds it.
• A folder gate enforces the checklist's mechanical half.

Observable behavior.

• no performance history -> a chart per benchmark arm over commits in three tiers, plus a per-group index
• a broken benchmark goes unnoticed -> the nightly reds and names the file, the lane and the reason
• a night whose build fails leaves nothing -> the summary leads with the build log tail

Where to look. bench_runner.das polices the child, bench_history.das decides every verdict, nightly.sh is the privileged surface.

Validation, claims, ledger

Validation

• The three tiers were run against this tip on core/math/exp.das: interp 3.32 / 4.19 / 3.33 ns/op, jit 1.75 / 1.96 / 1.98, aot 2.99 / 1.62 / 3.00. That reproduces the shape plans/benchmark_followups.md already records for scalar exp through the vecmath lane - AOT level with or behind the interpreter on exp and exp_std, ahead of the JIT only on exp_est.
• test_aot links with the benchmark stubs in it (275 MB, 1080-odd TUs) and answers the lane's probe.
• 82 dastest arms pass (bin/daslang dastest/dastest.das -- --test utils/internal/bench-stand), reached per PR through run_utils_tests. The memory-cap arm needs perl, no longer /proc, so it runs on the darwin lane too. On a RelWithDebInfo host the process still exits 1 after the report: the C++ allocation tracker's exit-time leak dump does that to every suite in this repo, this one included.
• The two-command local flow was run on this branch tip: run --filter core/math/ --repeat 1 then report, both exit 0, 20 series, and the summary renders. That path found two defects now fixed - a meta with no build section was read as a failed build, and the run verb's exit code did not follow the record's status.
• The box pipeline was exercised locally in a scratch layout rooted at BENCH_STAND_HOME, against a worktree with a built binary: two clean nights, then a night whose build fails. The failed night published a summary leading with the CMake error, and the failed record was written by the tool. Linux x86_64, BENCH_STAND_BUILD=skip.
• Each of the seven REVIEW.das cells was negative-controlled: break the invariant once, confirm the finding, restore. Gate green on the branch.
• preflight fast tier: format, lint (both rails), hash-refs, review-md-tests, md-ascii, ast-verify, ci-das, ci-matrix all pass.
• Two preflight gates are red for reasons outside this change. review-md reports nine findings from utils/REVIEW.das, all about utils/ast-fuzz/ and utils/dasFormatter/ - untracked local directories on the author's box, one of them excluded through .git/info/exclude. Control: the same gate on a clean worktree of this branch prints REVIEW.das utils: OK. compile-sweep reports 122 of 761 program roots failing, 50 of them missing prerequisite 'sqlite', on a build configured -DDAS_SQLITE_DISABLED=ON -DDAS_LLVM_DISABLED=ON; none of the 122 is in this diff.
• The first CI run on this branch turned two macOS defects up, both now fixed and pushed. nightly.sh computed its build parallelism with a bare $(nproc) at load, which is GNU-only, so on the darwin lane every verb of the script died with exit 127 before reaching its work - the count now falls back to sysctl hw.ncpu and then to 4, verified by hiding nproc from PATH. And process_rss_mb truncated to whole megabytes, so a process under 1 MB read as 0; it rounds up now, which also makes a cap fire a hair early rather than late. The memory-kill arm itself passed on darwin, so the ps fallback works where the limit matters.
• The external reviewer round was skipped: no codex on PATH.
• The workflow itself has never executed. A schedule lane cannot fire on a branch, and workflow_dispatch needs the file on the default branch, so its first run is after merge - and it reds until the box is provisioned (below).

Claims - stated, not tested

• The JIT lane's availability probe reports the lane unavailable rather than failing the night. Verified with a stand-in binary that exits non-zero, and locally where -jit exits 1 under the allocation tracker; a break would show as a night with no JIT numbers and no reason recorded beside the lane.
• The ssh gate refuses everything but its four verbs and rejects arguments outside path characters. Verified by driving nightly.sh gate with hostile SSH_ORIGINAL_COMMAND values; a break would let a key holder run a shell on the web box.
• The -DDAS_LLVM_DISABLED=OFF build on the box fetches the prebuilt LLVM the CI linux lanes fetch. Not verified on Debian 13. A break records the JIT lane as unavailable with the reason, and the interpreter lane still produces a night.
• Timings quoted in README.md (a cold build of 20-40 minutes, a suite of 1-2 hours) are estimates from this author's box, not from the VPS.

Not attributed to this change

• build (windows, 32, Release, none) fails one arm of tests/module_cache/test_deferred_modules.das, about a module whose C++ dependencies are deferred. This diff adds no C++, touches no module-cache code and no file under tests/. I did not obtain a master-side control run, so this is an attribution by content, not by a red-on-master comparison.

Not done

• The box is not provisioned. Owed before the first night: generate an ed25519 key, sudo bench-stand-deploy.sh provision <key.pub>, sudo bench-stand-deploy.sh caddy, store the private half as the DASWEB_BENCH_KEY environment secret on github-pages, then dispatch the workflow once. README.md section 3 has the commands.
• Two benchmarks are broken on master and will red the first nights. benchmarks/decs/bench_from_decs_count.das does not compile (int += int64). benchmarks/core/array/test01.das allocates ~19 GB and is OOM-killed; it is skipped in suite.json with that reason and shows as skipped every night until fixed. Neither is fixed here.
• site/app.js has no test rail. The repo has two that would fit - node --test on the pure helpers, as utils/internal/dasweb-verify/browser/ does, and Playwright against the static page, as site/tests/playground/ does - and neither is wired up, so the viewer's DOM branches are unproven.
• bench-stand-deploy.sh has no test: its testable core needs a splice-into-a-fixture entry point first, the shape utils/internal/dasweb-buildd/roll_toolchain.sh uses.
• Four gate cells were proposed and not written: run-record fields against their readers, a temp-path literal check in the test files, and two more the round noted. The checklist carries the prose instead.