forkrun v3.5.1 - #544
Merged
Merged
forkrun v3.5.1#544
Conversation
… hygiene Verification log for the porting-plan (v1.3 §2.0) preconditions, plus the two repo-hygiene defects that gate surfaced. - docs_port/GATE_v3.5.x.md: the preconditions-gate log. Build gate (product loadable, link-checked canary, Stage-0 python surface, wrapper syntax, frun.bash<->nob64 pre-blob parity); host behavioural matrix (UMA / --nodes=2 / --nodes=4 / byte mode); unit suites (test_frun.sh 89/89, test_frun_comprehensive.sh 254 total / 248 pass, SECTION=R 13/13 with THP enabled); the aarch64 leg (native arm64 build + targeted NUMA/byte/ordered/ resume-exactly-once, both UMA and NUMA); and an explicit section on what the log does NOT prove (qemu is not silicon, one physical NUMA node, no sanitizers, no TUI, THP sensitivity). - The six comprehensive-suite failures observed were the sweep tests, and were an environment artifact, not a regression: pristine v3.5.0 and current produce byte-identical output (6 lines with 2>/dev/null, 16 with 2>&1 -- the 10 extra being the THP NOTICE), the sweep tests are exactly those that capture 2>&1, and UNIT_TESTS/test_all.sh:3 sets shmem_enabled=always before running. With 'always', SECTION=R is 13/13. - Makefile.substrate: fix a false green. make tracks timestamps, not command lines, so `canary CC=aarch64-linux-gnu-gcc` printed "canary OK" without relinking anything, validating a stale x86-64 artifact. Adds a compiler-identity stamp (forces a rebuild when CC/CFLAGS change) and an architecture assertion (hard failure if the artifact's ELF machine does not match the requested compiler); OUT is now overridable so per-arch canary builds stop clobbering each other. - Untrack artifacts that every test/CI path regenerates: libforkrun.so and the canary objects, the c_plugin_test .so files plus its copied forkrun_plugin.h and generated input_*.txt, python __pycache__, and UNIT_TESTS/.forkrun_resume (per-run runtime state, not repo content). .gitignore rules are deliberately surgical rather than a blanket *.so: ring_loadables/forkrun-libs/*.so are CI-built per-arch blobs and ring_loadables/forkrun_plugin.h is the frozen ABI single source -- a blanket rule would silently drop a newly added arch's blob from a CI PR. - python/forkrun/__init__.py: document the deliberate namespaced shadowing of forkrun.map and the private validation re-export (review nit).
…44035994 🤖 Auto-Build: Update Ring Loadables
Write site in MAIN block after trap installation, before fd_spawn pipe: FORKRUN_TEST_CLEANROOM_PIDFILE doubles as readiness signal (traps guaranteed installed when file appears). Rewrite R2/R10 to wait on the file (10s bounded) and signal $(cat pidfile); drop ps --ppid/CPID+3 arithmetic and fixed sleep. Add --nodes=@2 to both so the SLURM path runs indexers, locking in D6 exit-code preservation (143/138 not 1).
core_scanner_loop: over-claim of the victim's queue now 'continue's to
re-check the own queue instead of exiting; termination routes through
the Instant NUMA Tear-down path (converges at global EOF). One statement
plus rewritten comment; no other loop changes.
Lock-in tests F15a (chunk-conservation matrix {file,pipe}x{@2,@4}x
{default,-s}) and F15b (10x 1M-line EOF-herd stress) in the same commit.
Also mirrors the missing F30 entry into DOCS_ALL.md (twin sync).
The handler ran 'exec {fd_indexer_death_r}‹- 2>/dev/null' as a bare exec,
so the redirect persisted in the main shell and swallowed all post-reactor
stderr (telemetry, verbose, checkpoint hints) while stdout stayed byte-exact
and rc stayed 0. One-line fix matching SCAN_DEATH's close form; F15a/F15b
now require Node frames in every combo/iteration; changelog flake-caveat
replaced with D8 entry.
Per owner determination the two constructions behave identically under restricted bash; the final code uses PATH=''. Rewrites the F29-F6 comment that claimed a proven self-recursing touch probe (unverified, contradicts the determination) with the honest decision record. All four sites (main sandbox + re-render, both twins).
T14 called bare 'frun' in the suite shell where it is undefined (rc=127, no ERROR line, permanent red). Route all three sub-cases through the suite convention: bash -c "source '$FRUN_SOURCE' && frun ...". Verified @513/@abc: rc=1, [ERROR] on stderr, no checkpoint.
…elog New lock-in tests (both twins): T1g (positional forgery rejected fail-closed), T1h (cmdline-token forgery neutralized by re-render, benign ORIG_ARGS for deterministic termination), T1i(i) (plain setup declare reaches layer-3 gate, no shape overblock), T1i(ii) (escaped substitution accepted as literal), T1a-ext (double-forked delayed orphan with token knowledge beats legit emission via greedy-anchor then neutralized; SECONDS-delay, order-independent assertions; tripwire run proved the orphan frame wins), F6 (CWD-planted exec-form touch characterization, record-only). T1a stale EXPECTED-TO-FAIL note corrected. SECURITY.md (+DOCS_ALL mirror): sandbox-before-ownership ordering, token secrecy not a property, F6 result (2026-09-16, bash 5.3.9: empty PATH resolves CWD ./touch; /nonexistent does not; no default-PATH fallback), pre-consent execution bounded to same-UID tampering. F29 changelog entry (both copies). D9 sandbox comment corrected for F6 (construction unchanged). Verified: SECTION=T 21/21, SECTION=M 24/24 (M20/M21 post-PATH-revert), SECTION=T2 16/16.
Adds fr_simd_available() (TLS-cached probe factored WITHOUT touching try_simd_scan — byte-identical) and -L-only scan_nth_delim(); rewrites the Scanner-Handoff Chain inner loop to claim need-th delimiters with -n budget clamping, single tail-count, unchanged flush sites/invariants. Verified on locally-built v4 blobs (identical flags; local-before matches shipped size 203912): F7 exact, F 7/7, T2 16/16, -L+-n bit-identical x4 shapes; F8 pins the budget arm (green on shipped blob too). Before/after 100M-line table in changelog: perf-neutral (scan not binding here); BORN_LOCAL_NUMA s5 stands. Blobs NOT committed — CI rebuilds in W-I.
Embed job ran set +e with a pass-on-stale Verify step: a silently failed update_frun_base64 merged blobs no longer matching the built artifacts. New hard gate before the PR step: for each arch with a built artifact, the embedded entry must decode (repo decoder, checksums enforced) to bytes identical to the artifact; mismatch always fails; missing artifact fails on tag refs, warns otherwise. NOTE: sketch asked for regen-compare, but the encoder gzip-embeds an mtime so regen never compares equal (demonstrated); decode-and-cmp proves the same correspondence deterministically. Plus a canary step (apt libbash-dev + make canary). All four gate paths proven locally: match-OK, stale-FAIL, tag-missing-FAIL, corrupt-FAIL.
Tests (both twins): delete simple M17/M20/M21 (diagnostic variants kept), both always-green T10b_diag blocks (diagnostics folded into a single DEBUG-on T10b failure path), M20 stray debug lines (-6 tests); reconcile three pre-existing test-twin drifts toward the newer variant (T1f shadow/grep, T13 killer fraction) — test twins now byte-identical. Verified SECTION=M 21/21, T2 13/13. Docs: INVARIANTS s14-17 renumbered to s13-16 (+v3.5.0 changelog ref updated to current numbering by convention); F3 total-vs-useful labeling; F38 1B-line run-length qualifiers (README/OVERVIEW/DOCS_ALL mirrors). CI: twin-check job (frun lockstep modulo the single blob line — literal cmp would false-fail by construction — plus literal test-twin cmp). GATE log: rounds 8-14 appended, does-NOT-prove list kept current.
Changelog: W-B mechanism entry, test-inventory consolidation, retitle v3.5.x (unreleased) -> v3.5.1 2026-09-17 (both copies). Bump checklist, all sites: -V echo (both frun twins), FORKRUN_RING_VERSION fallback, META, both suite pins (+comprehensive .txt twin). Historical v3.5.0 comment references intentionally untouched. frun -V now reports v3.5.1.
…tion POSIX PATH search treats an empty component as CWD, so PATH='' is not dead (F6 probe). Both restricted shells now share one mktemp-created, immediately-deleted directory PATH. F6 upgraded to hard assertion. SECURITY/CHANGELOG + DOCS_ALL twins in lockstep. Verified: SECTION=T 21/21, M 21/21 (M20/M21 green), F 8/8 (F8 green).
The frun twins carry different embedded-b64 payloads by construction; compare only the pre-marker region instead of sed-normalizing one line. Test twins stay byte-identical. Verified locally.
ubuntu-latest runners carry no libbash-dev; the pipeline is fedora-based. First live canary run failed on config.h for this reason only.
ubuntu runners carry no bash-headers package (libbash-dev absent; first live canary run failed on config.h). Fedora for everything: native x86_64 container, dnf bash-devel, same make target the build jobs' image already satisfies. Artifacts are gitignored.
🤖 Auto-Build: Update Ring Loadables
Owner's manual matrix (baselines/sanitizers/aarch64/twins/tag), blob provenance (run 35173548966, PR #534, smoke results), twin-check green in CI (run 35174209682), sh-checker red noted as pre-existing.
…75093479 🤖 Auto-Build: Update Ring Loadables
v3.5.0 review finding, still present: a positive-short sendfile fell through to a full-range copy, duplicating the emitted prefix. Resume at (off+s, len-s) on s>0; full copy on s<0 non-EPIPE (zero progress). Harness: old +64 dup, fixed byte-exact, hard-fail exact both; basic suite 89/89 on locally built v4 blob.
🤖 Auto-Build: Update Ring Loadables
Plain --nodes=2 degrades to UMA on single-socket hosts, never entering the F28 is_numa handoff arms the test claims to lock in. Verified F8 still green with @2 on x86_64 (SECTION=F 8/8).
…findings record LA3 wedged on every budget: line-args sleep sums batch lines as durations (batch 1 ~= 84 min). Redesign (test-side only, both twins): printf payload, endless SIGPIPE-clean feeder, liveness-gated kill, five-fact failure capture. 3x standalone green + Section L 57/57. CHANGELOG/DOCS_ALL twins + GATE 18 in lockstep.
The reactor's local _ret_val/_fr_signalled reset trap-recorded signal state for signals landing in the spawn window. Init before the pidfile write in both frun twins; x86_64 R 13/13 unchanged, emulated R 11/13 -> 13/13. GATE 19.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @jkool702, your pull request is larger than the review limit of 150,000 diff characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
💡 Codex ReviewLine 535 in a22b0a4  Restore the trust gate before evaluating resume data
When forkrun/.github/workflows/forkrun_release.yml Lines 5 to 8 in a22b0a4  Trigger release builds for compiled header changes
A push that changes only ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…ntain permissions' make github token read-only Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…81542589 🤖 Auto-Build: Update Ring Loadables
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
forkrun v3.5.1 — bugfix & hardening release
No new features, no perf changes, no breaking changes. 5,008 test
executions, all green: both suites × x86_64 UMA/NUMA ×
{baseline, TSan, ASan/UBSan} + 396-run benchmark matrix + C-plugin
suite + QEMU aarch64 (full suite, first time — previously aarch64
builds were only compile-time checked).
Highlights:
instead of hanging silently (kernel-level death detection)
RISC-V) — static + functional verified; silicon validation pending
scraping, forged frames, CWD-planted PATH binaries all closed);
prompts preview what will actually run
Full notes + leg logs + gate record: docs_port/