feat(sync): tell a forked child the truth about its inherited handles - #230
Merged
Conversation
This was referenced Aug 25, 2026
fork copies the address space but not the threads, so a child inherits every handle, producer and consumer the parent held, and none of the runtime thread that makes them work. The failure was silence rather than a crash: the child's publish returned Ok and the value went into a buffer nobody drains — the same failure the graph-start gate exists to prevent, reappearing on the other side of a fork. Worse across a C ABI than in Rust, where a daemon that double-forks or a supervisor that forks per job is an ordinary shape. There was a second failure behind it. A child that runs a destructor joins a JoinHandle for a thread that does not exist in this process, which panics inside std with "threads should not terminate unexpectedly" — a backtrace on fd 2 from inside a destructor. detach and Drop now release the handle instead. Detection is a pthread_atfork child handler, not a pid comparison: measured, std::process::id() is 321ns against a 121ns try_set, so reading the pid per publish would cost more than twice the work it guards. The check that sits on the hot path is a relaxed atomic load. A generation counter rather than a flag, because a bool would poison the child permanently — including for a database the child itself attaches afterwards. The guard is also placed before the Weak upgrade, deliberately: a forked child's upgrade succeeds, since the Arc was copied with the address space. fork is std-gated and libc is optional behind the std feature, so the no_std graph pulls neither. The Makefile assertion that used to grep only for tokio now covers libc too — libc defaults to a std feature, so a target-specific dependency added for a std-only path would otherwise un-no_std the crate silently. SyncError::ForkedChild is additive rather than breaking because SyncError became #[non_exhaustive] in #228. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hild `fork::generation` is public so a layer above this crate can stamp state of its own, but the `pthread_atfork` handler was armed only by `attach`. An FFI door opens before any database exists, so a caller stamping there was handed a number that could never change: a `fork` in that window went uncounted and the stamp compared equal forever. That is the silent-success failure this module exists to prevent, reappearing one layer up. `generation` now arms the handler itself. The counter read is split out into a private `load`, and `forked_since` uses it directly, so the publish path is still one relaxed load — arming stays on the cold construction-time call. The regression test lives in its own binary on purpose: any `attach` anywhere in the process arms the handler, so sharing a binary with the rest of the fork suite would make it pass whether or not `generation` arms. Verified by reverting the fix — it fails there and passes here. Also: - `ForkedChild` was in neither the `kind()` test nor the `SyncError` list in the crate docs, the only variant missing from both. Added to each. No intra-doc link, since `fork` is std-only and the link is unresolved in the `--no-default-features` doc build. - Cover an inherited `SyncConsumer`: all five read methods refuse, and it drops without blocking or panicking. The guard was implemented but untested, so a refactor could have dropped it silently. - The `SAFETY` note on `in_forked_child` claimed the child was effectively single-threaded. It is not — the parent runs the runtime thread and the test harness, so the child inherits allocator locks held by threads that no longer exist. State the residual risk rather than argue it away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKAH7JNjPLPmG4mPTvWfth
…t hung The first CI run of the fork suite deadlocked in `a_child_can_attach_its_own_database_after_forking` and sat there until GitHub's six-hour job ceiling killed it, taking "Comprehensive Development Check" with it (it `needs:` that job). The suite has never gone green. `fork` in a multi-threaded process leaves the child holding every lock that was held by a thread which did not come across — the allocator's above all. Proving a post-fork `attach` works means allocating and spawning a thread, so that one case cannot stay async-signal-safe; sharing a binary with the rest of the suite ran it beside several live Tokio runtimes, which is what made the inherited-lock window wide enough to hit. Two changes, neither of which skips the test: - A watchdog. `in_forked_child` now polls with `WNOHANG` and, after `CHILD_TIMEOUT`, kills the child and fails. A blocking `waitpid` is what turned a deadlock into a six-hour job; the worst case is now a fast, legible failure. Verified by hanging a child on `pause()`: it is killed, the test fails with the timeout message, and no zombie is left. - Isolation. The post-fork `attach` case gets its own binary, where the parent attaches nothing before forking and holds only the harness main thread and the test thread — the quietest parent the assertion can be made from. The shared helper moves to `tests/fork_child/mod.rs` so all three fork binaries get the bound, and carries the reasoning where the next reader of these tests will find it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKAH7JNjPLPmG4mPTvWfth
lxsaah
force-pushed
the
feat/sync-fork-safety
branch
from
August 26, 2026 14:35
f0bf83d to
4338210
Compare
… guesswork The watchdog bounded the six-hour CI hang but left the test failing far more often than "rare": 11 of 60 runs. One green CI run was a lucky draw on an 18% failure rate, not evidence the suite was healthy. Measuring instead of arguing, 60-120 runs per variant: | variant | hangs | |---|---| | as written, tests in parallel | 11/60 | | as written, --test-threads=1 | 17/60 | | children leak instead of freeing | 1/60 | | children leak where they can | 11/120 | | children leak where they can, plus settle | 0/120 | Harness parallelism is not the driver: serialising made it no better. The trigger is the allocator lock — the child takes it on every free, and the parent's own freshly-spawned runtime threads are what hold it. So both ends are addressed. Children now mem::forget their inherited state wherever the destructor is not what is under test; they _exit immediately, so nothing is lost, and freeing was the unsafe act. And the parent waits for its runtime threads to finish starting and park before forking, because fork is only safe from a quiescent parent and a test that just called attach is the opposite of one. Alone the first halves the rate; together they reach zero over 120 runs per binary, 360 runs total. Two comments claimed things the data refutes, and are corrected rather than left to mislead: - The SAFETY note recommended --test-threads=1. It measurably does not help. - The post-fork-attach test's own binary was justified by isolation from other tests' runtimes. Parallelism does not matter, so that was the wrong reason. The right one is that the test attaches nothing before forking, so its parent has no runtime thread to be mid-allocation — a property a single attach anywhere else in the process would destroy. The watchdog stays. dropping_an_inherited_handle_does_not_panic must free to test what it tests, and the settle is a duration rather than a handshake, so neither guarantee is absolute on a slower machine. What it converts is a silent six-hour hang into a bounded failure, and that should not depend on the rate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKAH7JNjPLPmG4mPTvWfth
`fork::generation` and `fork::forked_since` were made public earlier in this PR on the argument that a facade built on this crate has the same problem for the same reason. The argument is sound but the caller is hypothetical: nothing in the workspace uses them. The four example crates that depend on aimdb-sync never mention fork, the only C-ABI crate does not depend on aimdb-sync, and no FFI layer exists here. The one external user was this crate's own integration test. Publishing them would commit the crate in semver to the stamp-and-compare model — record a number at construction, compare it later — chosen against no real caller, and that is the shape most likely to change: the same fact is currently copied into three types and checked by nine call sites that each have to remember to. Widen it when something real needs it, so the API can be designed against that caller instead of guessed at. The arming regression test moves into the module as a unit test, which is where it belongs anyway. Its precondition is that nothing in the process has armed the handler yet, and the lib test binary holds only the compile-time Send/Sync markers and the SyncError::kind tests — none attach a database. An integration test cannot have that precondition once it shares a binary with the fork suite, whose first attach arms the handler; the earlier version of this test passed against broken code for exactly that reason. Re-verified by reverting the arming fix: it fails there and passes here. Its waitpid is bounded with a SIGKILL fallback. That child reads one atomic and _exits so it cannot deadlock, but "cannot deadlock" is what was said about the fork suite before it hung a CI job for six hours, and the bound costs nothing. Soaked 120 runs: no failures. Not a breaking change: these items are introduced and withdrawn inside this same unmerged PR, so no released version ever exposed them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKAH7JNjPLPmG4mPTvWfth
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
forkcopies the address space but not the threads. A child inherits everyAimDbHandle,SyncProducerandSyncConsumerthe parent held — and none of the runtime thread that makes them work.The failure was silence, not a crash: the child was told the database was open, its
publishreturnedOk, and the reading was discarded. That's the same failure the graph-start gate exists to prevent —set()returningOkinto a buffer nobody drains — reappearing on the other side of afork. It's worse across a C ABI than in Rust, where a daemon that double-forks or a supervisor that forks per job is an ordinary shape rather than an exotic one.There were two failures, not one
The original reproduction ended the child with
_exit(0), so it never ran a destructor. Running one finds the second:A forked child holds a
JoinHandlefor a thread that does not exist in this process, and joining it panics insidestd. A Rust backtrace on fd 2 from inside a destructor — and this predates the recent work; the pre-#226 code joined on a helper thread too.Detection: measured, not assumed
The obvious check is a pid comparison. It is far too expensive for where it has to sit:
try_set()std::process::id()So detection is a
pthread_atforkchild handler, and the check on the publish path is a relaxed atomic load. Re-measured after the change,try_set()is 107 ns — the guard does not show up.A generation counter, not a flag
A
boolwould poison the child permanently, including for a database the child itself attaches afterwards — a supervisor that forks per job and then does its own work would find the API dead for no reason.GENERATIONis anAtomicU64the child handler increments; anything made after the fork is fine. There's a test for exactly that.The guard sits before the
Weakupgrade, deliberately: a forked child's upgrade succeeds, because theArcwas copied with the address space. That is precisely why the buffer would otherwise accept a value nobody will read.On installing a process-global
This crate's own rule is that no aimdb library installs a process-global. This installs one, and I think it's the exception the rule anticipates, on two conditions that are both met:
attach, never at load time. A program that never uses the sync facade never gets one.The handler does a single relaxed
fetch_add, which is permitted in a fork handler.fork::generationandfork::forked_sinceare public, which was not the original plan. A facade built on this crate has the same problem for the same reason — it must answer "is this still usable" without taking a lock the runtime thread might hold — so the two query functions are part of the surface rather than an internal detail.no_std, and the guard that enforces it
forkis#[cfg(feature = "std")]-gated andlibcis optional behind thestdfeature, so the--no-default-featuresgraph pulls neither.I widened the Makefile assertion that covers this. It grepped only for
tokio;libcdefaults to astdfeature, so a target-specific dependency added for a std-only path un-no_stds the crate silently — the tree check would have passed while the build broke. It's now a list (SYNC_NO_STD_FORBIDDEN := tokio|libc), and I negative-tested it by reintroducing the leak:Tests
tests/fork_safety_test.rsactually forks, which is what the acceptance criterion asked for:set/try_setare refused, and the parent keeps publishing across the forkdetachin a child is refused rather than fatal, and an inherited handle left to drop returns quietly (the parent assertsWIFEXITED, so a panic in the destructor fails the test)Note on layering
SyncError::ForkedChildis additive rather than breaking becauseSyncErrorbecame#[non_exhaustive]in #228. That ordering was deliberate — reversed, this would have been a breaking change for a bug fix.One thing left open: which layer answers first is a diagnostic question this doesn't settle. A facade's own closed-check can win over aimdb's more specific message, so a caller may read "this station is closed" rather than "created before a fork()". Closing that gap means teaching a third layer about forks to improve one string, which didn't seem worth it.
Related Issue
#[non_exhaustive]) and fix(sync): remove both busy-waits, and report why startup failed #226.Checklist
make check) — relying on CI for the full matrix;cargo test -p aimdb-sync(std and no_std), both clippy legs, and the Makefile no_std guard are green locally.