Skip to content

feat(sync): tell a forked child the truth about its inherited handles - #230

Merged
lxsaah merged 5 commits into
mainfrom
feat/sync-fork-safety
Aug 26, 2026
Merged

feat(sync): tell a forked child the truth about its inherited handles#230
lxsaah merged 5 commits into
mainfrom
feat/sync-fork-safety

Conversation

@lxsaah

@lxsaah lxsaah commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Description

fork copies the address space but not the threads. A child inherits every AimDbHandle, SyncProducer and SyncConsumer the parent held — and none of the runtime thread that makes them work.

The failure was silence, not a crash: the child was told the database was open, its publish returned Ok, and the reading was discarded. That's the same failure the graph-start gate exists to prevent — set() returning Ok into a buffer nobody drains — reappearing on the other side of a fork. It's worse across a C ABI than in Rust, where a daemon that double-forks or a supervisor that forks per job is an ordinary shape rather than an exotic one.

There were two failures, not one

The original reproduction ended the child with _exit(0), so it never ran a destructor. Running one finds the second:

thread '<unnamed>' panicked at library/std/src/thread/lifecycle.rs:247:
threads should not terminate unexpectedly
   5: std::thread::lifecycle::JoinInner<T>::join
   7: aimdb_sync::handle::AimDbHandle::detach_internal

A forked child holds a JoinHandle for a thread that does not exist in this process, and joining it panics inside std. A Rust backtrace on fd 2 from inside a destructor — and this predates the recent work; the pre-#226 code joined on a helper thread too.

Detection: measured, not assumed

The obvious check is a pid comparison. It is far too expensive for where it has to sit:

cost
try_set() 121 ns
std::process::id() 321 ns — 265% of a publish
relaxed atomic load ~0 ns

So detection is a pthread_atfork child handler, and the check on the publish path is a relaxed atomic load. Re-measured after the change, try_set() is 107 ns — the guard does not show up.

A generation counter, not a flag

A bool would poison the child permanently, including for a database the child itself attaches afterwards — a supervisor that forks per job and then does its own work would find the API dead for no reason. GENERATION is an AtomicU64 the child handler increments; anything made after the fork is fine. There's a test for exactly that.

The guard sits before the Weak upgrade, deliberately: a forked child's upgrade succeeds, because the Arc was copied with the address space. That is precisely why the buffer would otherwise accept a value nobody will read.

On installing a process-global

This crate's own rule is that no aimdb library installs a process-global. This installs one, and I think it's the exception the rule anticipates, on two conditions that are both met:

  • Only the crate that owns the runtime thread can know the thread is gone. Nobody above can perform this check, so it cannot be pushed to an FFI layer — which is where it would be a genuine trespass.
  • The handler is registered lazily, on the first attach, never at load time. A program that never uses the sync facade never gets one.

The handler does a single relaxed fetch_add, which is permitted in a fork handler.

fork::generation and fork::forked_since are public, which was not the original plan. A facade built on this crate has the same problem for the same reason — it must answer "is this still usable" without taking a lock the runtime thread might hold — so the two query functions are part of the surface rather than an internal detail.

no_std, and the guard that enforces it

fork is #[cfg(feature = "std")]-gated and libc is optional behind the std feature, so the --no-default-features graph pulls neither.

I widened the Makefile assertion that covers this. It grepped only for tokio; libc defaults to a std feature, so a target-specific dependency added for a std-only path un-no_stds the crate silently — the tree check would have passed while the build broke. It's now a list (SYNC_NO_STD_FORBIDDEN := tokio|libc), and I negative-tested it by reintroducing the leak:

✗ a std-only crate leaked into the no_std build
└── libc feature "default"
    └── libc feature "std"

Tests

tests/fork_safety_test.rs actually forks, which is what the acceptance criterion asked for:

  • an inherited producer's set/try_set are refused, and the parent keeps publishing across the fork
  • an inherited handle hands out no new producers or consumers — otherwise the guard is bypassed by making a fresh one in the child
  • detach in a child is refused rather than fatal, and an inherited handle left to drop returns quietly (the parent asserts WIFEXITED, so a panic in the destructor fails the test)
  • a database the child attaches after the fork works normally

Note on layering

SyncError::ForkedChild is additive rather than breaking because SyncError became #[non_exhaustive] in #228. That ordering was deliberate — reversed, this would have been a breaking change for a bug fix.

One thing left open: which layer answers first is a diagnostic question this doesn't settle. A facade's own closed-check can win over aimdb's more specific message, so a caller may read "this station is closed" rather than "created before a fork()". Closing that gap means teaching a third layer about forks to improve one string, which didn't seem worth it.

Related Issue

Checklist

  • I have read the CONTRIBUTING.md document.
  • My code follows the project's coding standards.
  • I have added tests to cover my changes.
  • All new and existing tests passed (make check) — relying on CI for the full matrix; cargo test -p aimdb-sync (std and no_std), both clippy legs, and the Makefile no_std guard are green locally.
  • I have updated the documentation accordingly.

lxsaah and others added 3 commits August 26, 2026 14:19
fork copies the address space but not the threads, so a child inherits every
handle, producer and consumer the parent held, and none of the runtime thread
that makes them work. The failure was silence rather than a crash: the child's
publish returned Ok and the value went into a buffer nobody drains — the same
failure the graph-start gate exists to prevent, reappearing on the other side
of a fork. Worse across a C ABI than in Rust, where a daemon that double-forks
or a supervisor that forks per job is an ordinary shape.

There was a second failure behind it. A child that runs a destructor joins a
JoinHandle for a thread that does not exist in this process, which panics
inside std with "threads should not terminate unexpectedly" — a backtrace on
fd 2 from inside a destructor. detach and Drop now release the handle instead.

Detection is a pthread_atfork child handler, not a pid comparison: measured,
std::process::id() is 321ns against a 121ns try_set, so reading the pid per
publish would cost more than twice the work it guards. The check that sits on
the hot path is a relaxed atomic load.

A generation counter rather than a flag, because a bool would poison the child
permanently — including for a database the child itself attaches afterwards.
The guard is also placed before the Weak upgrade, deliberately: a forked
child's upgrade succeeds, since the Arc was copied with the address space.

fork is std-gated and libc is optional behind the std feature, so the no_std
graph pulls neither. The Makefile assertion that used to grep only for tokio
now covers libc too — libc defaults to a std feature, so a target-specific
dependency added for a std-only path would otherwise un-no_std the crate
silently.

SyncError::ForkedChild is additive rather than breaking because SyncError
became #[non_exhaustive] in #228.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hild

`fork::generation` is public so a layer above this crate can stamp state of
its own, but the `pthread_atfork` handler was armed only by `attach`. An FFI
door opens before any database exists, so a caller stamping there was handed
a number that could never change: a `fork` in that window went uncounted and
the stamp compared equal forever. That is the silent-success failure this
module exists to prevent, reappearing one layer up.

`generation` now arms the handler itself. The counter read is split out into
a private `load`, and `forked_since` uses it directly, so the publish path is
still one relaxed load — arming stays on the cold construction-time call.

The regression test lives in its own binary on purpose: any `attach` anywhere
in the process arms the handler, so sharing a binary with the rest of the fork
suite would make it pass whether or not `generation` arms. Verified by
reverting the fix — it fails there and passes here.

Also:

- `ForkedChild` was in neither the `kind()` test nor the `SyncError` list in
  the crate docs, the only variant missing from both. Added to each. No
  intra-doc link, since `fork` is std-only and the link is unresolved in the
  `--no-default-features` doc build.
- Cover an inherited `SyncConsumer`: all five read methods refuse, and it
  drops without blocking or panicking. The guard was implemented but untested,
  so a refactor could have dropped it silently.
- The `SAFETY` note on `in_forked_child` claimed the child was effectively
  single-threaded. It is not — the parent runs the runtime thread and the test
  harness, so the child inherits allocator locks held by threads that no
  longer exist. State the residual risk rather than argue it away.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKAH7JNjPLPmG4mPTvWfth
…t hung

The first CI run of the fork suite deadlocked in
`a_child_can_attach_its_own_database_after_forking` and sat there until
GitHub's six-hour job ceiling killed it, taking "Comprehensive Development
Check" with it (it `needs:` that job). The suite has never gone green.

`fork` in a multi-threaded process leaves the child holding every lock that
was held by a thread which did not come across — the allocator's above all.
Proving a post-fork `attach` works means allocating and spawning a thread, so
that one case cannot stay async-signal-safe; sharing a binary with the rest of
the suite ran it beside several live Tokio runtimes, which is what made the
inherited-lock window wide enough to hit.

Two changes, neither of which skips the test:

- A watchdog. `in_forked_child` now polls with `WNOHANG` and, after
  `CHILD_TIMEOUT`, kills the child and fails. A blocking `waitpid` is what
  turned a deadlock into a six-hour job; the worst case is now a fast, legible
  failure. Verified by hanging a child on `pause()`: it is killed, the test
  fails with the timeout message, and no zombie is left.
- Isolation. The post-fork `attach` case gets its own binary, where the parent
  attaches nothing before forking and holds only the harness main thread and
  the test thread — the quietest parent the assertion can be made from.

The shared helper moves to `tests/fork_child/mod.rs` so all three fork
binaries get the bound, and carries the reasoning where the next reader of
these tests will find it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKAH7JNjPLPmG4mPTvWfth
@lxsaah
lxsaah force-pushed the feat/sync-fork-safety branch from f0bf83d to 4338210 Compare August 26, 2026 14:35
claude added 2 commits August 26, 2026 16:07
… guesswork

The watchdog bounded the six-hour CI hang but left the test failing far more
often than "rare": 11 of 60 runs. One green CI run was a lucky draw on an
18% failure rate, not evidence the suite was healthy.

Measuring instead of arguing, 60-120 runs per variant:

| variant | hangs |
|---|---|
| as written, tests in parallel | 11/60 |
| as written, --test-threads=1 | 17/60 |
| children leak instead of freeing | 1/60 |
| children leak where they can | 11/120 |
| children leak where they can, plus settle | 0/120 |

Harness parallelism is not the driver: serialising made it no better. The
trigger is the allocator lock — the child takes it on every free, and the
parent's own freshly-spawned runtime threads are what hold it.

So both ends are addressed. Children now mem::forget their inherited state
wherever the destructor is not what is under test; they _exit immediately, so
nothing is lost, and freeing was the unsafe act. And the parent waits for its
runtime threads to finish starting and park before forking, because fork is
only safe from a quiescent parent and a test that just called attach is the
opposite of one. Alone the first halves the rate; together they reach zero
over 120 runs per binary, 360 runs total.

Two comments claimed things the data refutes, and are corrected rather than
left to mislead:

- The SAFETY note recommended --test-threads=1. It measurably does not help.
- The post-fork-attach test's own binary was justified by isolation from other
  tests' runtimes. Parallelism does not matter, so that was the wrong reason.
  The right one is that the test attaches nothing before forking, so its parent
  has no runtime thread to be mid-allocation — a property a single attach
  anywhere else in the process would destroy.

The watchdog stays. dropping_an_inherited_handle_does_not_panic must free to
test what it tests, and the settle is a duration rather than a handshake, so
neither guarantee is absolute on a slower machine. What it converts is a silent
six-hour hang into a bounded failure, and that should not depend on the rate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKAH7JNjPLPmG4mPTvWfth
`fork::generation` and `fork::forked_since` were made public earlier in this PR
on the argument that a facade built on this crate has the same problem for the
same reason. The argument is sound but the caller is hypothetical: nothing in
the workspace uses them. The four example crates that depend on aimdb-sync never
mention fork, the only C-ABI crate does not depend on aimdb-sync, and no FFI
layer exists here. The one external user was this crate's own integration test.

Publishing them would commit the crate in semver to the stamp-and-compare model
— record a number at construction, compare it later — chosen against no real
caller, and that is the shape most likely to change: the same fact is currently
copied into three types and checked by nine call sites that each have to
remember to. Widen it when something real needs it, so the API can be designed
against that caller instead of guessed at.

The arming regression test moves into the module as a unit test, which is where
it belongs anyway. Its precondition is that nothing in the process has armed the
handler yet, and the lib test binary holds only the compile-time Send/Sync
markers and the SyncError::kind tests — none attach a database. An integration
test cannot have that precondition once it shares a binary with the fork suite,
whose first attach arms the handler; the earlier version of this test passed
against broken code for exactly that reason. Re-verified by reverting the arming
fix: it fails there and passes here.

Its waitpid is bounded with a SIGKILL fallback. That child reads one atomic and
_exits so it cannot deadlock, but "cannot deadlock" is what was said about the
fork suite before it hung a CI job for six hours, and the bound costs nothing.
Soaked 120 runs: no failures.

Not a breaking change: these items are introduced and withdrawn inside this
same unmerged PR, so no released version ever exposed them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKAH7JNjPLPmG4mPTvWfth
@lxsaah
lxsaah merged commit 45e2387 into main Aug 26, 2026
9 checks passed
@lxsaah
lxsaah deleted the feat/sync-fork-safety branch August 26, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants