fix: refuse to open a graph with an unreplayed WAL from a dead process

[pradeepmouli]

Problem

A stale WAL left by a process that died without a clean checkpoint can
crash the whole process with SIGBUS (EXC_BAD_ACCESS /
KERN_PROTECTION_FAILURE), deep inside Kuzu's WAL-replay-on-open
(WALReplayer::replayNodeTableInsertRecord -> NodeTable::insert ->
DiskArrayInternal::get) — on a plain read-only open reached by an
ordinary query, not just a risky write path, and before any Result
exists for calling code to catch.

Observed directly: three identical macOS crash reports while
indexing/searching a project whose watcher process had died uncleanly,
still holding the write lock's identity payload with a PID that was no
longer running. open_read_only's existing
.throw_on_wal_replay_failure(false) doesn't help here — that flag
governs whether Kuzu throws a catchable exception on a WAL validation
failure; it has no bearing on a hard memory-safety crash inside the
replay code itself.

Fix

Adds GraphStore::unclean_shutdown_wal_holder(db_path, lock_path), which
flags a graph only when both signals are present:

• A WAL-family sibling exists (<db>.wal/<db>.wal.*) — Kuzu did not
  complete a clean checkpoint before the graph was last closed.
• The write lock's recorded holder (lockfile::read_holder) is confirmed
  dead via a fresh sysinfo process lookup.

Either signal alone is routine and must not be flagged: a live writer's
WAL mid-transaction is the common case and needs to proceed to a normal
Database::new/replay, or every ordinary write would start refusing
reads. It's specifically the combination — a WAL that will never be
finished checkpointing because the process that would have finished it is
gone — that matches the observed crash.

Wired into both open() and open_read_only(), before Database::new:
on a positive match, both refuse up front with a clear, actionable error
instead of attempting the risky replay.

New dependency

sysinfo (infigraph-core only), for the PID-liveness check. Well
established, cross-platform, already used for exactly this purpose in
other Rust tooling.

Testing

• 5 new unit tests covering unclean_shutdown_wal_holder's four signal
  combinations (WAL+dead holder flags it; live holder, no WAL, or no lock
  file all correctly don't).
• 1 integration test proving both open() and open_read_only() refuse
  rather than attempt Database::new against a synthetic
  WAL-plus-dead-holder fixture.
• Full infigraph-core --lib suite (315 tests) green.
• cargo clippy --all-targets -- -D warnings and cargo fmt --all -- --check clean.
• Full workspace cargo check clean (confirms the new dependency doesn't
  disturb anything downstream).

Scope note

This fixes the code graph (GraphStore). The docs index (DocStore,
docs.kuzu) has the identical Database::new exposure but currently has
no cross-process identity-tracked lock — only an in-process Mutex — so
there's no reliable holder-liveness signal to check yet. Deliberately left
out of this PR rather than building a weaker heuristic; happy to follow up
once/if there's interest.

[murari316]

Automated review (Fable, adversarially verified). Recommendation: request-changes.

The PR adds a pre-open guard to GraphStore::open/open_read_only that refuses to hand Kuzu a database whose WAL was left by a process that died uncleanly (previously SIGBUS'd the whole process during WAL replay, uncatchable). The two-signal design (WAL-family sibling present AND lock payload recording a dead PID, with clean lock release truncating the payload) is thoughtful and well-documented, and the helper is well unit-tested. However, the auto-recovery path in Infigraph::init interacts badly with the new check (wipe_graph doesn't remove everything the check looks at, so recovery can hard-fail or falsely re-trip), and the PID-liveness probe contradicts the lockfile module's own documented model and is defeated by PID reuse — reintroducing the exact crash after a reboot.

---

[MAJOR] correctness — crates/infigraph-core/src/lib.rs:239

wipe_graph is asymmetric with the new unclean_shutdown_wal_holder check: it removes only the db and db_path.with_extension("wal"), but neither the <db>.wal.* family files that wal_family_paths explicitly detects nor the stale lock payload in graph.lock. The new bail text does not contain "Could not set lock on file", so Infigraph::init classifies it as potential corruption, retries (every retry bails identically since the on-disk state is unchanged), then calls wipe_graph and reopens — and that reopen can re-trip the very same check.

Failure scenario: Unclean shutdown leaves graph.wal, graph.wal.0 (family file), and graph.lock with the dead holder's payload. User runs infigraph index: init -> open bails (new check) -> OPEN_RETRY_BACKOFF_MS retries all bail -> wipe_graph removes graph + graph.wal but leaves graph.wal.0 and graph.lock -> reopen: wal_family_paths still non-empty, read_holder still returns the dead PID -> bail again -> init fails hard with "graph still unreadable after wipe" and auto-recovery is permanently bricked until the user manually deletes graph.wal.0/graph.lock. Even without a .wal.* file there is a secondary race: post-wipe reopen runs init_schema (creating a fresh WAL) before any writer re-stamps graph.lock, so a concurrent open_read_only (e.g. the MCP server) can see fresh-WAL + stale-dead-PID payload and falsely refuse a healthy, rebuilding graph. Fix: wipe the whole .wal.* family and truncate/remove the stale lock payload in wipe_graph. Also, no test exercises Infigraph::init end-to-end recovery from the unclean state (with a .wal.* family file) — the tests verify the refusal but not the recovery, which is where this bug lives.

[MAJOR] concurrency — crates/infigraph-core/src/graph/store.rs:66

pid_is_alive trusts the lock-file JSON payload for a liveness decision, which the lockfile module's own doc explicitly forbids: "a kernel advisory flock ... is the source of truth for 'held' ... The JSON identity payload ... is never trusted for liveness decisions." PID reuse silently defeats the check and reintroduces the exact SIGBUS this PR fixes.

Failure scenario: Machine loses power mid-index (the canonical unclean shutdown), leaving graph.wal + graph.lock with PID N. After reboot — precisely when unclean shutdowns are most likely — PID N is very plausibly reassigned to some unrelated live process (Linux PIDs restart from low numbers; sysinfo just reports 'a process with PID N exists'). unclean_shutdown_wal_holder sees a live holder, returns None, Database::new replays the stale WAL, and the whole process aborts with SIGBUS — the original bug, silently unfixed in its most common trigger. Fix: probe the flock instead — try_lock_exclusive on the lock file WITHOUT stamping/truncating: lock acquirable + payload non-empty <=> holder died while holding (flock auto-releases on death; clean release truncates the payload). This is race-free against PID reuse, matches the module's documented model, and removes the entire sysinfo dependency tree (windows/objc2-*/ntapi) added for one PID check. Note honestly: neither approach closes the TOCTOU where a live holder dies between the check and Database::new — the guard narrows the crash window rather than eliminating it.

[MINOR] test-coverage — crates/infigraph-core/src/graph/store.rs:377

DEAD_PID = 999_999 is not guaranteed dead: on modern Linux, systemd raises kernel.pid_max to 4194304, so 999999 can be a real running PID on a long-lived CI host, making the four unclean_shutdown tests flaky on the Linux leg of the fmt/clippy/test matrix.

Failure scenario: Linux CI runner with pid_max=4194304 happens to have a process at PID 999999 -> pid_is_alive returns true -> unclean_shutdown_wal_holder_flags_wal_plus_dead_holder and open_refuses_a_graph_with_an_unreplayed_wal_from_a_dead_process fail intermittently. Use a sentinel above every platform's ceiling, e.g. u32::MAX (Linux max pid_max is 2^22; macOS ~99998; Windows PIDs are multiples of 4, and u32::MAX is not).

[MINOR] scope — crates/infigraph-docs/src/store.rs:60

DocStore wraps a second Kuzu database (.infigraph/docs.kuzu, WAL sibling docs.kuzu.wal) with the identical unguarded WAL-replay-on-open SIGBUS exposure, untouched by this PR — worth an explicit follow-up rather than silence.

Failure scenario: A process dies mid-doc-index leaving docs.kuzu.wal; the next open of the doc store hands the stale WAL straight to Database::new and can SIGBUS the whole process exactly as the graph store did. The staged-temp-dir docs build reduces but doesn't eliminate the window, and DocStore has no per-db PID-stamped lock file, so the two-signal check doesn't transfer directly (another reason to prefer the flock-probe design, which generalizes). Follow-up issue is fine; note also that any doc-store wipe using with_extension("wal") on "docs.kuzu" would compute "docs.wal" — the exact append-vs-replace trap this PR's own wal_family_paths comment documents.

Generated with Claude Code (Fable). Findings verified by an adversarial refute pass; false positives removed.