[jaredLunde]

Problem

fjall (3.1.4 through 3.1.13) halts a writer while L0 has 30 runs (check_write_halt spins), but compaction is only enqueued by flushes (try_send().ok()), and a halted writer makes no flushes. If the queued compactions finish without getting L0 under 30, apply parks forever.

Bench (examples/halt_bench.rs: FjallSnapshot::apply, random keys, 200 B values, 1024-entry batches, fjall 3.1.5)

WORKERS=4 MEMTABLE_MB=64 (the config that wedged raw fjall 3.1.5 at 257M entries):

build	result
no watchdog	WEDGED entries=281824256 writes/s=0 l0=33 active_compactions=0 halt_rescues=0
watchdog	DONE entries=400000000 l0=15 halt_rescues=1 wedged=no (rescue at ~220M, writes resume at ~385k/s)

Raw fjall (no slipstream) wedged at 189.9M on 3.1.4 and 257.5M on 3.1.5. Shipped tuning (16 workers, 256 MiB) did not wedge through 400M on 16 cores, nor pinned to 4 cpus (no watchdog: DONE entries=400000000 l0=108 wedged=no). On that 4-cpu run L0 tables sat at 100+ while writes flowed, so table count alone is not the wedge signal: the first cut of the watchdog (tables + idle compactor) forced 6 needless full rewrites there. The watchdog now also requires the sequence number to be still (a halted writer commits nothing).

Change

• Watchdog thread per FjallSnapshot: after 5 one-second samples of no commits, L0 >= 30 tables, no running compaction, no queued flush, it runs major_compact on the data partition (a multi-minute stall instead of a permanent one). Stopped and joined on drop.
• FjallStats via FjallSnapshot::stats / FjallReader::stats (the reader works while apply is parked): l0_tables, tables, outstanding_flushes, active_compactions, halt_rescues.
• FjallConfig::worker_threads / memtable_bytes (0 = shipped tuning). Struct-literal construction without ..Default::default() no longer compiles (a minor-version bump).
• ARCHITECTURE.md section added.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WpMBiw38GkJFe49jfZowsH

[jaredLunde]

Closing: J11 (fjall write-halt) was closed as unreachable for our workload. Only instd uses fjall, and its fold is the node's own slice (tens of thousands of entries, far below the ~190M+ trigger). The control plane uses the RocksDB backend. If a fjall host ever needs this scale, the decision is to switch it to RocksDB, not to patch fjall. The bench harness and numbers here remain as the record.