DurableStore

A Rust key-value store with synchronized writes and crash recovery.

DurableStore stores byte keys and values in a checksummed log with bounded recovery, an exclusive writer lock and atomic compaction. It includes a Rust library, a JSON command-line interface, a crash-recovery demonstration and a repeatable benchmark.

Try it

Requirements: Rust 1.89 or newer, Linux or macOS, and a local filesystem. No third-party Rust dependencies, server, account, or network access at runtime.

cargo build --release --locked
mkdir demo-store
./target/release/durablestore demo-store init
./target/release/durablestore demo-store put 68656c6c6f 776f726c64
./target/release/durablestore demo-store get 68656c6c6f
# {"found":true,"value_hex":"776f726c64"} means hello -> world
./target/release/durablestore demo-store inspect
./target/release/durablestore demo-store compact

Keys and values use hexadecimal in the CLI, allowing arbitrary bytes. "" represents empty bytes. list returns keys in byte-sorted order. delete <key-hex> appends a tombstone, including for absent keys. get exits with code 3 for a missing key; operational/input errors exit 2 with a JSON error on stderr. Successful commands emit JSON on stdout and exit 0. Only init creates a database; the directory must already exist. The library supports values larger than common shell argument limits.

use durablestore::Store;

fn save_result(directory: &std::path::Path) -> durablestore::Result<()> {
    let mut store = Store::open(directory)?;
    store.put(b"experiment", b"complete")?; // sync succeeds before returning
    assert_eq!(store.get(b"experiment"), Some(b"complete".as_slice()));
    store.delete(b"experiment")?;
    store.compact()?;
    Ok(())
}

Atomic batches

Use a batch when related keys must change together. For example, mark a queued job complete while removing its pending entry and adding its result. A batch uses one checksummed record and one synchronization; recovery exposes all its changes or none.

use durablestore::{Operation, Store};

fn complete_job(store: &mut Store) -> durablestore::Result<()> {
    store.write_batch(&[
        Operation::Put(b"job:42", b"complete"),
        Operation::Delete(b"pending:42"),
        Operation::Put(b"result:42", b"artifact.txt"),
    ])
}

The CLI reads a file or standard input. Fields are separated by literal tabs; keys and values are hexadecimal. Empty fields represent empty bytes. Input is validated before opening the store.

printf 'put\t6a6f623a3432\t636f6d706c657465\ndelete\t70656e64696e673a3432\n' > changes.tsv
./target/release/durablestore demo-store batch changes.tsv
# {"ok":true,"durable":true,"operations":2}

Run python3 scripts/batch_demo.py target/release/durablestore for a complete outbox-state example using disposable local data. It sends no messages.

batch - reads standard input. Each line is put<TAB>key<TAB>value or delete<TAB>key. LF and CRLF are accepted; blank lines, extra fields and empty files are rejected. At most 1,024 operations and 32 MiB of encoded batch payload are allowed. Per-key and per-value limits still apply. Operations run in listed order, including repeated keys. An empty library batch does nothing; it still rejects a poisoned handle.

Old single-operation logs remain readable. Older DurableStore binaries reject batch records; keep the new binary with stores that use batches. Compaction writes the resulting live state as ordinary put records. This is atomic grouped writing, without rollback commands, conditional updates or concurrent transactions. A complete batch may survive a crash even if the caller did not receive acknowledgement.

Persistence contract

Event	Behavior
put / delete / write_batch returns Ok	Complete record was written and File::sync_all succeeded.
Process dies before acknowledgement	That operation may be absent or present. Earlier acknowledged state survives under the supported filesystem contract.
Incomplete terminal record	Opening discards only its incomplete bytes, synchronizes the repair, and reports the count in Stats::tail_bytes.
Full record has bad checksum, bad lengths, or wrong sequence	Opening fails without modifying data.wal.
Append or compaction fails	Further writes are rejected on that handle. Drop and reopen to determine the persisted state. Reads on the old handle remain its last acknowledged in-memory state.
Another handle owns the store	Open and inspection fail immediately instead of waiting indefinitely.
Compaction is interrupted	Recovery uses the original log or the atomically replaced compacted log. Both represent the same live state.

inspect acquires the lock and scans without creating files or repairing a tail. Ordinary opens can repair a tail. All commands require exclusive access; this intentionally avoids a second concurrent-reader consistency contract. Never delete or replace LOCK, or rename/delete the store directory, while any handle is open.

Spawning a child that immediately executes a new program is supported. Do not fork and then use or drop an inherited Store in the child: its lock refers to the same operating-system object as the parent, and unlocking either copy releases that shared lock. Open a fresh store only after executing the child program.

A checksum detects accidental damage, not hostile changes. An incomplete suffix caused by a failed append is indistinguishable from some kinds of external truncation; recovery cannot reconstruct bytes that a separate program destroyed. The process-crash tests do not demonstrate power-loss safety on every controller or filesystem. Durability depends on the filesystem and hardware honoring synchronization. macOS sync_all is not a claim of hardware-cache flush via F_FULLFSYNC.

Inside the engine

flowchart LR
    A[Byte key and value] --> B[Bounds and sequence]
    B --> C[Append header and payload]
    C --> D[Sync log]
    D --> E[Update ordered memory index]
    E --> F[Acknowledge]
    G[Open log] --> H[Validate checksums and replay]
    H --> E
    E --> I[Write compacted temporary log]
    I --> J[Sync, rename, sync directory]

Loading

• Ordered index: a BTreeMap owns live keys and values. Reads are in memory; this is not an on-disk B-tree.
• Write-ahead log: versioned, little-endian, length-bounded records with separate header and payload CRC-32 checksums. Bounds are checked before allocating payload memory.
• Compaction: sorted live entries replace the log atomically. The lock file keeps a stable inode across replacement. Sequence numbers are local to a log and restart during compaction.
• Initialization: a synchronized temporary file is renamed into place. Existing partial or invalid file headers are rejected rather than treated as an empty store.

See the binary format and design decisions. The entire implementation uses safe Rust and the standard library.

Crash-recovery checks

cargo test --locked
cargo test --locked --all-features
cargo build --locked --features fault-injection
python3 scripts/crash_demo.py target/debug/durablestore

The optional feature makes specifically named boundaries terminate with exit code 86, bypassing Rust destructors. The demonstration checks acknowledged values and deletions after each termination, then confirms the reopened store accepts another write. The default binary ignores those environment variables. Do not enable fault-injection for real data.

Tests also compare seeded operation sequences against an independent BTreeMap model, flip every individual byte of a fixture, cut its final record at every offset, interrupt initialization and tail recovery, reject oversized inputs, exercise lock contention across processes, and inject I/O errors that must poison a handle. These tests cover the listed failure cases.

Benchmark

cargo run --release --locked --example benchmark -- ./bench-store 1000 > benchmark.json

The destination must not already exist. The benchmark writes each 256-byte value twice with a sync per operation, reads warm in-memory keys, deletes a quarter of keys, compacts, and verifies every remaining key after reopening. JSON includes p50/p95/p99 latency, operation counts, compiler/OS/CPU details, log sizes, and compaction/reopen time. The directory is retained for inspection. Read timings measure the memory index, not disk lookup. Short microbenchmarks are sensitive to timer overhead, cache state, filesystem, and competing work; no cross-engine performance claim is made.

Measured local run and verification record.

Supported scope

Maximum key: 1 MiB. Maximum value: 16 MiB. Empty keys and values work. Startup is linear in log bytes and holds all live data in RAM; compaction needs temporary disk space for the full live set. There is no interactive transaction, SQL, replication, encryption, TTL, concurrent reader process, background compaction, or network-filesystem support. Data directories must be trusted: this is not a hardened service that accepts arbitrary paths from hostile clients. Compaction removes historical versions, so preserve an offline copy if you need a forensic history.

MIT licensed; copyright nazeeh111.