fix(core): delete merged MemWAL generation dirs after WAL merge

[beinan]

Problem

RolloutStore::merge_own_shard folds each flushed MemWAL generation into the base table (Dataset::append) and drains it from the shard manifest (commit_update) — but never deletes the underlying _mem_wal/{shard}/{gen}/ blob directory. Every merged generation leaves a zombie directory (data.lance + bloom_filter.bin + manifest) on object storage forever.

This is a pure storage leak:

• Data is correct — the drained manifest no longer references the directory, so the read path never sees it and rows are not double-counted. This is why it went unnoticed.
• _mem_wal/ grows without bound — the number of orphaned directories equals the number of successfully merged generations, accumulating across the store's entire lifetime.

Observed in production (conntest_lianghongxu): blob had more _gen_ directories than any shard manifest referenced. The surplus was exactly the store's historically merged generations — not a fence/concurrency artifact.

Root cause is the merge drain, not fencing

An earlier hypothesis blamed a fence between flush's data-write and manifest-commit. That was ruled out experimentally: a concurrent-fence repro produced 40 fences and 0 orphans (fences land before write_data_file, leaving nothing behind). A fully serial, single-shard, zero-fence repro reproduced the leak deterministically (30 merges → 30 leaked dirs), pinning it on the merge path.

Fix

Delete each merged generation's directory after the manifest drain, best-effort (a delete failure logs a warning but does not fail the already-successful merge). Ordering is deliberate and crash-safe: the drain removes the ids first (so no reader can resolve them), then the data is deleted.

Test plan

• New serial regression test wal_merge_generation_cleanup.rs (no concurrency, no fence) asserting on-disk _gen_ dirs == manifest pending generations after count-triggered merges
• Verified it fails without the fix (30 on-disk vs 0 pending = 30 leaked) and passes with it (0 leaked, all 30 rows readable exactly once)
• cargo test -p lance-context-core — 130 passed + 2 regression tests
• cargo fmt --all + cargo clippy -p lance-context-core clean

🤖 Generated with Claude Code