Bound the key-listing resync's memory (external sort + merge-join)

ARCHITECTURE.md

@@ -234,7 +234,7 @@ The one exception: an update carrying the **unknown** version (an unparseable AC
 
 **Cursor-expiry repair.** <a id="cursor-expiry-repair"></a> The cursor expires at resume (`CursorExpired` from the `*_from` watch) or mid-watch (the live floor guard ends an All-scope watch with `CursorExpired` when retention overruns it). Both take one path. Correct means **every write minus every real delete** — not "whatever NATS still lists"; the two agree only when retention never evicts current values. The repair (`ExpiryRepair`, `src/repair.rs`) is chosen accordingly:
 
-- **Key-listing diff** (`Relist`; `Auto` on a bucket that keeps current values). The watch task lists the bucket's live keys, hands them to the main loop, and waits for an ack; the main loop diffs the fold's in-scope keys against the listing and runs a synthetic `KvUpdate::Delete` (unknown version — never advances the cursor) through `parse`/`apply`/store for each key that vanished; only then does the full-scope re-list (`watch_all` / `watch_prefix` / `watch_prefixes`) start. That ack ordering is the invariant: a synthetic delete always precedes the re-list put for the same key, so delete-then-recreate during the gap converges. **Refused** — the watch fails — when the watcher reports retention that evicts current values (`KvWatcher::retention`, read live from the stream config), or the fold remembers seeing it do so (below): there it deletes valid keys. `tests/eviction.rs` reproduced exactly that against a live server before the fix.
+- **Key-listing diff** (`Relist`; `Auto` on a bucket that keeps current values). The watch task hands the main loop the reader and waits for an ack; the main loop lists the bucket's live keys, sorting them externally (`KeySorter`: 64 MiB runs spilled to the restore source's scratch dir under `Auto`, else `TMPDIR`), merge-joins that against the fold's own key-ordered scan — so neither the listing nor the fold is held in memory, only the stale keys — and runs a synthetic `KvUpdate::Delete` (unknown version — never advances the cursor) through `parse`/`apply`/store for each key that vanished; only then does the full-scope re-list (`watch_all` / `watch_prefix` / `watch_prefixes`) start. That ack ordering is the invariant: a synthetic delete always precedes the re-list put for the same key, so delete-then-recreate during the gap converges. (With the listing taken by the main loop after its drain, `tests/model_repair.rs` shows axiom 6 alone already keeps it correct; the barrier is defense in depth.) **Refused** — the watch fails — when the watcher reports retention that evicts current values (`KvWatcher::retention`, read live from the stream config), or the fold remembers seeing it do so (below): there it deletes valid keys. `tests/eviction.rs` reproduced exactly that against a live server before the fix.
 - **Artifact restore** (`Restore`; `Auto` on a bucket that evicts current values or can't say). The watch task peeks the newest artifact's manifest and checks it before downloading: it must cover the watch scope (recorded in the manifest by the exporting `watch_applied`), and pass `protocol::restore_allowed` — strictly ahead of the local fold, cursor still inside retention. It then downloads and imports it into scratch as a temporary fold of the consumer's backend, re-checks what actually arrived, and hands it to the main loop. The main loop re-checks "ahead" against its own applied cursor (a mid-watch expiry delivered past the resume cursor) and diffs the in-scope fold against the artifact in one key-ordered pass: keys the artifact's entry should replace are spilled to a scratch file next to the imported artifact (`RestoredFold::scratch_in`), and local keys the artifact lacks are collected as delete candidates. Only if there are candidates does it stream the bucket's live keys past them (`KvReader::for_each_key`) and then re-read retention: the listing vouches for writes after the artifact's cursor only while the log still holds all of them, so if retention passed that cursor meanwhile the restore is refused before anything is folded (`tests/model_repair.rs`, `NoListingRecheck`). It then folds the difference through `parse`/`apply`/store in `max`-sized chunks, read back from the spill, then advances the cursor to the artifact's in one final commit and replies. Memory is the delete candidates — the keys deleted between the two cursors — never the fold, the artifact, or the bucket's listing, so a billion-key fold (or a fresh node seeding from an artifact) restores in bounded memory. The diff takes an artifact entry only when it is newer than the local one, and deletes a key the artifact lacks only when the bucket does not list it live. An artifact is complete *together with the log after its cursor* — a NATS cursor C means every retained message ≤ C is applied, not "the truth at C": a key whose latest write is after C may be missing or older in an artifact exported mid catch-up — and in both skipped cases the key's latest write is after C, so the resume delivers it. The consumer therefore never sees a key move backward or a live key deleted, not even transiently. The watch resumes from that cursor (which can itself expire later and go round again, each time only for a strictly newer artifact). A failed check fails the watch and logs the reason at `error`: a stale artifact has no safe recovery.
 - **None** (`None`, or no store): the re-list alone, with a warning that deleted keys may persist.
 

README.md

@@ -378,7 +378,7 @@ Turning eviction off later doesn't make the key list trustworthy again: keys tha
 
 On a bucket that evicts only by `max_age`, a node restarted within half of `max_age` resumes even when every message after its cursor was overwritten while it was down: none of those can have aged out, so there is nothing to restore.
 
-A restore holds only the keys it will delete in memory; the keys it rewrites are spilled to the scratch directory, so it works at any fold size.
+A restore holds only the keys it will delete in memory; the keys it rewrites are spilled to the scratch directory, so it works at any fold size. The key-listing repair likewise sorts the bucket's listing on disk and walks it alongside the fold, holding only the keys it deletes. Under `Auto` it uses the restore source's scratch directory; under `Relist` it uses `TMPDIR`, so point that at disk, not tmpfs, for large buckets.
 
 `Some(reader)` from older code still compiles and means `Relist`. `None` (or no `store`) falls back to the re-list alone and warns that deleted keys may persist.
 

src/applied.rs

@@ -717,11 +717,6 @@ where
         self.open
     }
 
-    /// The store, between flushes.
-    pub(crate) fn store(&self) -> Option<&S> {
-        self.store.as_ref()
-    }
-
     /// Move the store out (to a blocking task); [`put_store`](Self::put_store)
     /// hands it back. A flush in between would skip the store fold.
     pub(crate) fn take_store(&mut self) -> Option<S> {