dash-spv: gap-limit scripts derived after a batch commits never re-open the committed range — wallet funds stay invisible (Found-034)

[Claudius-Maginificent]

Registered in the platform-wallet e2e test plan as Found-034 (pinned by
the found_coinjoin_gap_limit_sync::sim_tests simulation suite).

Summary

FiltersManager's new-script rescan (added in #820) re-applies matched
blocks only within active batches. Once a batch commits, it is removed
from active_batches and its per-wallet processed records are pruned, and
the per-wallet synced_height advances past its range. Scripts derived
later by wallet gap-limit maintenance are therefore never re-tested against
that range — the re-scan gate is effectively keyed by (wallet,
block/commit progress) instead of (wallet, address/script). Outputs that
pay gap-window addresses whose scripts did not exist yet when their block's
batch committed are silently and permanently missed: a fresh re-sync
from genesis walks into the same wall deterministically, so the funds are
invisible until the user manually pre-derives addresses.

This is the residual of the CoinJoin gap-limit sync defect found by the
rs-platform-wallet e2e suite (dashpay/platform#3549,
found_coinjoin_gap_limit_sync.rs): the dense same-batch shape (empirical
stall at highest_used = 59) was fixed by #820; the cross-commit-boundary
shape is still live. The e2e ground-truth data for the affected testnet
wallet shows real index↔height inversions (first at index ~1767), so this
is not hypothetical for deep CoinJoin usage; a plain BIP-44 wallet paid
out of index order across a batch boundary (5000 blocks) hits it too.

Affected code (dev @ 647fa98)

Site	Role
dash-spv/src/sync/filters/manager.rs:623-632	rescan_batch bails for any batch_start not in active_batches — committed ranges unreachable
dash-spv/src/sync/filters/manager.rs:479-512	commit-time rescan covers only the committing batch + later scanned active batches
dash-spv/src/sync/filters/manager.rs: 519, 535	commit removes the batch and prunes tracker records at/below the committed height
dash-spv/src/sync/filters/manager.rs:648-666	rescan skips heights at/below the wallet's own synced_height
dash-spv/src/sync/filters/block_match_tracker.rs:72-117	gate keyed by (wallet, block); track_for_new_scripts (#820) bypasses it only for blocks the rescan can still reach
dash-spv/src/sync/filters/sync_manager.rs:164	wallet recorded done for a block on BlockProcessed
key-wallet/src/transaction_checking/account_checker.rs:644	check_transaction_for_match recognises only already-generated addresses — correct, but it makes the SPV-side re-test the only recovery path

Root cause

Discovery suppression is keyed by scan progress (which blocks/batches this
wallet has already had applied/committed), not by which scripts the block
was actually tested against. #820 fixed the in-flight half by letting
rescan_batch re-queue already-processed blocks of active batches. The
committed half still loses information: after commit there is no record of
which scripts a block was matched against, no way to requeue it, and the
wallet's advanced synced_height filters the heights out of any later
match anyway. Newly derived gap-limit scripts consequently have zero
effective backward reach across a commit boundary.

Reproduction (deterministic, crate-level, no network)

Crate-level repro: dash-spv/src/sync/filters/coinjoin_gap_discovery_tests.rs (committed on
branch repro/pr3549-rdc) drives the real pipeline — FiltersManager +
BlockMatchTracker + a real WalletManager<ManagedWalletInfo> with a
CoinJoin account — through the production event loop (try_process_batch →
BlocksNeeded → process_block_for_wallets → BlockProcessed →
commit-time rescan) with synthetic blocks and real BIP-158 filters:

1. coinjoin_gap_limit_dense_same_batch_recovers — the original e2e
   stall-at-59 shape (block funds CoinJoin indices 0..=51, next block
   52..=139, one batch). PASSES (regression guard for fix(dash-spv): reprocess blocks on rescan against newly derived scripts #820).
2. coinjoin_gap_limit_inversion_within_batch_recovers — indices 40..=51
   funded at height 10, indices 0..=29 at height 20, same batch.
   PASSES (fix(dash-spv): reprocess blocks on rescan against newly derived scripts #820's fixpoint recovers the earlier block).
3. coinjoin_gap_limit_stall_across_committed_batch — identical shape,
   but height 10 sits in batch 0..=99 and the in-window block at height 110
   in batch 100..=199. FAILS (RED):

assertion `left == right` failed: CoinJoin External indices 40..=51 were funded at
height 10 in a batch that committed before their scripts were derived, and the
new-script rescan never looks below the committed boundary [...]
highest_used stalls at Some(29), used_count=30
  left: Some(29)
 right: Some(51)

Run: cargo test -p dash-spv --lib coinjoin_gap

Expected vs actual

• Expected: any output paying an address within the gap-limit recovery
  contract (index ≤ highest_used + 1 + gap at fixpoint) is discovered
  regardless of which batch/commit its block landed in — the guarantee
  BIP-44 recovery semantics give and that dash-core/Electrum-style
  per-address history scans provide. In the repro: highest_used = Some(51),
  all 42 funded indices credited.
• Actual: highest_used = Some(29); indices 40..=51 (watched by then —
  highest_generated = Some(59)) are never re-tested against the committed
  block; their UTXOs and balance are missing. Re-syncing from scratch
  reproduces the loss deterministically.

Severity (OWASP-normalized floats — label derived by pipeline)

• risk: 0.4 — needs an index↔height inversion (or restart-shaped
  equivalent) crossing a 5000-block batch boundary; ground-truth e2e data
  shows such inversions on a real CoinJoin wallet; deterministic once the
  shape exists, and completely silent (no error, no log).
• impact: 0.7 — silent integrity failure of wallet state: confirmed
  funds invisible to balance/UTXO views, unrecoverable by resync; user may
  act on a wrong balance.
• scope: 0.4 — dash-spv historical filter sync; wallets with
  out-of-index-order funding across batch boundaries (deep CoinJoin usage
  is the known concrete population).

Likelihood caveat: the repro compresses the batch boundary to ~100
blocks so the inversion is deterministic to construct; production batches
are 5000 blocks, so a genuine index↔height inversion spanning that wider
boundary is rarer in practice than the repro's density implies. The risk: 0.4 estimate sits at the upper edge of what that lower real-world
frequency would justify — it is defensible mainly because the e2e
ground-truth data already shows a real inversion on a live testnet wallet
(first at index ~1767), not because the condition is common. Severity
MEDIUM stands regardless: impact and scope are unaffected by this
caveat.

overall = (0.4 + 0.7 + 0.4) / 3 ≈ 0.5 → MEDIUM

Fix direction

Any of (roughly in order of fidelity):

1. Key match suppression by script coverage: have BlockMatchTracker (or
   filter storage) record the script-set revision a block/batch was
   matched against, and re-open committed ranges for a wallet when its
   script set grows (re-run check_compact_filters_for_script_pubkeys
   over stored filters below committed_height with only the new
   scripts — cheap, filters are already on disk).
2. On new-script derivation, lower the owning wallet's effective rescan
   floor (the tick catch-up path already knows how to restart a wallet
   whose synced_height fell behind — reuse it with synced_height = min(synced_height, birth-of-new-scripts-uncertainty) or simply the
   batch start of the oldest still-stored filter).
3. Minimum: iterate commit-time rescans over stored (not just active)
   filter ranges for the wallets named in new_scripts.

The GREEN tests (1)–(2) above pin the already-fixed behaviour so a fix for
the committed-range hole can't regress the #820 fixpoint; test (3) flips to
GREEN when the residual is fixed.

Cross-references

• Origin: test(platform-wallet): e2e framework + full test suite — triage pins, Found-*/PA-* guards, fail-closed persist, Stage-2 merge platform#3549 e2e found_coinjoin_gap_limit_sync.rs
  (diagnose-only reproduction + block-atomic simulation of the old
  stall-at-59; its SPV_BACKWARD_RESCAN_BLOCKS = 0 constant models the
  pre-fix(dash-spv): reprocess blocks on rescan against newly derived scripts #820 behaviour and should be revisited after this issue).
• Partial fix: fix(dash-spv): reprocess blocks on rescan against newly derived scripts #820 (1b534581) — reprocess blocks on rescan against newly
  derived scripts (active batches only).
• Related account-resolution gap for CoinJoin funding: Found-012: wait_for_proof and recover_asset_lock_blocking hard-code BIP-44 — asset-lock proof flow misses CoinJoin / legacy BIP-32 funding platform#3642.

_{🤖 Co-authored by Claudius the Magnificent AI Agent}

[bfoss765]

Real-world confirmation of this defect from Android integration testing (dashj→Kotlin-SDK migration work), plus one datum on gap-limit sizing:

Setup: a real, long-lived testnet wallet — 12.08713251 DASH, 1,047 transactions, 104 UTXOs, 4,517 used CoinJoin keys (heavy mixer), birth height 0. Restored from seed side-by-side: dashj 22.0.3 as oracle, and a fresh SDK wallet (dash-spv via rs-platform-wallet/unified-sdk JNI) with DEFAULT_COINJOIN_GAP_LIMIT already widened to 100 (dashj lookahead parity, per the #868 proposal).

Result:

• dashj restored the wallet exactly (balance, tx count, and outpoint set all certified).
• The fresh SDK wallet completed a full header+filter scan and landed synced with 0 transactions / 0 duffs, per-wallet synced_height at tip, no fault raised — the funds silently invisible, deterministically, exactly the Found-034 shape.
• A host-side mitigation (detect the impossible "complete scan + birth-0 + zero results" state → delete and recreate the SDK wallet) recovered it: the recreated wallet re-scanned and reached byte-exact parity with dashj (1,208,713,251 duffs / 1,047 txs), stable across repeated probes.

Two takeaways for this issue:

1. Gap sizing does not close this hole. The strand occurred at gap 100. The recreate-recovery works because the second pass starts with the full script set (including the 4,517 CoinJoin keys' window) registered before the first batch commits — which is exactly this issue's root cause viewed from the other side: it's the re-scan gate keying, not the window size.
2. Host apps can band-aid this with recreate-on-impossible-state, but that costs a full rescan, and other SDK consumers won't have it — this needs the in-engine fix.

A fix PR for this issue is in progress (re-keying the re-scan reach so scripts derived after a batch commits can still be tested against the committed range), targeting the red repro test here (coinjoin_gap_limit_stall_across_committed_batch).