feat(wallet): accelerate storage synchronization

[BraydenLangley]

Keep this pull request in draft until local validation is complete. After
every push, wait for all applicable checks on the exact head to finish and
fix every failure before requesting review or calling the work complete.

Program and scope

• Tracker or issue: p2ppsr/metanet-client-desktop#42
• Program gate(s) advanced: practical remote-to-local wallet replication, exact progress/ETA, backward-compatible provider rollout
• Why this change is needed: a real isolated Metanet Client benchmark copied 3,358 records but spent 212.3s in remote getSyncChunk reads (15.16s/page average) while IndexedDB writes took about 0.1s/page. The source page builder was performing up to 25 sequential database reads to fill one 250-record entity page, and common user-scoped transaction proof lookups lacked composite indexes.
• Explicitly out of scope: changing WalletStorageManager's global sync lock or allowing normal writes to interleave with offset-based replication. That requires a cursor/keyset or consistent-snapshot protocol so mutations cannot shift offsets. No service deployment or npm publication is performed here.
• Exact head SHA reviewed: 3d420ab

Impact

• No public package source or manifest changed
• Public package source or manifest changed; affected packages are listed below
• Infrastructure source, dependency, image, or deployment configuration changed
• Public API, exports, types, runtime targets, or browser/mobile behavior changed
• Security-sensitive boundary changed
• Documentation or examples changed

Affected packages/services and intended patch versions (publication occurs only
through the release workflow after approval):

• @bsv/wallet-toolbox 2.11.0 (minor: additive includeTotals / SyncChunkTotals API and runtime optimization)
• @bsv/wallet-toolbox-client 2.11.0 (minor: carries the additive API)
• @bsv/wallet-toolbox-mobile 2.11.0 (minor: carries the additive API)
• Wallet storage service deployments must consume the released toolbox and run the new Knex migration before clients receive the faster source reads and exact totals.

Implementation

• Fill sync pages with bounded adaptive source reads: 10 records initially, then size-aware 8x growth capped at 250. A 250-record small-transaction fixture now uses 3 source queries (10, 80, 160) instead of 25.
• Add (userId, provenTxId) and (userId, txid) transaction indexes. SQLite EXPLAIN tests prove both plans use the new indexes; a synthetic 5,000-row offset query improved from about 0.15s to 0.01s locally.
• Add opt-in first-page source totals for an exact progress bar and ETA. Counts execute concurrently and are best-effort, so a count failure never fails replication. Legacy clients do not request totals and legacy providers omit them without a wire break.
• Preserve totals across storage remoting validation and add a full-sync assertion over more than 1,000 records.
• Compared with bsv-blockchain/bsv-desktop: it uses the same WalletStorageManager replication path with local StorageKnex behind Electron IPC. Its SQLite/WAL destination architecture is useful for persistence but does not remove this shared source-page bottleneck; its performance suite benchmarks ordinary storage operations rather than remote-to-local replication.

Verification

• Local commands and results: full toolbox test and coverage suites passed (217 suites, 2,030 passed, 1 skipped); focused adaptive paging/totals suite passed (5/5); client tests 17/17; mobile tests 19/19; workspace build completed.
• Hosted CI run: https://github.com/bsv-blockchain/ts-stack/actions/runs/32073739379 passed on the exact head, including the merge gate.
• Conformance evidence: pnpm health:check passed all policy tests; package docs and version governance checks passed.
• Coverage delta: full toolbox, client, and mobile coverage commands exited successfully; new source paging behavior has direct unit coverage and migration query-plan coverage.
• Lint/typecheck delta: root lint, format check, and all 37 workspace project typechecks passed.
• Browser/mobile/packed-consumer evidence: toolbox/client/mobile pack checks passed. Browser contract: Vite 1,607,393 raw (hosted Linux; 1,607,123 on macOS) / 378,833 gzip / 296,769 Brotli; esbuild 1,252,871 raw / 345,202 gzip (hosted Linux; 1,252,651 / 344,199 on macOS) / 276,993 Brotli. Mobile contract: Metro 1,660,502 raw / 419,132 gzip / 326,047 Brotli; Hermes 3,366,760 raw / 1,346,422 gzip / 1,059,779 Brotli.
• Performance or bundle-size delta: source queries for the representative 250-record page fall from 25 to 3. Vite raw/gzip ceilings advance by 400/100 bytes to 1,607,400/378,900; esbuild raw/gzip advances by 400/300 bytes to 1,252,900/345,300. Other browser compressed and mobile ceilings are unchanged.
• I self-reviewed the complete diff for correctness, security,
  compatibility, public API, artifacts, dependencies, docs, and operations
• All applicable checks are terminal and successful on the exact head; any
  scope-based skip is expected and validated by the merge gate

Security and dependencies

• No dependency or lockfile change
• Changelog, runtime relevance, peer compatibility, transitive graph, and
  audit results were reviewed
• CodeQL/negative tests cover any changed trust boundary
• The exact-head CodeQL analysis has no new alert
• The exact-head repository quality gate reports zero new Sonar findings
  (including accepted or false-positive issue states) and zero unreviewed hotspots;
  Sonar's aggregate Quality Gate passed verdict alone is not merge evidence
• No new override, advisory dismissal, quality suppression, or skipped test
• Any temporary exception is registered with owner, evidence, review date,
  and removal condition
• Workflow permissions and lifecycle-script behavior remain least privilege

Dependency evidence

• Release notes and necessity: no third-party dependency changes; additive package API and source performance changes are documented in the changelog, package docs, API migration ledger, and governed release notes.
• Runtime, build, and peer compatibility: older clients omit includeTotals; older providers return no totals; both paths remain supported. Metanet Client production build/tests passed while linked to this candidate.
• Deduplicated lockfile: unchanged.
• Audit and CodeQL: pnpm audit:security exited 0 with existing governed findings; exact-head hosted CodeQL remains pending.
• Package and consumer tests: full toolbox/client/mobile suites, packed consumers, browser, mobile, and a production Metanet Client consumer build passed locally.
• Bundle and performance impact: recorded above.
• Affected public package versions: toolbox, client, and mobile 2.11.0.

Release and operations

• No npm publication was performed from a workstation or from this PR
• Required npm patch bumps are included or intentionally deferred by the
  controlling program
• Image/SBOM/provenance/deployment/rollback impact is documented
• Documentation, changelog, migration, and operational guidance are current

Rollout must be staged: merge/release packages only after review, rebuild storage service images against the release, let the Knex migration add the indexes, then validate the Metanet isolated benchmark before enabling/advertising production local backup. Rolling clients back is safe because totals are optional; rolling the service back forfeits the speed/totals improvement but preserves the legacy protocol.

Completion evidence

• The linked tracker is updated only for work fully proved by merged code,
  passing checks, resolved alerts, measurements, or an approved exception
• Review conversations are resolved
• Documentation, changelog, migration notes, release notes, and operator
  guidance are current or concretely not applicable
• No pending, failed, stale, cancelled, or unexpectedly skipped check is
  being handed to another contributor as “complete”
• One qualified maintainer approval is sufficient; no last-pusher
  restriction is assumed

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[codecov]

Codecov Report

❌ Patch coverage is 95.65217% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...s/wallet/wallet-toolbox/src/storage/StorageKnex.ts	85.71%	0 Missing and 1 partial ⚠️
...wallet/wallet-toolbox/src/storage/StorageReader.ts	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!