test(replication): 24h kill-9 soak harness gating v0.7.0 release tag (task #61) - #327
Merged
Merged
Conversation
|
Warning Review limit reached
Next review available in: 8 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
TinDang97
force-pushed
the
worktree-agent-a829cff77591549b8
branch
from
July 14, 2026 03:20
1f64f25 to
d3a6bf0
Compare
…(task #61) Adds the acked-write soak that must go green before the v0.7.0 "Replication GA for multi-shard masters" tag ships. A false-green here would ship a durability lie, so the harness is deliberately machine-verifiable rather than eyeballed: - scripts/soak_replication_driver.py: raw-RESP driver with three modes. `writer` continuously SETs soak:{seq} {seq}:{ts} against the master then WAITs for 1 replica ack; only WAIT>=1 appends the seq to an fsync'd acked ledger (a WAIT timeout is recorded separately as "in-flight" -- allowed to be lost or present, never a failure; a connection failure pauses the writer and retries the SAME seq once the master is back, so acking is never claimed across a known-down window). `catchup` polls a replica's last few acked seqs until they read back correctly -- the data-driven resync gate, since master_link_status:up only proves the TCP link is back, not that the backlog/RDB replay landed. `verify` samples >=1000 random + the last 200 acked seqs (or --full for an end-of-soak sweep) and asserts exact parity on both master and replica, printing `SOAK-FAIL seq=<n> side=<m|r> cycle=<k>` and exiting 1 immediately on any mismatch. - scripts/soak-replication-24h.sh: orchestrator. Builds moon from a VM-local clone, boots master (--shards 4 --appendonly yes --appendfsync always) + replica (--shards 1 -- moon's streaming replica currently only supports single-shard, confirmed against tests/replication_multishard.rs and a live replica startup error; R2/task #20 made the MASTER side multi-shard-capable, not the replica), attaches REPLICAOF, starts the writer, then alternates kill -9 master/replica every ~12 minutes (PID-targeted only, never a broad pkill, per the SO_REUSEPORT hang trap), restarts the killed side, runs the catchup gate then the verify sample, and prints hourly `SOAK-OK hour=<h> acked=<n> cycles=<k> master_kills=<a> replica_kills=<b>` progress. --smoke runs a 30-minute/3-cycle validation instead of the full 24h. Validated with a 30-minute --smoke run on moon-dev: SOAK-PASS, duration=1800s cycles=3 acked=493 inflight=429 master_kills=2 replica_kills=1, zero acked-write loss, full-ledger sweep clean. Driver logic (writer/catchup/verify, including induced failure detection) additionally smoke-tested locally against two moon instances before the VM run. CAVEAT surfaced by the smoke run (not a harness bug -- reproduced twice, cycles 1 and 3): after a MASTER kill -9 + restart, WAIT 1 <timeout> stopped acknowledging writes for the existing (non-restarted) replica connection for the rest of that ~8min window -- data kept replicating correctly (GETs on the replica matched) and the end-of-cycle catchup+verify passed, but live WAIT/ACK signaling appears to wedge after a master restart until the replica itself is also restarted (cycle 2 fixed it). No data was lost against this harness's guarantee, but this looks like a real WAIT/ACK regression worth a follow-up ticket before the v0.7.0 tag. Does not run the full 24h soak here -- that is armed separately per the task. author: Tin Dang
TinDang97
force-pushed
the
worktree-agent-a829cff77591549b8
branch
from
July 14, 2026 03:48
d3a6bf0 to
4e5f75e
Compare
pilotspacex-byte
added a commit
that referenced
this pull request
Jul 14, 2026
… AOF recovery (task #67) (#329) After a multi-shard master (--shards >= 2) was kill -9'd and restarted with prior write history, the surviving replica kept streaming and applying writes correctly, but WAIT 1 <timeout> on the restarted master timed out for minutes — REPLCONF ACK kept arriving every second but never registered as "caught up". Reproduced 2x by the v0.7.0 replication soak (scripts/soak-replication-24h.sh, PR #327). Root cause: ReplicationState::seed_master_offset (AOF recovery, RFC §2 Rule 3) seeded ONLY the process-wide master_repl_offset from the recovered max LSN, leaving every per-shard shard_offsets[i] at the fresh-boot 0. handle_psync_inline_multi_shard's full-resync handshake advertises Σ shard_offset(i) — not total_offset() — as a reconnecting replica's new baseline, because each shard captures its own offset atomically with its RDB body (the invariant the exactly-once live-fanout `cut` gate depends on). A replica reconnecting post-restart therefore adopted a near-zero baseline while wait_for_replicas kept comparing ACKs against the correctly-seeded (large) total_offset() — a gap the replica could never close, since both axes only ever grow. The data plane was unaffected: the per-shard `cut`/end_offset filtering that guarantees exactly-once delivery never references total_offset(), only each shard's own counter. Fix: seed_master_offset now also seeds shard 0 to the same recovered value, restoring the Σ shard_offsets == total_offset() invariant every write already maintains going forward (increment_shard_offset/issue_lsn bump both axes by the same delta in lockstep). The exact per-shard split of the seed doesn't affect correctness — each shard's counter is only ever compared against itself — so concentrating it on shard 0 is sufficient and avoids a real double-count hazard (per-shard AOF replay only tracks each shard's max *global* LSN tag, not its own cumulative byte length, so seeding every shard from that tag would over-count the sum by roughly Nx). Added tests/replication_hardening.rs::master_kill_restart_wait_acks (RED on unmodified code — WAIT returned 0 with replica ACK lag stuck at ~2050 bytes behind a freshly-seeded ~2134-byte master offset; GREEN 5x consecutively after the fix) plus two state.rs unit tests pinning the seed invariant and its fetch_max never-regress semantics. Local gates green: fmt, clippy (default + runtime-tokio,jemalloc), full replication_hardening + replication_multishard suites, replication:: unit tests under both runtimes. author: Tin Dang Co-authored-by: Tin Dang <tindang.ht97@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 24h replication soak harness that gates the v0.7.0 tag (PRODUCTION-CONTRACT
REPL-SOAK-01):scripts/soak-replication-24h.sh— orchestrator: VM-local build, master--shards 4 --appendfsync always+ replica, alternating PID-targeted kill -9 every 12 min, per-cycle + final full-ledger verification, hourlySOAK-OKprogress lines.scripts/soak_replication_driver.py— raw-RESP driver (writer/catchup/verify): WAIT-confirmed writes to an fsync'd ledger; WAIT timeout ≠ loss (recorded in-flight); any acked seq missing/wrong on either side ⇒SOAK-FAIL+ exit 1.Design deviation (verified, not assumed): replica runs
--shards 1— the streaming replica hard-errors on multi-shard (src/replication/replica.rs); R2 made only the master side multi-shard.Verification
SOAK-PASS duration=1800s cycles=3 acked=493 inflight=429 master_kills=2 replica_kills=1— zero acked-write loss, full sweep clean.Finding (follow-up filed, blocks the 24h run)
Reproduced 2×: after a master kill-9+restart,
WAIT 1 <t>stops acknowledging on the surviving replica connection until the replica also restarts (~8 min window). Replication itself stays correct (verify passes); only ACK signaling wedges. Tracked as task #67 — to be fixed before the 24h soak so acked coverage isn't halved.