Repository navigation
fix(replication): one client under two host NQNs is not a stranded host - #1692
Open
schmidt-scaled wants to merge 10 commits into
Open
schmidt-scaled wants to merge 10 commits into
schmidt-scaled wants to merge 10 commits into
Conversation
#1674's guard compared host NQNs between the side whose namespace goes and the side that stays. A Kubernetes node connects with the simplyblock host NQN (node UID) on one path and with the kernel default on another: on bed bf28 every client reached the source as nqn.2014-08.io.simplyblock:uuid:<uid> and the early-prepared target as nqn.2014-08.org.nvmexpress:uuid:<machine>. The guard took each as a different host, so every cutover freeze waited for a reconnect that had already happened, and R1 stopped at its first three cutovers; a discarded target clone was held the same way. Nothing records which host NQNs belong to one machine, so hosts are matched by name where possible and every unmatched host on the keeping side counts for one unmatched host on the removing side. A host is stranded only when the removing side has more unmatched hosts than that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a cutover the prepared clone named in tgt_lvol_id no longer exists: the volume runs on the other cluster under a new record with the same NQN. Bed bf29 (2026-11-20, round 2): the guard looked the target up by tgt_lvol_id, found nothing, took every host as stranded and after 120 s kept the migrated-away source of two volumes. That leaks a full volume per migration and leaves a same-named volume on the cluster the next round migrates back to. _live_target_node_ids: every other live volume exporting the source's NQN, plus the clone while it still exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
schmidt-scaled
pushed a commit
that referenced
this pull request
Nov 20, 2026
…test/repl-bf14 (bf29c)
Bed bf29 (2026-11-20): five volumes of cluster A ran their cutover endgame at once under sustained fio overwrite. Every shrink round snapshotted its volume every 15-30 s, every snapshot pinned what was overwritten after it, and the superseded snapshots were deleted more slowly than new ones were taken: A went 41% -> 82% -> 99% in 16 minutes and ran out of space. Three rules (simplyblock_core/migration_space.py) bound what a migration consumes on its source. They hold back only the migration's own snapshots, never client IO or client volume creation; cap_warn/cap_crit stay alarms. 1. At most REPL_CUTOVER_MAX_ENDGAMES_PER_CLUSTER (1) endgames per source cluster (_acquire_lvs_claim). A queued volume holds no lvstore and keeps its ordinary cadence. The claim is re-checked against a fresh task read after it is written and ranked by claim time, so volumes that catch up in the same runner pass cannot all take the slot, and a running endgame is never displaced. 2. An endgame takes its next round snapshot only once none of the volume's snapshots is still in deletion (_round_snapshot_hold). A deletion stuck for REPL_ENDGAME_DELETION_WAIT_MAX_SEC (300 s) is set aside, logged. 3. No migration snapshot - endgame round, or cadence snapshot of a migration-mode volume (snapshot_monitor) - while the source's free space is under max(5% of the cluster, 2 x bytes written since the volume's last snapshot). Warning event once per wait. A missing or stale (>120 s) capacity record holds for at most 120 s, then the snapshot proceeds. The endgame does not take the lvstore claim while its first round would wait. A round measured on a pass that then could not snapshot is not re-measured with the waiting time on the next pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… replication Bed bf29 R3b (2026-11-20): the target primary node was shut down during an A -> B migration while one volume was in its cutover endgame. Before the cutover clone exists the endgame waits for the target primary (the clone is built through it), and the wait kept the endgame in place: lvstore claim, round in flight. The endgame takes no snapshot while it waits, and the cadence snapshot skips a volume in its endgame, so nothing replicated the volume: no completed cycle for ~20 min, RPO 1280 s. The wait dates from feb45b9 (2026-08-27), which made it a deferral instead of a burned retry but left the endgame held through it. After REPL_CUTOVER_TARGET_DOWN_GRACE_SEC (60 s) of target outage the endgame is now suspended (_suspend_endgame_for_target): round state dropped, lvstore claim and cluster endgame slot released, source write cap lifted, no attempt burned and no back-off. The volume replicates on its ordinary cadence (still under the migration space rules) until the target is back; then the catch-up gate admits it to a fresh endgame. A shorter blip keeps the endgame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound A direct round's snapshot is never replicated (its pages go straight into the cutover clone), so the landing volume _prestage_next_landing created under REP_<repl_round_...> was never adopted, and _discard_spare_landing only knows the name of the round after it. Every endgame whose first direct round coincided with a still-running cadence transfer left one full-size internal volume on the target for good (bed bf29 2026-11-20: three REP_repl_round_* volumes on cluster 225e0e10 from rounds 2 and 6). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d landing volume Bed bf29 (2026-11-20): three 10 GiB REP_repl_round_* landing volumes stayed on cluster A for good. _take_shrink_snapshot replaces the pre-recorded round name (next_shrink_snap_name) when its kind does not match the round (direct vs ordinary), when a snapshot of that name exists but is not usable, and when the name turns out to be taken; each time the landing volume the replication runner had pre-staged as REP_<old name> was never adopted and nothing reclaimed it. Each of the three now discards it first (_discard_landing_named, the body of _discard_spare_landing taking the name as a parameter): deleted when it exists, is online and no live replication task lands in it. A task table that cannot be read keeps the volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
schmidt-scaled
pushed a commit
that referenced
this pull request
Nov 20, 2026
…ding-volume leaks) into test/repl-bf14 (bf31)
| # the volume replicates nothing (bed bf29 R3b): hand it back to the | ||
| # ordinary cadence after a short grace (_suspend_endgame_for_target). | ||
| if params.get("cutover_lvs") or params.get("shrink_snap_id"): | ||
| task = _suspend_endgame_for_target(task, lvol, tgt_node) |
… every member online bf32 2026-11-21 (R6 tgt-dual-p shutdown, target cluster 70edb423): at 05:16:27.95 a write to LVS_7 (primary 484ff828) and to LVS_10 (primary 9154330e) failed while two other nodes' SPDK were gone, and each primary's SPDK dropped the leadership itself (spdk_lvs_queued_failed_IO). SPDK takes it back only on the next IO; both were replication targets, and replication waits for a leader before it sends. The restarts that followed rebuilt the other members as non-leaders behind the demoted primaries, and nothing restores the leadership of a healthy LVS: 76 min leaderless with every member online, until a manual _recover_leaderless_lvs. - storage_node_ops.healthy_lvs_leadership_state / restore_healthy_lvs_leadership: leaderless-but-healthy = cluster active/degraded, owner online primary, every member ONLINE and answering get_lvstore, none leader; then the guarded _recover_leaderless_lvs on the primary (lock+cooldown, no restart/port-allow task, JM quorum, proven-free twice, reload-then-grant). Not on sync-site or two-node arbitrated clusters. - node monitor: per primary, probe every 30 s, act after 60 s confirmed leaderless, recovery on a background thread. - replication: a leaderless-target wait >= 60 s asks that recovery (every 120 s per task); replication still grants nothing itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…anding volume under creation bed bf32 (2026-11-21, cluster 70edb423): two replication workers enqueued the same chain ancestor (snapshot 4d20399d) 100 ms apart. _add_task's dedupe is a scan followed by a write, so both passed it and two tasks (5571400c, a6f9b7b7) existed for one snapshot. The worker pool dedupes by task uuid and _dispatch yielded only to a RUNNING sibling, so both ran at once: one created landing volume 5eeb989b while the other took it (in_creation, 2 s old) for a dead attempt's leftover and deleted it. The delete raced the creator's registers on the secondary/tertiary, which kept a stale LVOL_2398 that later crashed SPDK on both nodes. - DBController.ensure_snapshot_replication_task: the task write and a repl_snap_task/<cluster>/<snapshot> claim key in ONE transaction; _add_task creates every snapshot_replication task through it. - Runner: _Workers runs one task per snapshot at a time; _dispatch lets only one live task of a snapshot run (RUNNING, then landing owner, then oldest), looked up via the repl_snapshot_id index. - Half-created landing cleanup only deletes a provably abandoned volume: not being created in this runner (_LANDING_CREATING), no other live task of the snapshot, older than REPL_LANDING_ABANDON_SEC (600 s). After the delete it confirms the bdev is gone on every lvstore member (sync delete of a leftover registration; durable lvol_sync_del task where unconfirmed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| getattr(primary, "tertiary_node_id", "")): | ||
| if nid and nid not in ids: | ||
| ids.append(nid) | ||
| except KeyError: |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1674. On bed bf28 (2026-11-19) R1 stopped at its first three cutover freezes, each deferred forever with "host(s) nqn.2014-08.io.simplyblock:uuid:… reach … only through the namespace being removed".
Cause: the guard identified a client by its host NQN, but one Kubernetes node uses two of them. On the bed:
nqn.2014-08.io.simplyblock:uuid:5892b49a-…(k8s node UID)5892b49a-…nqn.2014-08.org.nvmexpress:uuid:ec2d55d6-…(kernel default)ec2d55d6-…Same machine, no common key. The guard took them for two hosts, so every freeze waited for a reconnect that had already happened. A discarded target clone was held the same way.
Fix: match hosts by name where possible. Every unmatched host on the side that stays counts for one unmatched host on the side that goes. A host is stranded only when more unmatched hosts go than stay.
Tests:
test_ns_host_guard.pyhas 11 tests, including the bed's exact case. The replication/cutover suites pass (508).Separately, the CSI connects the early-prepared target paths without the volume's
--hostnqn. That deserves its own look, because host fencing revokes by host NQN.🤖 Generated with Claude Code