Skip to content

fix(replication): one client under two host NQNs is not a stranded host - #1692

Open
schmidt-scaled wants to merge 10 commits into
mainfrom
fix/ns-guard-host-nqn-forms
Open

schmidt-scaled wants to merge 10 commits into
mainfrom
fix/ns-guard-host-nqn-forms

Conversation

@schmidt-scaled

Copy link
Copy Markdown
Contributor

Follow-up to #1674. On bed bf28 (2026-11-19) R1 stopped at its first three cutover freezes, each deferred forever with "host(s) nqn.2014-08.io.simplyblock:uuid:… reach … only through the namespace being removed".

Cause: the guard identified a client by its host NQN, but one Kubernetes node uses two of them. On the bed:

side hostnqn hostid
source (3 nodes) nqn.2014-08.io.simplyblock:uuid:5892b49a-… (k8s node UID) 5892b49a-…
early-prepared target (3 nodes) nqn.2014-08.org.nvmexpress:uuid:ec2d55d6-… (kernel default) ec2d55d6-…

Same machine, no common key. The guard took them for two hosts, so every freeze waited for a reconnect that had already happened. A discarded target clone was held the same way.

Fix: match hosts by name where possible. Every unmatched host on the side that stays counts for one unmatched host on the side that goes. A host is stranded only when more unmatched hosts go than stay.

Tests: test_ns_host_guard.py has 11 tests, including the bed's exact case. The replication/cutover suites pass (508).

Separately, the CSI connects the early-prepared target paths without the volume's --hostnqn. That deserves its own look, because host fencing revokes by host NQN.

🤖 Generated with Claude Code

#1674's guard compared host NQNs between the side whose namespace goes and
the side that stays. A Kubernetes node connects with the simplyblock host
NQN (node UID) on one path and with the kernel default on another: on bed
bf28 every client reached the source as nqn.2014-08.io.simplyblock:uuid:<uid>
and the early-prepared target as nqn.2014-08.org.nvmexpress:uuid:<machine>.
The guard took each as a different host, so every cutover freeze waited
for a reconnect that had already happened, and R1 stopped at its first
three cutovers; a discarded target clone was held the same way.

Nothing records which host NQNs belong to one machine, so hosts are matched
by name where possible and every unmatched host on the keeping side counts
for one unmatched host on the removing side. A host is stranded only when
the removing side has more unmatched hosts than that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the target/main Targets the development branch label Nov 19, 2026
michixs and others added 2 commits November 20, 2026 03:12
After a cutover the prepared clone named in tgt_lvol_id no longer exists:
the volume runs on the other cluster under a new record with the same NQN.
Bed bf29 (2026-11-20, round 2): the guard looked the target up by
tgt_lvol_id, found nothing, took every host as stranded and after 120 s
kept the migrated-away source of two volumes. That leaks a full volume per
migration and leaves a same-named volume on the cluster the next round
migrates back to.

_live_target_node_ids: every other live volume exporting the source's NQN,
plus the clone while it still exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
schmidt-scaled pushed a commit that referenced this pull request Nov 20, 2026
michixs and others added 4 commits November 20, 2026 14:15
Bed bf29 (2026-11-20): five volumes of cluster A ran their cutover endgame
at once under sustained fio overwrite. Every shrink round snapshotted its
volume every 15-30 s, every snapshot pinned what was overwritten after it,
and the superseded snapshots were deleted more slowly than new ones were
taken: A went 41% -> 82% -> 99% in 16 minutes and ran out of space.

Three rules (simplyblock_core/migration_space.py) bound what a migration
consumes on its source. They hold back only the migration's own snapshots,
never client IO or client volume creation; cap_warn/cap_crit stay alarms.

1. At most REPL_CUTOVER_MAX_ENDGAMES_PER_CLUSTER (1) endgames per source
   cluster (_acquire_lvs_claim). A queued volume holds no lvstore and keeps
   its ordinary cadence. The claim is re-checked against a fresh task read
   after it is written and ranked by claim time, so volumes that catch up
   in the same runner pass cannot all take the slot, and a running endgame
   is never displaced.
2. An endgame takes its next round snapshot only once none of the volume's
   snapshots is still in deletion (_round_snapshot_hold). A deletion stuck
   for REPL_ENDGAME_DELETION_WAIT_MAX_SEC (300 s) is set aside, logged.
3. No migration snapshot - endgame round, or cadence snapshot of a
   migration-mode volume (snapshot_monitor) - while the source's free space
   is under max(5% of the cluster, 2 x bytes written since the volume's
   last snapshot). Warning event once per wait. A missing or stale (>120 s)
   capacity record holds for at most 120 s, then the snapshot proceeds.

The endgame does not take the lvstore claim while its first round would
wait. A round measured on a pass that then could not snapshot is not
re-measured with the waiting time on the next pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… replication

Bed bf29 R3b (2026-11-20): the target primary node was shut down during an
A -> B migration while one volume was in its cutover endgame. Before the
cutover clone exists the endgame waits for the target primary (the clone is
built through it), and the wait kept the endgame in place: lvstore claim,
round in flight. The endgame takes no snapshot while it waits, and the
cadence snapshot skips a volume in its endgame, so nothing replicated the
volume: no completed cycle for ~20 min, RPO 1280 s. The wait dates from
feb45b9 (2026-08-27), which made it a deferral instead of a burned retry
but left the endgame held through it.

After REPL_CUTOVER_TARGET_DOWN_GRACE_SEC (60 s) of target outage the
endgame is now suspended (_suspend_endgame_for_target): round state dropped,
lvstore claim and cluster endgame slot released, source write cap lifted,
no attempt burned and no back-off. The volume replicates on its ordinary
cadence (still under the migration space rules) until the target is back;
then the catch-up gate admits it to a fresh endgame. A shorter blip keeps
the endgame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound

A direct round's snapshot is never replicated (its pages go straight into
the cutover clone), so the landing volume _prestage_next_landing created
under REP_<repl_round_...> was never adopted, and _discard_spare_landing
only knows the name of the round after it. Every endgame whose first direct
round coincided with a still-running cadence transfer left one full-size
internal volume on the target for good (bed bf29 2026-11-20: three
REP_repl_round_* volumes on cluster 225e0e10 from rounds 2 and 6).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d landing volume

Bed bf29 (2026-11-20): three 10 GiB REP_repl_round_* landing volumes stayed
on cluster A for good. _take_shrink_snapshot replaces the pre-recorded round
name (next_shrink_snap_name) when its kind does not match the round (direct
vs ordinary), when a snapshot of that name exists but is not usable, and
when the name turns out to be taken; each time the landing volume the
replication runner had pre-staged as REP_<old name> was never adopted and
nothing reclaimed it.

Each of the three now discards it first (_discard_landing_named, the body of
_discard_spare_landing taking the name as a parameter): deleted when it
exists, is online and no live replication task lands in it. A task table
that cannot be read keeps the volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
schmidt-scaled pushed a commit that referenced this pull request Nov 20, 2026
…ding-volume leaks) into test/repl-bf14 (bf31)
# the volume replicates nothing (bed bf29 R3b): hand it back to the
# ordinary cadence after a short grace (_suspend_endgame_for_target).
if params.get("cutover_lvs") or params.get("shrink_snap_id"):
task = _suspend_endgame_for_target(task, lvol, tgt_node)
michixs and others added 2 commits November 21, 2026 07:55
… every member online

bf32 2026-11-21 (R6 tgt-dual-p shutdown, target cluster 70edb423): at
05:16:27.95 a write to LVS_7 (primary 484ff828) and to LVS_10 (primary
9154330e) failed while two other nodes' SPDK were gone, and each primary's
SPDK dropped the leadership itself (spdk_lvs_queued_failed_IO). SPDK takes
it back only on the next IO; both were replication targets, and replication
waits for a leader before it sends. The restarts that followed rebuilt the
other members as non-leaders behind the demoted primaries, and nothing
restores the leadership of a healthy LVS: 76 min leaderless with every
member online, until a manual _recover_leaderless_lvs.

- storage_node_ops.healthy_lvs_leadership_state / restore_healthy_lvs_leadership:
  leaderless-but-healthy = cluster active/degraded, owner online primary,
  every member ONLINE and answering get_lvstore, none leader; then the
  guarded _recover_leaderless_lvs on the primary (lock+cooldown, no
  restart/port-allow task, JM quorum, proven-free twice, reload-then-grant).
  Not on sync-site or two-node arbitrated clusters.
- node monitor: per primary, probe every 30 s, act after 60 s confirmed
  leaderless, recovery on a background thread.
- replication: a leaderless-target wait >= 60 s asks that recovery (every
  120 s per task); replication still grants nothing itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…anding volume under creation

bed bf32 (2026-11-21, cluster 70edb423): two replication workers enqueued the
same chain ancestor (snapshot 4d20399d) 100 ms apart. _add_task's dedupe is a
scan followed by a write, so both passed it and two tasks (5571400c,
a6f9b7b7) existed for one snapshot. The worker pool dedupes by task uuid and
_dispatch yielded only to a RUNNING sibling, so both ran at once: one created
landing volume 5eeb989b while the other took it (in_creation, 2 s old) for a
dead attempt's leftover and deleted it. The delete raced the creator's
registers on the secondary/tertiary, which kept a stale LVOL_2398 that later
crashed SPDK on both nodes.

- DBController.ensure_snapshot_replication_task: the task write and a
  repl_snap_task/<cluster>/<snapshot> claim key in ONE transaction; _add_task
  creates every snapshot_replication task through it.
- Runner: _Workers runs one task per snapshot at a time; _dispatch lets only
  one live task of a snapshot run (RUNNING, then landing owner, then oldest),
  looked up via the repl_snapshot_id index.
- Half-created landing cleanup only deletes a provably abandoned volume: not
  being created in this runner (_LANDING_CREATING), no other live task of the
  snapshot, older than REPL_LANDING_ABANDON_SEC (600 s). After the delete it
  confirms the bdev is gone on every lvstore member (sync delete of a leftover
  registration; durable lvol_sync_del task where unconfirmed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
getattr(primary, "tertiary_node_id", "")):
if nid and nid not in ids:
ids.append(nid)
except KeyError:
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

target/main Targets the development branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants