Skip to content

feat: erasure-coding resource floors; multi-attach array LUNs with fast takeover; array adapters - #1626

Open
schmidt-scaled wants to merge 54 commits into
mainfrom
feat/ec-floors
Open

schmidt-scaled wants to merge 54 commits into
mainfrom
feat/ec-floors

Conversation

@schmidt-scaled

@schmidt-scaled schmidt-scaled commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR has grown into three parts that build on each other:

  1. Erasure-coding resource floors for an n+k scheme, the raid_devices cluster setting, and relaxed migration at exactly n+k nodes. Design: https://claude.ai/artifact/ANm8UNf6MQZAGdmWpYS5Pb
  2. Multi-attach array LUN mode (1+0). Each storage node serves one LUN of an external array (EBS io2 Multi-Attach for now), and the LUN is attached to FTT+1 nodes. When a node is lost, another attached node takes its LUN over within seconds. Every volume depends on every LUN, so the takeover time is the I/O stall.
  3. Array adapters and a credential store (the storage_arrays package and credentials), merged in from feat/array-credentials. These commits are independent of 1 and 2 and can move to their own PR if reviewers prefer.

Companions: simplyblock-operator #887 (same branch name; includes the CSI keep-alive change) and ultra #236 (alceml standby, persistent reservations, warm standby, io_uring bdev).

1. Resource floors

Floors are binding, and journal devices are excluded from every count:

Floor Value
Data devices per node 1
Data devices, --raid-devices n+k
Data devices, unprotected n+2k
Failure domains, if used n+k
Nodes n+2k (single-node 1, two-node 2)
  • New simplyblock_core/floors.py (cluster_floors, data-device counting, violation text).
  • Fresh activation checks all floors. Re-activation keeps the current gate so recovery is not blocked. The FD minimum at fresh activation is max(k+2, n+k).
  • Volume create, planned shutdown (--force downgrades to a warning) and node removal refuse below a floor. The old rebalancing-headroom formula is removed.
  • Monitor: DEGRADED while a floor is violated (never SUSPENDED for that alone), with a FLOOR cluster event naming the floor.
  • --raid-devices on cluster create / cluster add and the v1/v2 APIs. It is immutable and persisted as allow_device_colocation for distribs.
  • needs_relaxed_placement: < becomes <=, so a cluster at exactly n+k nodes migrates relaxed.
  • The affinity_floor / allow_device_colocation RPC params stay behind constants.DISTR_AFFINITY_FLOOR_RPC = False until the ultra data plane (feat/affinity-floor) is deployed.

2. Multi-attach array LUNs

  • Setup (phases 2–4): cluster options, device identity by serial/WWN, sn configure --lblk-journal-only, array plugins (AWS EBS io2) with KMS credentials, and LUN namespaces exported with an explicit NGUID. Device-mapper nodes are served through bdev_uring, because dm-multipath refuses RWF_NOWAIT.
  • Loss detection (services/lun_watchdog.py):
    • Probes every node that serves a LUN every 0.5 s: SPDK RPC with a 0.8 s timeout, then the node agent.
    • A node is lost at once when SPDK is down, and after 2 s with no answer otherwise. Never while the agent reports SPDK running.
    • Observer guard: no takeover when more nodes fail than the FTT allows, or when no other node answered recently.
  • Takeover (controllers/lun_takeover.py):
    • Fencing: fence by NVMe persistent reservation. The standby's alceml opens with its key and preempts the failed node's reservation in milliseconds; the EBS detach follows in the background. Without reservation support, the array fence runs first.
    • Standby choice: the standby is chosen while the fence runs. An export that fails on an unresponsive standby moves on to the next candidate.
    • Path switch: the peers' paths are switched in parallel.
    • Cached array connection: the array connection is cached and kept warm.
  • Clients: the multi-attach NVMe keep-alive is 3 s (MULTI_ATTACH_CLIENT_KEEP_ALIVE_TO). Regular clusters stay at 8 s.
  • Recovery:
    • A fenced node that comes back becomes a standby again, but not while its fence is still running or its SPDK does not answer.
    • Auto-restart admits up to max_fault_tolerance nodes down.
    • An array LUN is never resolved by its old kernel name.
    • LUN growth and stats run on the serving node.
  • Monitor hardening (multi-attach only):
    • A home node's I/O error on a LUN served elsewhere is treated as a peer path error.
    • A node's own LVS port that SPDK reports blocked gets the fence grace.
    • Port blocks on more nodes than the FTT at once do not set nodes DOWN.
    • Reintegration runs off the node check.
  • Node agent: keeps the log tail and exit state of an SPDK container that died on its own, in /etc/simplyblock/spdk-crash-logs.

3. Array adapters and credentials

  • credentials: a KMS-backed store for external-system secrets.
  • storage_arrays: an adapter framework, a fake-array test harness and adapters for the supported array families, wired in as multi-attach array plugins. Not yet tested against real arrays.

Tests

  • Unit: full suite 7205 passed. The 43 failures are in four files that fail on Windows on main too (deploy-order script, device_health_info, lblk_persistent_names, spdk_proxy). New suites: test_cluster_floors, test_lun_watchdog, test_lun_takeover (reservations, standby failover, fence/reintegrate races), test_multi_attach_device_events, test_multi_attach_port_check, test_multi_attach_auto_restart, test_spdk_crash_logs.
  • Beds: EBS io2 Multi-Attach, 20 GiB LUNs, fio on every volume. A case fails above an 11 s I/O stall, on EIO, on a hang, on a suspension, or when the cluster does not heal.
    • FTT1, 4 nodes (22cff86): 19/19 passed: network cut, SPDK kill, reboot and hard reset of each node, plus short network cuts. Worst stall 7.3 s.
    • Resize on FTT2 (7a4a765): 6/6 passed: growth refused while a node is out, growth after a takeover with capacity following it, takeover of the grown LUN, reboot and hard reset after growth, fill into the grown capacity.
    • FTT2, 5 nodes (7a4a765): 7/8, including a hard reset of a node serving 3 LUNs at 6.8 s and two dual outages. Case 8 (reboot + hard reset) chose the rebooting node as standby; that is fixed in 44b4cd7.
    • Running now: a full FTT2 rerun on 44b4cd7 and a full FTT1 rerun on 3e921e0.

Open decisions for review

  1. Two-failure-domain clusters now report DEGRADED permanently (domains < n+k).
  2. Replace the k+2 FD rule by n+k outright, or keep max?
  3. Two-node 1+1 with one plain device per node is refused (device floor 3).
  4. Any 2-member cluster counts as two-node, not only arbitrated ones.
  5. Strict node anti-affinity still SUSPENDS below n+k online nodes (kept unchanged).
  6. The per-node device floor is checked at fresh activation only; add/restart/expansion are reported by the monitor, not gated.
  7. Warm standby for large LUNs: the takeover opens the alceml cold, which is fine at 20 GiB but takes minutes at 100 TiB. ultra pull thread from spdk and push to prometheus pushgateway #236 supports standby instances with a page map refreshed every 2 s, at the cost of holding every attached LUN's map in hugepages on every attached node (about 3× the page-map memory per node at FTT2).
  8. The port-block grace and the correlation guard are gated to multi-attach. Regular clusters could use the same rules but need their own review and soak.
  9. Split part 3 into its own PR?

🤖 Generated with Claude Code

michixs and others added 4 commits October 30, 2026 16:54
Replaces the reweight/rebalance after uneven growth (8fd013c): the
weight of an existing device in a live distrib map is never changed.

- cluster grow-devices (CLI, v2 POST /clusters/{id}/grow-devices) rescans
  every data device and grows all of them by the SMALLEST growth factor
  of their block devices (1.5 / 2.0 -> every device grows by 1.5). A
  device that did not grow stops all growth; so does an offline device or
  node. Sizes are committed only after every alceml reopened, so a
  failure in between leaves the cluster uniform; repeated runs are no-ops.
- Cluster-map weights (placement_size) never change. Removed: rebalance
  option, --no-rebalance, ?rebalance=, Cluster.reweight_pending, map push
  and rebalance queueing, devices_reweighted event.
- Single-device growth (sn resize-device, v2 .../devices/{id}/resize) is
  removed: growth is cluster-wide only.
- NVMeDevice.backend_size records the rescanned block-device size. alceml
  always spans the whole block device (no size limit: alceml takes
  blockcnt*blocklen of the backend on create and on resize reopen), so the
  capacity statistics scale npages_nmax down to the device size; the rest
  is reported as unused_size. With frozen weights placement fills every
  device in proportion to its weight, so the rest is never reached.
- A block device smaller than its device is refused, logged as an error
  event, and raises the new device_shrunk alert until a rescan sees the
  restored size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Product rules of 2026-11-09 for an n+k cluster: >= 1 data device per
node, >= n+2k online data devices (n+k with the new raid_devices
declaration), >= n+k failure domains when they are used, >= n+2k nodes;
journal devices never count. Single-node (1 node, n+k devices) and
two-node (2 nodes) clusters keep their topology overrides.

- simplyblock_core/floors.py: Floors, cluster_floors, count_data_devices,
  floor_violations, evaluate.
- Cluster.raid_devices (create-time, immutable) with `cluster create/add
  --raid-devices` and the v1/v2 API; --strict-node-anti-affinity help no
  longer claims n+k+1 nodes.
- Fresh activation refuses below any floor (a node without data devices
  included); re-activation keeps the old device gate. The fresh FD
  activation needs max(k+2, n+k) domains.
- Volume creation refuses below the floors of the volume's scheme.
- Planned shutdown (force overrides) and node removal refuse when the
  cluster without the node is below a floor; the rebalancing headroom
  formula is gone.
- Monitor: DEGRADED (never SUSPENDED) while online resources are below a
  floor, with a FLOOR cluster event naming it; suspension rules unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- needs_relaxed_placement: `<=` n+k member nodes (exactly n+k has no
  strict target for a moved chunk; closes a migration dead end).
- migration_affinity_floor(cluster): "relaxed" with raid_devices, else
  "device"; passed by every migration runner as affinity_floor.
- rpc_client: optional affinity_floor on distr_migration_failure_start /
  distr_migration_expansion_start and allow_device_colocation on
  bdev_distrib_create, sent only while constants.DISTR_AFFINITY_FLOOR_RPC
  is True (False until the ultra feat/affinity-floor data plane ships;
  older data planes reject unknown parameters).
- create_lvstore persists allow_device_colocation on raid_devices clusters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the target/main Targets the development branch label Nov 9, 2026
michixs and others added 3 commits November 12, 2026 22:40
… restart

A new lblk device now gets a 4096-byte AIO base bdev; the device's logical
block size is used only when 4096 is impossible (logical > 4096 or not a
divisor, size not a 4K multiple, partition start not 4K aligned). 512e
devices (EBS, most array LUNs) previously ended up with 512-byte bdevs.

alceml's on-disk layout depends on the base bdev block size, so a device
keeps the size it was formatted with: NVMeDevice.aio_block_size (0 = legacy,
created without a block size, which keeps being re-created that way) is
passed on every bdev_aio_create (addAioDevices at add-node and restart,
restart_device, new_device_from_failed), and the bdev's block size is
checked after creation. The size is also kept on the node's lblk_devices
entry, which a control-plane service that predates the model field keeps on
a read-modify-write; the restart path recovers it from there and the
reconcile restores it on the device record.

The node agent reports logical/physical block size and partition start in
the block-device inventory. Without them (older agent) 4096 is chosen when
the size allows it, and a new device whose 4096 create is refused is
retried without a block size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SPDK's bdev_aio submits every I/O on a block device with RWF_NOWAIT so
io_submit never blocks the reactor. device-mapper honours RWF_NOWAIT only
when every target does; dm-multipath does not, so every AIO submit to a
multipath map fails with EOPNOTSUPP and alceml cannot even format it.
Dropping RWF_NOWAIT is not an option (a blocking submit during a path
failover would freeze the reactor), so device-mapper nodes are served by
SPDK's io_uring bdev instead.

Each lblk device now has an I/O engine, NVMeDevice.lblk_io_engine ("aio"
default, or "uring"), also kept on the node's lblk_devices entry next to
aio_block_size. Rule (utils.lblk_io_engine_for): a kernel name starting with
dm- (including a kpartx partition of a multipath map) or an lsblk TYPE in
mpath/lvm/crypt/dm gets uring; whole disks, sd/nvme partitions and md
arrays stay on aio. For an agent that predates the fields, a /dev/dm-,
/dev/mapper/ or by-id dm-name-/dm-uuid- path still selects uring. The
recorded engine is kept, except that the rule's uring wins over a recorded
aio (aio cannot serve the device; logged). The engine does not change the
on-disk format.

The node agent reports KNAME in the block-device inventory;
resolve_lblk_entries carries kname, type, the derived engine and the
recorded engine. rpc_client gains bdev_uring_create/delete/rescan, and
every create (addAioDevices, restart_device, new_device_from_failed) and
the growth rescan go through engine-aware helpers (create_aio_bdev with
io_engine, rescan_lblk_bdev, delete_lblk_bdev). The bdev name (aio_<serial>)
and bdev_type "aio" are unchanged, and the block-size rules apply to both
engines. An existing bdev's engine is taken from its driver_specific info.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, paths)

A cluster whose data devices are protected array LUNs presented to several
nodes runs 1+0; an owner's LUN is reopened on another node on failure
(phase 3). This adds everything below the takeover itself. Every new path
is gated on Cluster.multi_attach / NVMeDevice.multi_attach; regular
clusters are unchanged.

- cluster create/add --multi-attach and --ftt 1|2 (CLI, v1/v2 API,
  Cluster.multi_attach): requires --raid-devices, ha_type 'ha', not
  single-node; forces 1+0 and max_fault_tolerance = ftt. --ftt alone is
  refused; the npcs>=2 rule for FTT2 does not apply to multi-attach.
- floors: nodes >= FTT+1 (>= 3 for FTT2), >= 1 data device per node (one
  LUN per node recommended), no single/two-node override.
- NVMeDevice.multi_attach / lun_wwn / serving_node_id / node_paths; lblk
  devices of a multi-attach cluster are flagged at add-node / restart
  (WWN from the lsblk inventory, else the serial).
- Device subsystem NQN <cluster nqn base>:lun:<uuid>; namespace UUID
  derived from the device UUID (uuid5) -- the device UUID itself is the
  alceml bdev's UUID alias and would collide with the owner's loopback
  bdev; device_controller now adds the namespace with that UUID too.
- Per-node controller-ID ranges: StorageNode.cntlid_block claimed in one
  FDB transaction (DBController.claim_cntlid_block), range
  [i*64+1, i*64+64], at most 1022 nodes; subsystem_create max_cntlid.
- Peer controllers to a multi-attach device: multipath mode and
  per-controller hold (ctrlr_loss/fast_io_fail 120 s, reconnect 2 s).
- alceml stuck_io_timeout_ms (60 s, ultra feat/alceml-stuck-io-timeout)
  for multi-attach devices only; sent only when non-zero.
- Owner loopback: the owner's distribs map its own multi-attach devices
  as remote_alceml_<uuid>n1; _connect_to_remote_devs, the coverage gate,
  sync_remote_devices_from_spdk and reconnect_dropped_remote_devs include
  them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
if not subsys_found:
logger.info("Creating subsystem %s", subsystem_nqn)
ret = rpc_client.subsystem_create(subsystem_nqn, 'sbcli-cn', alceml_id)
ret = rpc_client.subsystem_create(
michixs and others added 3 commits November 13, 2026 12:31
When the node serving a multi-attach LUN fails, the monitor starts a
takeover per LUN (controllers/lun_takeover.take_over_lun) instead of
marking the device unavailable:

1. fence the failed node at the array first (array plugin, host = node
   hostname); no plugin or no acknowledgement -> no takeover, event;
2. standby = owner's LVS secondary, else tertiary, else the owner itself;
   ONLINE, not fenced from the LUN;
3. the standby's agent (new POST /lun_info, docker + k8s) finds the LUN
   by WWN and reads block 0 for the alceml signature (read-only);
4. on the standby: aio base bdev, alceml OPEN-ONLY (pba_init_mode=1,
   never 2/3), PT, subsystem under the same host-independent NQN with the
   standby's cntlid range, same namespace UUID, standby listeners;
   partial exports are torn down (never formatted);
5. record serving_node_id / node_paths / nvmf_ip atomically (device stays
   ONLINE, no cluster-map change), then add the new path to
   remote_alceml_<uuid> on every ONLINE node (active_passive) and detach
   the dead node's path.
Idempotent (re-run only completes paths), FDB lease + process lock per
LUN, 90 s deadline inside the 120 s hold. No readable standby: hold,
unavailable when the hold expires. Any other failure: unavailable.

- simplyblock_core/array: ArrayPlugin (fence/unfence/capabilities),
  registry over entry-point group simplyblock.array_plugins,
  FakeArrayPlugin (tests; never registered by default);
  Cluster.array_plugin/array_endpoint/array_options,
  cluster_ops.set_array_plugin. NVMeDevice.fenced_node_ids.
- monitor: set_node_offline, set_node_schedulable and the lost-site
  force-offline share _set_node_devices_unavailable; takeovers start
  right after the OFFLINE write, before (and in parallel with) the LVS
  ANA failover. LUNs of the node served elsewhere are left alone.
- connect / repair: transport from the serving node; multi-attach
  controllers always inspected for missing paths, no wait on the dead
  path's transient state, active_passive policy.
- owner return: restart builds no stack for LUNs served elsewhere or
  fenced from it, keeps those records (not REMOVED when unseen), does
  not flip their status; discovery and add-node never adopt (format)
  another node's LUN (matched by WWN/serial); remote-device connect /
  sync / reconnect include an offline owner's LUNs served by an ONLINE
  node; graceful shutdown leaves them alone; health check skips their
  local stack. No failback in this phase.
- device_set_state: CAUSE_LUN_TAKEOVER allows ONLINE while the owner is
  down if the serving node is ONLINE; ONLINE refused while the serving
  node is down or fenced.
- lun_takeover.unfence_node for an operator (never automatic).

Regular clusters: every new path is gated on cluster/device multi_attach.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ntials, AWS EBS io2)

simplyblock now creates, multi-attaches, grows, fences and deletes the LUNs
of a multi-attach cluster itself, through a vendor plugin. Every new path is
gated on a multi-attach cluster with an array plugin (and, for LUN creation
at node add, Cluster.array_lun_size); regular clusters are unchanged.

- KMS: put_secret / get_secret / delete_secret on the KMS base, HCPClient
  (kv v2 on kv_mount) and LocalKMS (kv_store, prefix secrets/ apart from the
  DEK keys/); values in and out as SecretStr, plaintext only at the store.
- array plugin interface: connect(endpoint, credentials, options),
  capabilities() dict, ensure_host, create_lun, attach, detach(force),
  resize (grow only), get, delete_lun, list_luns; fence / unfence default to
  a forced detach / attach. Every created LUN carries the
  simplyblock-cluster-id tag; untagged or foreign LUNs raise
  ForeignLunError. plugin_for_cluster connects with credentials read from
  the KMS (cluster/<id>/array/<array id>). FakeArrayPlugin implements it all
  in memory. Built-in fallback for the aws-ebs entry point.
- registration: controllers/array_controller.register_array / get_arrays /
  delete_array; POST/GET/DELETE /api/v2/clusters/{id}/arrays; CLI cluster
  set-array (--credentials-file) / get-array / remove-array. The plugin is
  loaded, connected and its capabilities checked (max_attachments >= FTT+1)
  before anything is saved; credentials never reach the cluster record or a
  response. cluster_ops.set_array_plugin delegates to it.
- aws-ebs plugin (simplyblock_core/array/aws_ebs.py, entry point aws-ebs):
  io2/io1 Multi-Attach volumes in options.az; LUN id = volume id, wwn = the
  NVMe serial vol<id without dash> (lsblk SERIAL); host = EC2 instance id;
  attach with a free device name, forced detach as fence (waits until the
  attachment is gone), modify_volume grow (waits for optimizing/completed),
  delete after detaching all. Access key / secret from the KMS or the
  instance role. Agent cloud info now carries availability_zone / region.
- lifecycle: node add creates the node's LUN (ensure_host, create, attach,
  wait for it in the agent inventory) and continues as an lblk add with it
  (a lone local device becomes the journal); LVS role changes (activation,
  expansion, and a periodic pass from the health-check service) present each
  LUN to owner + secondary + tertiary + serving node, fill node_paths, and
  detach from nodes that lost the role - never the serving node, never
  re-attaching a fenced node; grow-devices --lun-factor grows every LUN at
  the array and waits until every node sees it (agent rescan now resizes
  dm-multipath maps); node delete detaches; cluster delete deletes the
  cluster's LUNs and the KMS secret.
- takeover fences by NVMeDevice.array_lun_id / StorageNode.array_host_id
  when the plugin created them, else by WWN / hostname as in phase 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CI type checker failed on 800f763 (claim_cntlid_block assigned an
Optional to an int); phases 3-4 added six more: the Python < 3.10 entry-point
fallback (requires-python is >= 3.11), getattr with a str default on a typed
field, a loop variable reused outside its except block, an untyped work list,
and two Optional list assignments in _connect_to_remote_devs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread tests/unit/test_array_lifecycle.py Fixed
Comment thread tests/unit/test_array_lifecycle.py Fixed
michixs and others added 7 commits November 13, 2026 13:06
These fixtures predate the resource floors (simplyblock_core.floors); give
them enough resources instead of weakening the floors:
- dual_ft_e2e: 3 data devices per primary (6 >= n+2k = 5 for 1+2)
- dhchap_pool_level: the second node gets one data device (>= 1 per node)
- shutdown_no_suspension: a third bystander node, so the cluster is not a
  two-node one and the shutdown floor guard does not refuse
- cluster_suspend_recovery: 5 ONLINE nodes in the ACTIVE cases so the
  monitor's floor DEGRADED overlay does not apply

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	simplyblock_core/controllers/events_controller.py
Combines the lblk changes of #1572 (uniform-only growth, 4 KiB AIO block
size, io_uring for device-mapper nodes) with the multi-attach work of this PR.

Conflict resolutions:
- grow-devices: #1572's uniform growth replaces the reweight/rebalance path
  phase 4 had extended. grow_cluster_devices(cluster_id, lun_factor=None) now
  runs the array LUN resize first (array_controller.grow_luns) and grows
  nothing on the cluster when any LUN failed; otherwise it runs #1572's
  growth (_grow_cluster_devices) and adds "luns" to the result. API, CLI and
  DTO follow #1572 plus lun_factor/luns; --no-rebalance is gone.
- resolve_lblk_entries: both sides' carried fields (LUN identity; block-size
  and I/O-engine inputs).
- LUN takeover: the standby's base bdev is created through #1572's
  create_aio_bdev (recorded block size, verified after creation; io_uring when
  the standby's own path to the LUN is a device-mapper node), and torn down
  through delete_lblk_bdev.
- Tests: the reconcile tests of #1572 patch the multi-attach foreign-LUN
  lookup (a DB read); the array growth tests follow the uniform growth.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a multi-attach cluster whose array plugin creates each node's data LUN at
add-node, a storage node has only its journal disk at configure time. The
existing lblk layout either refuses one device (two-unit minimum) or splits a
single whole disk into a journal and a local data partition, and a local data
device on a 1+0 multi-attach cluster is not shared: losing the node loses its
data.

--lblk-journal-only (CLI sn configure and the k8s node_configure) takes exactly
one whole disk, named with --blk-names or --blk-serials, flags it as the
journal and never partitions it. add-node then adds the array LUN as the data
device, which satisfies the two-unit minimum there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validate_node_config (run by the node agent when add-node reads the node's
config) still required two lblk entries, so a node configured with
--lblk-journal-only was refused at add-node ("Please run sbcli sn configure
before adding the storage node"; agent log: "lblk mode requires at least 2
partitions or SSDs per node; node 0 carries 1"). Found on the multi-attach
bed. A single entry is now valid when it is the flagged journal; add-node
checks the minimum again once the array LUN is added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the multi-attach bed: add-node failed with "aws-ebs: host
ip-172-31-36-102 has no EC2 instance id". Two causes:

- Docker deployments start the node agent with WITHOUT_CLOUD_INFO=True
  (startup time, f93a3c7), so the agent never reports the instance.
- _amazon_info called ec2_metadata.EC2Metadata(session=...), which the
  installed ec2_metadata no longer accepts (TypeError), so it returned None
  on every AWS instance even where the probe runs.

The aws-ebs plugin now resolves a host without an instance id by the node's
private IP (exactly one pending/running instance in the volumes' zone);
host_initiators carries the IP, from add-node's node address and from
StorageNode.mgmt_ip. _amazon_info reads the identity document over IMDSv2
directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the multi-attach bed: activation failed because the owner's loopback
controller to its own LUN could not register its namespace bdev ("Unable to
add uuid:e6d24bda-... alias for bdev remote_alceml_<id>n1"). The namespace
was added with the derived UUID, but SPDK filled the NGUID from the exported
PT bdev's UUID, and the host side registers the namespace bdev under that
NGUID. The ptnonexcl bdev derives its UUID from its name (bdev_ptnonexcl.c,
uuid5 of "<alceml>_PT"), which is the same on every node, so on any node that
both holds the PT bdev and consumes the LUN (the owner's loopback, a takeover
node) the two UUIDs collide.

Multi-attach namespaces are now added with nguid = the derived namespace
UUID's hex digits, at node add, device restart and takeover. Regular devices
call the RPC exactly as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michixs and others added 9 commits November 13, 2026 17:42
…uspends

Found on the multi-attach bed: the first storage node loss suspended the
cluster at the moment the node went offline. The suspend rule counts lost
nodes against the erasure-coding parity, k = npcs, which is 0 on a 1+0
multi-attach cluster. The suspend then shut the healthy nodes down, the LUN
takeover's standby among them, so the takeover (fenced in 3 s, standby found)
failed at the export with "Could not reach remote".

On a multi-attach cluster a lost node's LUN is taken over by a standby, so the
node-loss tolerance is the cluster's FTT (max_fault_tolerance): with FTT1 one
lost node is DEGRADED, two are SUSPENDED (a LUN is attached to two nodes).
Regular clusters, 1+0 included, keep k = npcs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict: simplyblock_web/api/v1/cluster.py, cluster create arguments. main
replaced hashicorp_vault_settings with kms_settings (7581b22, KMIP backend
and generalized KMS config); kept the multi-attach arguments (raid_devices,
multi_attach, ftt) plus kms_settings.

Follow-ups for main's KMS change:
- KMIPClient (new on main) did not implement the generic-secret methods this
  PR made abstract on KMS (put/get/delete_secret), so it could not be
  instantiated. It now refuses them with an explicit KMSException: storing
  array credentials on a KMIP appliance is not implemented or tested yet;
  Vault/OpenBao, the local KMS, or an array without credentials (aws-ebs with
  an instance role) work.
- LocalKMS now imports DBController lazily; the secret tests patch it there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…offline

Found on the multi-attach bed (F1, hard power-off of an owner under fio): the
takeover started 6 minutes after the power-off, when the node was finally
declared OFFLINE, long after the peers' 120 s hold, so the client got EIO.
The OFFLINE verdict waits for every online peer to report the node's journal
disconnected; the node was UNREACHABLE 20 s after the power-off.

- The takeover of the LUNs a node serves starts once the node has been
  UNREACHABLE for MULTI_ATTACH_UNREACHABLE_TAKEOVER_SEC (15 s). The fence at
  the array comes first, so a node alive behind a management-network fault is
  cut off from the LUN before the standby opens it. The timer restarts with
  every new UNREACHABLE episode.
- MULTI_ATTACH_HOLD_SEC 120 -> 150: it must cover detection (~20 s), the
  trigger delay and the takeover deadline (90 s).
- A node that comes back ONLINE without a restart (a blip long enough to
  start a takeover) retires what it still holds of the LUNs taken over from
  it (lun_takeover.retire_stale_local_stack): its own loopback path in the
  LUN's controller, then the stale subsystem, PT and alceml. Nothing is
  formatted. A restart never builds these stacks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	simplyblock_cli/clibase.py
…y again

After a takeover, the failed node stayed fenced from its LUN until an
operator called unfence_node, so a node that returned after an outage was
never a standby again and a second failure of the serving node found no
standby.

The monitor now checks an ONLINE node that a takeover fenced (at most every
30 s) and unfences it at the array when nothing it held can still reach the
LUN: the LUN is served by another ONLINE node, the node holds no alceml or
PT bdev of it (a stale stack is retired first), and its host shows no disk of
the LUN. A host that still shows the LUN (e.g. a multipath map still
queueing) stays fenced for the operator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t serve

2026-11-19 FTT1 bed, SPDK container kill of a node serving two LUNs: the
takeover fenced it (EBS detach), it restarted within 22 s, and the device
monitor's lblk hot-removal sweep then found the fenced owner's LUN missing
from its host and called device_remove. On a 1+0 cluster a removed device
is lost stripes: read errors on every lvstore, nodes down, cluster
suspended.

- device_monitor: the presence sweep and the hung-IO check skip a
  multi-attach device that is not servable on the node (served elsewhere,
  or the node is fenced from it).
- device_repair_due / device_repair: never rebuild such a device's stack on
  its owner.
- unfence_node clears the fence record before the array call. The
  attachment reconcile detaches every host the record lists as fenced, so
  attaching first raced with it (the bed: attach at :09, reconcile detach
  at :10). With the record cleared the reconcile presents the LUN itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
grow-devices refused every device whose LUN another node serves after a
takeover ("grow it once its owner serves it again"). Nothing ever moves
a LUN back to its owner, so after the first outage the cluster could
not grow at all. Growth now rescans and reopens the alceml on the
serving node, over the base bdev and path the takeover recorded.

With --lun-factor the checks ran only after the LUNs were resized at
the array: a refusal left the LUNs grown and the cluster not, and a
re-run grew them again. Every device is now checked before the array
is touched.

A node that kept its base bdev over a LUN from before a growth (a
former owner) still sees the old size; a takeover onto it failed the
size check. The takeover now rescans the block device and the bdev
before giving up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The capacity and stats collector asked the owner for the capacity and
I/O stats of every device. After a takeover the owner has no alceml for
it, the call failed and the device was skipped, so the cluster capacity
dropped by one device per taken-over LUN (and grown capacity was never
seen for it). Read them on the serving node, over the base bdev the
takeover recorded; skip the device only while that node is not online.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With 1+0 every volume waits on a lost node's LUN until a standby serves it.
On the FTT1 bed the stall was 9-10 s for a killed SPDK and 24-35 s for a
reboot, hard reset or 20 s network cut: the takeover started from the
regular monitor's UNREACHABLE/OFFLINE verdicts plus a fixed 15 s wait.

- services/lun_watchdog (started by the storage node monitor): probes
  every node serving a LUN every 0.5 s (spdk_get_version, 0.8 s timeout;
  the node agent while it fails). Lost at once when the agent reports
  SPDK not running, after 2 s when neither answers; never while SPDK runs.
  A node answering again within the window is not taken over. No takeover
  while more nodes fail than the cluster tolerates or no other node
  answered recently (this monitor cut off).
- The 15 s confirmation (MULTI_ATTACH_UNREACHABLE_TAKEOVER_SEC) is gone:
  the regular monitor's UNREACHABLE starts the takeover at once (backstop).
- Takeover: the array connection is cached and warmed (connecting cost
  ~1 s per fence) with one retry on a fresh one; EBS fence polls the
  detach every 0.2 s instead of 2 s; the standby is chosen (read-only
  LUN probe) while the fence runs, nothing is built before its ack; the
  new path is added on all peers in parallel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An NVMe protocol without protocol_family created SCSI LUNs that the NVMe
hosts never see. The family now follows the plugin's protocol, and a stated
family that contradicts it is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Options set through Kubernetes (StorageCluster.spec.multiAttach.array.options)
arrive as strings. Adapters tested booleans by truthiness, so eradicate:
"false" on Pure eradicated deleted volumes, thin: "false" on IBM stayed
thin, and Hitachi host_mode_options "2,13" failed. ArrayAdapter.option now
reads a string as its default's type (true/false/yes/no/on/off/1/0, numbers,
comma-separated lists) and a leading { as JSON (PowerMax port_group map); a
value that cannot be read fails at registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread simplyblock_core/array/san.py Fixed
The vendor fakes answer the management API but present nothing to a host.
testing/lio.py mirrors a fake's volumes, hosts and mappings into the kernel's
iSCSI target: every volume a loop-backed block LUN, every host initiator an
ACL, every mapping an ACL LUN, an unmap a real fence. Sizes and identities are
read through the vendor's own adapter in process, so the host sees what the
array plug-in reports (identity as the LUN's unit serial, lsblk SERIAL).

- fakes: lio_volume_ids / lio_host_iqns / lio_all_iqns (10 iSCSI vendors).
- serve(after_request=...): reconcile after every request; jobs of
  asynchronous arrays (Hitachi) complete on GET polls.
- mock server: --lio, --lio-target, --lio-portal, --lio-dir, --lio-allow.
- testing/lio_selftest.py: the multi-attach plug-in (array.san) against the
  mock over HTTPS with two iSCSI initiators of one host as two nodes, the node
  agent's scan_for_new_luns / find_lun_devices / rescan_block_device on real
  block devices: attach, multi-attach data, fence (I/O fails), growth,
  unfence, delete. Passes 9/9 for all 10 iSCSI vendors (WSL2, kernel LIO).
- test_lio.py: the mirror against MemoryLio for every vendor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michixs and others added 2 commits November 20, 2026 19:35
2026-11-20 FTT2 bed, dual reboot sn2 + hard reset sn3: the standby chosen
for sn3's LUN was sn2, whose reboot had just begun (its agent still
answered the path probe). Every RPC to it then hung on retries (~28 s),
the reservation fence failed, the array fence followed and the export
was retried on the same dead node. The takeover failed although a third
host could read the LUN; the LUN went unavailable and the 1+0 cluster
suspended.

- An export that fails on a standby is retried there once only if its
  SPDK still answers; otherwise the next standby candidate is chosen
  (excluding those tried) until the deadline.
- The activation (role) polls use a 1 s client without retries, so a
  dead standby shows within MULTI_ATTACH_ACTIVATION_WAIT_SEC.
- No cleanup calls into a standby whose SPDK no longer answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… logs

2026-11-20 FTT2 bed, hard reset of sn3: one unavailable LUN made SPDK fence
the client port of every LVS of the 1+0 cluster; within 4 s the monitor set
three healthy nodes DOWN (every liveness probe passing), the node floor broke
and the cluster suspended.

Multi-attach clusters only; regular clusters unchanged:
- storage_node_monitor: a node's own LVS port that SPDK reports blocked gets
  the fence grace peer-owned ports have (PEER_PORT_FENCE_GRACE_SEC) instead
  of the 1 s re-read, so the stale-fence remediation can act first.
- storage_node_monitor: more nodes than the cluster tolerates failing only
  their port check within PORT_FAILURE_CORRELATION_SEC are not set DOWN; one
  cluster event (storage_events.snode_port_check_correlated) instead.
- lun_watchdog: an "SPDK running" agent answer restarts the silence; it
  survived such answers, and one agent timeout read as "no answer for 6.1s".

Node agent (docker): before a start/kill/cleanup removes an SPDK container
that had already exited on its own, its last 30000 log lines and exit state
are kept in /etc/simplyblock/spdk-crash-logs (newest 10). The three SPDK
deaths of that cascade left no trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@schmidt-scaled schmidt-scaled changed the title feat(floors): erasure-coding resource floors, raid_devices, relaxed migration at n+k feat: erasure-coding resource floors; multi-attach array LUNs with fast takeover; array adapters Nov 20, 2026
A SCSI LUN reached over several paths appears once per path as an sd disk
and once as the dm-multipath map over them. The Unified DataStore used one
raw path, beside multipathd. Now:

- node agent inventory reports multipath maps (type mpath, /dev/mapper/<name>,
  kname dm-N, the identity of their paths, by-id dm-uuid-mpath-<wwid> first);
  the path disks are marked with their multipath_map and stay held by it.
- find_lun_devices / array_controller._find_in_inventory return the map
  instead of its paths; a path of a map is never used alone.
- resolve_lblk_entries resolves a configured LUN to its map, not a path; the
  map's dm-N kernel name selects the io_uring engine (9783742), which
  device-mapper multipath needs (it refuses RWF_NOWAIT; data plane: ultra #236).
- lio_selftest --multipath: one host over two portals under multipathd:
  the LUN is found as the map, I/O survives a path loss, growth reaches the
  map (multipathd resize), a fence fails its I/O, an unfence restores it.
  8/8 for all 10 iSCSI vendors (WSL2: multipathd started directly, as
  systemd skips the unit in a container). Reads use aligned O_DIRECT as SPDK
  does (dd's O_DIRECT reads from dm fail there).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread simplyblock_core/array/san.py Fixed
michixs and others added 2 commits November 20, 2026 20:44
Problem: a LUN takeover opens the LUN's alceml cold on the standby
(pba_init_mode=1), which loads the whole page map. That is fine on
20 GiB but takes minutes at 100 TiB, far beyond the peers' I/O hold.

Design: the ultra data plane (PR #236) supports standby alceml
instances (standby=true, open-only) that keep their page map warm from
the LUN's region counters (refresh_interval_ms) and activate on their
first I/O: fence by reservation preemption, epoch bump, final refresh,
then serve.

- Opt-in per cluster: Cluster.multi_attach_warm_standby (immutable),
  --multi-attach-warm-standby on cluster create/add, v1/v2 APIs; needs
  multi_attach. multi_attach.warm_standby_enabled(cluster, dev) also
  requires the device's reservations (a standby fences by preemption).
- lun_takeover.ensure_warm_standbys(node), driven by the storage-node
  monitor (own thread per node, at most every 30 s, ONLINE nodes of
  warm clusters only): opens a standby alceml (open-only, the node's
  reservation key, the takeover's parameters) on every attached,
  non-serving, unfenced node of an ONLINE LUN, over the node's own path
  (agent lun_info + pick_lun_path). Never exported. Closes the standby
  (and, on a non-owner, its base bdev) when the node is no longer a
  standby; never an exported stack, an active instance or the serving
  node's. retire_stale_local_stack keeps a kept warm standby.
- Takeover: when the chosen standby holds a standby instance
  (alceml_get_role), it is reused: no alceml create and no activation
  wait before the export; PT/subsystem/listeners/ns are built, the
  record and paths switched as before, then the activation (driven by
  the peers' retried I/O) is awaited. If it does not activate: array
  fence + fenced record, export torn down, LUN reopened cold on the same
  standby (same NQN/IPs). If that fails too the device is set
  unavailable. A standby whose activation failed (act_failed) is closed
  and reopened active. Without a warm standby the cold path is unchanged.
- Memory: the page map counts in the node's SPDK system-memory budget
  (sys_memory_for_capacity, 0.7 GiB per raw TiB). A restart now sizes it
  over the node's own devices plus its warm-standby LUNs. At add-node the
  standby set is not known yet (documented); ensure_warm_standbys does
  not open a standby beyond the node's budget and raises a cluster event
  instead.
- rpc_client.bdev_alceml_create: refresh_interval_ms (standby only).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… partition

A node's journal on a multi-attach cluster whose array plugin creates the
nodes' LUNs is now placed by the array registration:

- journal "lun": node add creates a journal LUN next to the data LUN, with
  journal_options applied over the array's options (e.g. a pool of higher
  performance or priority) and journal_lun_size (default 3% of lun_size, at
  least 2 GiB, whole GiB). It is presented to that node only, never
  multi-attached, recorded as StorageNode.array_journal_lun_id, detached on
  node delete and deleted with the cluster. The host needs no local device:
  sn configure --lblk-array-only.
- journal "local" (default, unchanged for existing beds): one local disk or
  partition named with --lblk-journal-only, now a partition as well.

register_array / POST /clusters/{id}/arrays / cluster set-array gain
journal, journal_lun_size, journal_options; the array view reports them.
A journal partition on the data LUN is not offered: takeover and growth
would have to address a partition of a multi-attached LUN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…failure

grow-devices takes lun_size (API ?lun_size=, CLI --lun-size) besides
lun_factor: every array-created data LUN is grown to that many bytes, a LUN
already that large is left alone, so a growth that failed half-way is simply
repeated without growing a LUN twice (a factor would compound). Once every
LUN has the size, Cluster.array_lun_size becomes it, so nodes added later
match. This is what the operator drives from spec.multiAttach.array.lunSize.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration

register_array / POST /clusters/{id}/arrays take allow_local_kms_credentials
(true/false; omitted = unchanged): a cluster without an external KMS can be
told to keep the wrapped array login in its local key store with the
registration itself, so an operator-managed cluster never needs the CLI for
it. Applied to the store of that registration and saved only with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michixs and others added 3 commits November 20, 2026 21:15
… at add

2026-11-20 FTT2 bed, cases 7 (reboot sn2 + hard reset sn3) and 8 (hard reset
sn3 + network cut sn4): the takeover chose the other lost node as standby.
The standby failover worked, but the activation wait ran its full 5 s while
every poll failed to connect, then an array fence (1 s) and a second
reachability check (0.8 s) ran before the next standby: 13.1 s and 12.4 s
stalls against the 11 s limit.

- _wait_active: two consecutive connection errors raise StandbyUnreachable.
- A standby that is gone is left at once: no retry on it and no array fence
  first; the next standby opens with its own key and preempts by
  reservation (the array fence still follows in the background). A standby
  that answers but did not take the reservation is fenced at the array and
  retried as before.
- The export's cleanup reports an unreachable standby as StandbyUnreachable.

Warm standby: add-node sized the SPDK memory for the node's own LUN only, so
every warm standby was refused for budget until the node's first restart
(warm-standby bed). Add-node now sizes for FTT standby LUNs of the node's
own size (LUN sizes are uniform); multi_attach.warm_standby_cluster().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 warm-standby FTT2 bed (fb70786): a network cut of the serving
node made the takeover reuse the warm standby on sn1 (activation 141 ms);
fio crc32c verification then failed (EILSEQ) on 4 of 5 volumes. The same
case with a cold takeover on the same build verified clean. The SPDK log
of the activated instance was lost (restart), so the cause in the alceml
warm refresh/activation is not confirmed yet.

cluster create/add refuse --multi-attach-warm-standby unless the control
plane runs with SIMPLYBLOCK_ALLOW_WARM_STANDBY=1 (repro beds only). The
cold takeover, the default, is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 FTT2 bed (fb70786), net cut sn0 + SPDK kill sn1: the
reintegration unfenced the cut node at 21:32:37.754 and re-attached its
LUN; the owner (sn1, restarting) then wrote its full record five times in
1.4 s from a copy loaded before the unfence, restoring fenced_node_ids.
Record "fenced" + host showing the LUN = "operator unfences": the node
stayed fenced and the case never healed.

serving_node_id, fenced_node_ids and array_attached_node_ids of a
multi-attach device are changed only by atomic_update (takeover, fence,
unfence, attachment reconcile). A full-object StorageNode write now keeps
their stored values (BaseModel._write_tx -> StorageNode.merge_on_full_write)
and logs what it kept. Only nodes with multi-attach devices pay the
stored-record read; every other node write is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erts

- services/__main__.py: lun_watchdog runs as a thread inside
  storage_node_monitor and has no main(), so it is listed as a library
  (test_every_service_defines_main failed on CI for 'lun-watchdog').
- credentials.py: the store log line gives the number of fields, not their
  names (CodeQL clear-text logging of sensitive data).
- storage_arrays/testing: the mock array's TLS server refuses TLS < 1.2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
}

_CLUSTER_HEX = 8
_LUN_HEX = 16
if multi_attach.is_multi_attach(device_obj):
# Every node that exports the LUN must use the same namespace
# UUID: SPDK multipath joins paths by namespace identity.
ret = rpc_client.nvmf_subsystem_add_ns(
subsystem_nqn, pt_name, multi_attach.namespace_uuid(device_obj),
nguid=multi_attach.namespace_nguid(device_obj))
else:
ret = rpc_client.nvmf_subsystem_add_ns(subsystem_nqn, pt_name)
2026-11-21 FTT2 bed (7406ebd), network cuts of sn4 and sn0: two of the
three LUNs sn4 served moved within 1.5 s; the third chose its standby one
candidate after the other, and the first, sn0, had just lost its network.
Its agent query (15 s client timeout, default retries) held the choice
until the 90 s takeover deadline; the takeover failed, the LUN went
unavailable and the 1+0 cluster suspended.

- _choose_standby probes every eligible candidate in parallel; each agent
  query is bounded by STANDBY_PROBE_TIMEOUT_SEC (3 s) without retries, and
  so is the wait for them.
- Candidate order still wins; a candidate that reads the LUN waits at most
  STANDBY_ORDER_GRACE_SEC (0.5 s) for an earlier one that has not answered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 FTT2 bed (f1d109f), reboot sn2 + hard reset sn3: the takeover
took 1.3 s and the background EBS detach of the reset node succeeded, but
it landed while the host was booting. The NVMe controller reset failed and
the kernel kept a dead 0-byte namespace (lsblk STATE dead). The agent's LUN
inventory reported it, so the reintegration read "host still shows the
LUN, an operator unfences" and the node stayed fenced: the case never
healed.

find_lun_devices skips a disk whose controller state is "dead": it can
neither reach the array nor queue I/O for it. Live devices are reported as
before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
assert any(p.startswith(f"{prefix}/jobs/") for p in paths)
assert not any(p.startswith("/ConfigurationManager/v1/objects/ldevs") for p in paths)
finally:
client.close()
with pytest.raises(ArrayError):
client.get_info()
finally:
client.close()
client.create_volume(_uuid(), GIB)
assert 10 <= sum(sleeps) < 20
finally:
client.close()
with pytest.raises(ArrayError, match="No free LDEV"):
client.create_volume(_uuid(), GIB)
finally:
client.close()
g = next(g for g in fake.groups.values() if g["name"] == "sbtest-vm")
assert g["mode"] == "VMWARE_EX" and g["modeOptions"] == [54, 63]
finally:
client.close()
with pytest.raises(ArrayError, match="no configured fc port"):
client.ensure_host(HostSpec(name="sbtest-fc", initiators=(FC_A,)))
finally:
client.close()
… unavailable

2026-11-21 FTT2 bed (76b9e7e), hard reset of sn3: 1.7 s before its LUNs
were taken over, the event collector processed I/O-error events of nodes
reaching those LUNs through sn3, and sent them a per-node "unavailable"
(peer branch) and dropped their remote device. Nothing marked the devices
available again after the takeover; on 1+0 every read through such a map
fails. sn4's LVS failover then failed to read the superblock
(ParityNoneRead init_subtask -3), its SPDK aborted (core dumped), the two
LUNs it had just taken over moved again and the cluster suspended.

- main_distr_event_collector: an I/O-error event of a node that reaches a
  multi-attach LUN through another node's export (served_elsewhere) is left
  to the LUN watchdog/takeover: no per-node unavailable, no remote-device
  removal. The serving node's own errors and regular clusters unchanged.
- lun_takeover._switch_paths: once a node has its path to the new export,
  it is sent the device's ONLINE status (undoes anything sent in the gap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

target/main Targets the development branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants