Skip to content

feat(erasurecoding): device and failure-domain floors, raidDevices - #887

Open
schmidt-scaled wants to merge 12 commits into
mainfrom
feat/ec-floors
Open

schmidt-scaled wants to merge 12 commits into
mainfrom
feat/ec-floors

Conversation

@schmidt-scaled

Copy link
Copy Markdown
Contributor

Summary

Holds deployment documents to the erasure-coding resource floors in every topology and adds raidDevices. Design: https://claude.ai/artifact/ANm8UNf6MQZAGdmWpYS5Pb

  • StorageClusterSpec.RaidDevices / ClusterTemplate.RaidDevices (default false, immutable via CEL), sent to the control plane as raid_devices on cluster create.
  • Scheme: MinimumNodes unchanged (n+2k, Single/Dual overrides). New MinimumDataDevices(raid) (n+2k, or n+k with raidDevices), MinimumDataDevicesFor(topology, raid), MinimumFailureDomains() (n+k). MinimumDevicesPerNode is 1 in every topology (single-node keeps n+k).
  • StripeChecks (controller validation and approval webhook): device floor for every topology, including at least one data device per node; new reason StripeBelowMinimumFailureDomains. Dedicated journal devices (enableJournalDevice) are subtracted.
  • Regenerated CRDs, chart CRDs, deepcopy, install.yaml and the cpapi client. shared/openapi.json was edited by hand for raid_devices until the next sync from sbcli.
  • Docs: design-clusterdeploymentconfig.md, design-storagecluster.md, test plan rows.

Discovery is intentionally untouched; LUN discovery is covered by #884.

Tests

go test green for api, erasurecoding, deployment, cluster (envtest), webhook, discovery and atlas-lib controlplane/cpapi; go vet and gofmt clean; quality gate 9/9; CRD checker 0 errors. Run in a native LF clone (the Windows checkout panics in one unrelated discovery test).

Open decisions for review

  1. Two-node 1+1 with one plain device per node is refused (device floor 3); it passes with raidDevices.
  2. The existing FailureDomainLayout (k+2 equal domains) and the new n+k floor coexist; the new check stays silent when the old one already reports. Replace k+2 by n+k?
  3. Groups that list no devices escape the device floors until nodes report; the cluster activation gate still checks nodes only (sbcli activation is the backstop).

Related: sbcli feat/ec-floors, ultra feat/affinity-floor.

🤖 Generated with Claude Code

michixs and others added 8 commits November 9, 2026 14:56
A cluster whose data devices are RAID-protected below simplyblock says so on the create body as raid_devices, the name sbcli's cluster create takes for its --raid-devices flag; the control plane lowers its device floor for the stripe to ndcs+npcs. The field is added to shared/openapi.json beside is_single_node ahead of the next spec sync, and the client and the integration simulator are regenerated from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spec.raidDevices declares every data device protected below simplyblock (a hardware RAID LUN, md, or dm RAID), which lowers the erasure-coding data-device floor from ndcs+2*npcs to ndcs+npcs. It defaults to false and is immutable after creation (+k8s:immutable, exercised against envtest), is sent to the control plane as raid_devices at cluster create, and is copied from spec.cluster.raidDevices of the ClusterDeploymentConfig template as the other immutable layout fields are. CRDs, embedded upgrade CRDs, chart CRDs, deepcopy, and dist/install.yaml regenerated on Linux.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Beside MinimumNodes (ndcs+2*npcs, unchanged, with the single-node and two-node overrides as they were): MinimumDevicesPerNode is one in every topology and the stripe's width on a declared single-node cluster; MinimumDataDevices is ndcs+2*npcs, or ndcs+npcs when raidDevices is declared; MinimumDataDevicesFor keeps the single-node rule; MinimumFailureDomains is ndcs+npcs. Table tests over 1+0, 1+1, 2+1, 1+2, 2+2, 4+1, and 4+2, with and without raidDevices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oors

StripeChecks, which admission and the controller's validation both read, now checks every topology for what its nodes hand over and how they are spread, after the node and worker counts: every storage node needs at least one data device (StripeBelowMinimumDevices, naming the group or the node), the cluster needs MinimumDataDevices of them (raidDevices lowers it), and groups that declare failure domains need MinimumFailureDomains distinct ones (StripeBelowMinimumFailureDomains). A dedicated journal device is one per storage node and counts toward none of them. Devices are counted per worker from the groups that name them and from existing nodes' device names or reported count; an uncounted worker is held to neither device floor and holds the total back. The domain count is left to FailureDomainLayout where the control plane's own rule covers it, so one shortage is not reported under two names. A declared single-node cluster keeps its per-node device rule and is not held to the domain floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deployment-config and StorageCluster designs state the five floors a scheme sets and the raidDevices declaration; the event table gains StripeBelowMinimumDevices and StripeBelowMinimumFailureDomains; the test plans gain the new scenarios.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…istration

shared/openapi.json gains the multi-attach additions of sbcli #1626 only
(ClusterParams.multi_attach/ftt, ArrayParams/ArrayDTO and the
/clusters/{id}/arrays endpoints), spliced from an export of that branch; the
client and the integration simulator are regenerated from it.

atlas-lib: CreateClusterParams.MultiAttach/FaultTolerance, and
RegisterArray/ListArrays. The credentials travel only in the registration
request; the view read back carries the credential kind, never a value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…array

A multi-attach cluster runs on LUNs of an external array that the control
plane creates and presents to each node when it is added (sbcli #1626). The
operator now:

- StorageCluster.spec.multiAttach {faultTolerance, array}: plugin, endpoint,
  protocol, lunSize, caBundle, options, and credentials from a Secret, a
  Vault KV secret, or a KMIP secret (the latter two never pass through
  Kubernetes). Fixed at creation apart from the array block; requires
  raidDevices, deviceClass LogicalBlock, and a mode other than Single.
- creates the cluster with multi_attach and ftt, then registers the array
  once the cluster exists and again on a spec edit or a rotated login Secret;
  status.array reports the registration or the control plane's refusal.
- holds every node add (no slot taken) until the array is registered, since
  the add creates the node's LUN: reason AwaitingArray.
- writes LBLK_JOURNAL_ONLY for multi-attach nodes (--lblk-journal-only): the
  one local device a node names is its journal; the data device is the LUN.
- ClusterDeploymentConfig: cluster.multiAttach carried to the cluster; a
  multi-attach document is held to fault tolerance + 1 nodes on as many
  workers and exactly one named device per node (MultiAttachJournalDevice),
  the stated stripe being irrelevant (the cluster runs 1+0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The LUNs of a multi-attach cluster cannot be discovered: they are created on
the array when each node is added. A discovery run therefore takes the array
up front (spec.discover.multiAttach, with enableLogicalBlockDevices) and
drafts the document around it: the array goes into the draft's cluster,
raidDevices is set, every node keeps its smallest local device as the
journal, and a worker without an initiator for the array's protocol is a
finding (ArrayInitiatorMissing) before anybody approves.

The probe now reads each worker's iSCSI IQN, FC port names, and NVMe host
NQN (atlas-lib inventory.ReadSANInitiators, under the host root), carried in
the report as sanInitiators. Report version 12: a version-11 report has none
and would read as a worker the array cannot reach; the discovery fixtures are
regenerated for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@schmidt-scaled

Copy link
Copy Markdown
Contributor Author

Added the operator side of the multi-attach array LUN mode (sbcli #1626, which now also carries the SAN array plugins of #1701). Three commits, b743ef9..aec1f7c:

Control-plane client

  • shared/openapi.json: only the #1626 additions were spliced in (ClusterParams multi_attach/ftt, ArrayParams/ArrayDTO, /clusters/{id}/arrays); the client and cpsim are regenerated.
  • atlas-lib: CreateClusterParams.MultiAttach/FaultTolerance, RegisterArray, ListArrays.

StorageCluster / StorageNode

  • spec.multiAttach {faultTolerance, array}: the array block has plugin, endpoint, protocol, lunSize, caBundle and options. Its credentials.source is one of:
    • Secret;
    • VaultKV or KMIPSecret: the login never passes through Kubernetes;
    • None, for aws-ebs.
  • CEL rules: multiAttach requires raidDevices, LogicalBlock, and a mode other than Single. It and its faultTolerance are fixed at creation; the array block may change.
  • Creation sends multi_attach and ftt. The array is registered once the cluster exists, and again on a spec edit or a rotated login Secret; status.array reports the result.
  • A node add waits for the registration without taking a slot (reason AwaitingArray), since the add creates the node's LUN.
  • Multi-attach nodes get --lblk-journal-only: the one local device a node names is its journal, and its data device is the LUN.

ClusterDeploymentConfig and discovery (the order problem: LUNs can't be discovered before they exist)

  • cluster.multiAttach is carried to the cluster.
  • Floors for a multi-attach document: faultTolerance+1 nodes on as many workers, and exactly one named device per node (MultiAttachJournalDevice). The stated stripe is ignored, since the cluster runs 1+0.
  • spec.discover.multiAttach: the array, with its protocol and credential reference, is stated before discovery. The run then:
    • drafts the document around it, keeping each node's smallest disk as the journal;
    • raises ArrayInitiatorMissing for any worker without an initiator for the array's protocol.
  • The probe now reports each worker's iSCSI IQN, FC WWPNs and NVMe host NQN. Report version 12; the fixtures are regenerated.

Checks

  • atlas-lib: tests green. Operator: the touched packages pass (csilink TestCertReloader fails, as it does on the base).
  • check-crds.py, the house-style gate, verify-config-yaml, verify-crd-kustomization, build-installer and helm-sync are all clean.
  • golangci-lint was not run (the local binary predates Go 1.27).

Not done

  • No pre-approval reachability check of the array itself: there is no cluster-less validate endpoint in the control plane yet.
  • No bed run.

🤖 Generated with Claude Code

michixs and others added 4 commits November 20, 2026 20:10
… 3 s)

On a multi-attach cluster (1+0, shared array LUNs) a node lost without
closing its connections stalls its volumes' I/O until the host's keep-alive
notices; with the default that took 10-14 s on the bed, past the 11 s stall
limit. sbcli (#1626) now returns keep-alive-tmo 3 on such a cluster's connect
entries, but the driver passed the control plane's keep-alive only for
two-node clusters, so Kubernetes hosts kept the default.

The control plane's per-path keep-alive is now authoritative whenever it is
shorter than a regular cluster's 8 s: connectArgs (stage via the initiator and
the reconnect loop) adds -k, and ConnectionFrom (the volume stack: stage,
striped members, pNFS export) sets KeepAliveTMOSec. The driver needs no
multi-attach rule of its own and no new API field.

Unchanged: a regular cluster (keep-alive 8, or none from an older control
plane) connects with the same command line; two-node paths keep their
existing handling on both paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l device

StorageCluster.spec.multiAttach.array.journal {placement LUN|Local, size,
options} (sbcli #1626 journal placement):

- LUN (the default): the control plane creates each node's journal as a LUN
  of its own next to its data LUN, in the array's options updated by
  journal.options (e.g. a pool of higher performance or priority) and of
  journal.size (default 3% of lunSize, at least 2 GiB). Nodes name no local
  device: the per-node config writes LBLK_ARRAY_ONLY (--lblk-array-only) and
  no BLK_NAMES; a deployment document naming devices is refused
  (MultiAttachJournalDevice); discovery declines every local device
  (ArrayOnlyRule), keeps device-less workers, and drafts groups naming none.
- Local: unchanged — one local disk or partition per node, whole
  (LBLK_JOURNAL_ONLY).
- CEL: size and options only with the placement LUN. Registration sends
  journal / journal_lun_size / journal_options; shared/openapi.json and the
  generated client carry them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… opt-in

Everything a Unified DataStore needs is now declared on the StorageCluster
(sbcli #1626 f2a518c2c, 811da7162):

- Raising spec.multiAttach.array.lunSize grows every LUN to it at the array
  and the cluster's devices onto it, online (control plane grow-devices
  ?lun_size=). status.array.lunSize is the size the LUNs have; a failed
  growth is reported (message, ArrayLUNGrowthFailed) and repeated, and a LUN
  that already has the size is left alone, so nothing grows twice. Lowering
  lunSize is refused in status.array.message. Event ArrayLUNsGrown.
- spec.multiAttach.array.credentials.allowLocalKeyStore: the opt-in to
  keeping a Secret's login in the local key store of a cluster without an
  external KMS (test environments), sent with the registration (CEL: Secret
  source only). It replaces the CLI `cluster set ... allow_local_kms_credentials`.
- shared/openapi.json: grow-devices takes lun_factor / lun_size and returns
  the current DeviceGrowthDTO (grown, factor, devices, errors, reason,
  shrunk, luns); ArrayParams.allow_local_kms_credentials.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts:
- atlas-lib/controlplane/clusters.go: main's MdJournal and KmsSettings (which
  replaces HashicorpVaultSettings), plus this branch's RaidDevices.
- operatorops_types.go: both validation rules (multiAttach drafts; main's
  spdkOnHostNetwork vs networkAttachment).
- cel_validation_test.go: both tests (raidDevices immutable; main's
  enableMetadataJournal default).
- test-plan-clusterdeploymentconfig.md: this branch's rows U-242..U-247 and
  main's U-248.
- dist/install.yaml and internal/upgrade/crds/manifests: deleted on main, kept
  deleted.
- Generated files regenerated: deepcopy, CRDs, Helm chart (make helm-sync),
  cpapi, cpsim.gen.go, unimplemented.gen.go; discovery fixtures net-76..78
  re-recorded at report version 12 (go run ./hack/discoveryfixtures).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants