Repository navigation
feat(erasurecoding): device and failure-domain floors, raidDevices - #887
Open
schmidt-scaled wants to merge 12 commits into
Open
schmidt-scaled wants to merge 12 commits into
schmidt-scaled wants to merge 12 commits into
Conversation
A cluster whose data devices are RAID-protected below simplyblock says so on the create body as raid_devices, the name sbcli's cluster create takes for its --raid-devices flag; the control plane lowers its device floor for the stripe to ndcs+npcs. The field is added to shared/openapi.json beside is_single_node ahead of the next spec sync, and the client and the integration simulator are regenerated from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spec.raidDevices declares every data device protected below simplyblock (a hardware RAID LUN, md, or dm RAID), which lowers the erasure-coding data-device floor from ndcs+2*npcs to ndcs+npcs. It defaults to false and is immutable after creation (+k8s:immutable, exercised against envtest), is sent to the control plane as raid_devices at cluster create, and is copied from spec.cluster.raidDevices of the ClusterDeploymentConfig template as the other immutable layout fields are. CRDs, embedded upgrade CRDs, chart CRDs, deepcopy, and dist/install.yaml regenerated on Linux. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Beside MinimumNodes (ndcs+2*npcs, unchanged, with the single-node and two-node overrides as they were): MinimumDevicesPerNode is one in every topology and the stripe's width on a declared single-node cluster; MinimumDataDevices is ndcs+2*npcs, or ndcs+npcs when raidDevices is declared; MinimumDataDevicesFor keeps the single-node rule; MinimumFailureDomains is ndcs+npcs. Table tests over 1+0, 1+1, 2+1, 1+2, 2+2, 4+1, and 4+2, with and without raidDevices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oors StripeChecks, which admission and the controller's validation both read, now checks every topology for what its nodes hand over and how they are spread, after the node and worker counts: every storage node needs at least one data device (StripeBelowMinimumDevices, naming the group or the node), the cluster needs MinimumDataDevices of them (raidDevices lowers it), and groups that declare failure domains need MinimumFailureDomains distinct ones (StripeBelowMinimumFailureDomains). A dedicated journal device is one per storage node and counts toward none of them. Devices are counted per worker from the groups that name them and from existing nodes' device names or reported count; an uncounted worker is held to neither device floor and holds the total back. The domain count is left to FailureDomainLayout where the control plane's own rule covers it, so one shortage is not reported under two names. A declared single-node cluster keeps its per-node device rule and is not held to the domain floor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deployment-config and StorageCluster designs state the five floors a scheme sets and the raidDevices declaration; the event table gains StripeBelowMinimumDevices and StripeBelowMinimumFailureDomains; the test plans gain the new scenarios. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…istration
shared/openapi.json gains the multi-attach additions of sbcli #1626 only
(ClusterParams.multi_attach/ftt, ArrayParams/ArrayDTO and the
/clusters/{id}/arrays endpoints), spliced from an export of that branch; the
client and the integration simulator are regenerated from it.
atlas-lib: CreateClusterParams.MultiAttach/FaultTolerance, and
RegisterArray/ListArrays. The credentials travel only in the registration
request; the view read back carries the credential kind, never a value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…array
A multi-attach cluster runs on LUNs of an external array that the control
plane creates and presents to each node when it is added (sbcli #1626). The
operator now:
- StorageCluster.spec.multiAttach {faultTolerance, array}: plugin, endpoint,
protocol, lunSize, caBundle, options, and credentials from a Secret, a
Vault KV secret, or a KMIP secret (the latter two never pass through
Kubernetes). Fixed at creation apart from the array block; requires
raidDevices, deviceClass LogicalBlock, and a mode other than Single.
- creates the cluster with multi_attach and ftt, then registers the array
once the cluster exists and again on a spec edit or a rotated login Secret;
status.array reports the registration or the control plane's refusal.
- holds every node add (no slot taken) until the array is registered, since
the add creates the node's LUN: reason AwaitingArray.
- writes LBLK_JOURNAL_ONLY for multi-attach nodes (--lblk-journal-only): the
one local device a node names is its journal; the data device is the LUN.
- ClusterDeploymentConfig: cluster.multiAttach carried to the cluster; a
multi-attach document is held to fault tolerance + 1 nodes on as many
workers and exactly one named device per node (MultiAttachJournalDevice),
the stated stripe being irrelevant (the cluster runs 1+0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The LUNs of a multi-attach cluster cannot be discovered: they are created on the array when each node is added. A discovery run therefore takes the array up front (spec.discover.multiAttach, with enableLogicalBlockDevices) and drafts the document around it: the array goes into the draft's cluster, raidDevices is set, every node keeps its smallest local device as the journal, and a worker without an initiator for the array's protocol is a finding (ArrayInitiatorMissing) before anybody approves. The probe now reads each worker's iSCSI IQN, FC port names, and NVMe host NQN (atlas-lib inventory.ReadSANInitiators, under the host root), carried in the report as sanInitiators. Report version 12: a version-11 report has none and would read as a worker the array cannot reach; the discovery fixtures are regenerated for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Added the operator side of the multi-attach array LUN mode (sbcli #1626, which now also carries the SAN array plugins of #1701). Three commits, b743ef9..aec1f7c: Control-plane client
StorageCluster / StorageNode
ClusterDeploymentConfig and discovery (the order problem: LUNs can't be discovered before they exist)
Checks
Not done
🤖 Generated with Claude Code |
… 3 s) On a multi-attach cluster (1+0, shared array LUNs) a node lost without closing its connections stalls its volumes' I/O until the host's keep-alive notices; with the default that took 10-14 s on the bed, past the 11 s stall limit. sbcli (#1626) now returns keep-alive-tmo 3 on such a cluster's connect entries, but the driver passed the control plane's keep-alive only for two-node clusters, so Kubernetes hosts kept the default. The control plane's per-path keep-alive is now authoritative whenever it is shorter than a regular cluster's 8 s: connectArgs (stage via the initiator and the reconnect loop) adds -k, and ConnectionFrom (the volume stack: stage, striped members, pNFS export) sets KeepAliveTMOSec. The driver needs no multi-attach rule of its own and no new API field. Unchanged: a regular cluster (keep-alive 8, or none from an older control plane) connects with the same command line; two-node paths keep their existing handling on both paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l device
StorageCluster.spec.multiAttach.array.journal {placement LUN|Local, size,
options} (sbcli #1626 journal placement):
- LUN (the default): the control plane creates each node's journal as a LUN
of its own next to its data LUN, in the array's options updated by
journal.options (e.g. a pool of higher performance or priority) and of
journal.size (default 3% of lunSize, at least 2 GiB). Nodes name no local
device: the per-node config writes LBLK_ARRAY_ONLY (--lblk-array-only) and
no BLK_NAMES; a deployment document naming devices is refused
(MultiAttachJournalDevice); discovery declines every local device
(ArrayOnlyRule), keeps device-less workers, and drafts groups naming none.
- Local: unchanged — one local disk or partition per node, whole
(LBLK_JOURNAL_ONLY).
- CEL: size and options only with the placement LUN. Registration sends
journal / journal_lun_size / journal_options; shared/openapi.json and the
generated client carry them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… opt-in Everything a Unified DataStore needs is now declared on the StorageCluster (sbcli #1626 f2a518c2c, 811da7162): - Raising spec.multiAttach.array.lunSize grows every LUN to it at the array and the cluster's devices onto it, online (control plane grow-devices ?lun_size=). status.array.lunSize is the size the LUNs have; a failed growth is reported (message, ArrayLUNGrowthFailed) and repeated, and a LUN that already has the size is left alone, so nothing grows twice. Lowering lunSize is refused in status.array.message. Event ArrayLUNsGrown. - spec.multiAttach.array.credentials.allowLocalKeyStore: the opt-in to keeping a Secret's login in the local key store of a cluster without an external KMS (test environments), sent with the registration (CEL: Secret source only). It replaces the CLI `cluster set ... allow_local_kms_credentials`. - shared/openapi.json: grow-devices takes lun_factor / lun_size and returns the current DeviceGrowthDTO (grown, factor, devices, errors, reason, shrunk, luns); ArrayParams.allow_local_kms_credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts: - atlas-lib/controlplane/clusters.go: main's MdJournal and KmsSettings (which replaces HashicorpVaultSettings), plus this branch's RaidDevices. - operatorops_types.go: both validation rules (multiAttach drafts; main's spdkOnHostNetwork vs networkAttachment). - cel_validation_test.go: both tests (raidDevices immutable; main's enableMetadataJournal default). - test-plan-clusterdeploymentconfig.md: this branch's rows U-242..U-247 and main's U-248. - dist/install.yaml and internal/upgrade/crds/manifests: deleted on main, kept deleted. - Generated files regenerated: deepcopy, CRDs, Helm chart (make helm-sync), cpapi, cpsim.gen.go, unimplemented.gen.go; discovery fixtures net-76..78 re-recorded at report version 12 (go run ./hack/discoveryfixtures). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Holds deployment documents to the erasure-coding resource floors in every topology and adds
raidDevices. Design: https://claude.ai/artifact/ANm8UNf6MQZAGdmWpYS5PbStorageClusterSpec.RaidDevices/ClusterTemplate.RaidDevices(default false, immutable via CEL), sent to the control plane asraid_deviceson cluster create.Scheme:MinimumNodesunchanged (n+2k, Single/Dual overrides). NewMinimumDataDevices(raid)(n+2k, or n+k with raidDevices),MinimumDataDevicesFor(topology, raid),MinimumFailureDomains()(n+k).MinimumDevicesPerNodeis 1 in every topology (single-node keeps n+k).StripeChecks(controller validation and approval webhook): device floor for every topology, including at least one data device per node; new reasonStripeBelowMinimumFailureDomains. Dedicated journal devices (enableJournalDevice) are subtracted.shared/openapi.jsonwas edited by hand forraid_devicesuntil the next sync from sbcli.Discovery is intentionally untouched; LUN discovery is covered by #884.
Tests
go testgreen for api, erasurecoding, deployment, cluster (envtest), webhook, discovery and atlas-lib controlplane/cpapi;go vetand gofmt clean; quality gate 9/9; CRD checker 0 errors. Run in a native LF clone (the Windows checkout panics in one unrelated discovery test).Open decisions for review
FailureDomainLayout(k+2 equal domains) and the new n+k floor coexist; the new check stays silent when the old one already reports. Replace k+2 by n+k?Related: sbcli feat/ec-floors, ultra feat/affinity-floor.
🤖 Generated with Claude Code