Repository navigation
feat: select Prometheus with ControlPlane observability.prometheus.mode - #953
Draft
boddumanohar wants to merge 27 commits into
Draft
boddumanohar wants to merge 27 commits into
boddumanohar wants to merge 27 commits into
Conversation
…de agent (#734) * feat(storage-node): state Talos to the node agent, with a talosctl image slot spec.environment: Talos resolved only into the kubelet flag; the node agent itself was never told the host is Talos and kept discovering it by sniffing the kernel version, after which it skipped the huge-page reservation and left the SPDK pod Pending on a short pool. sbcli #1491 makes a stated Talos node reserve its pages through the Talos API (a talosctl Job patching machine.sysfs and restarting the kubelet); this is the operator half. * StorageNodesSpec.talos (*bool), set by the expansion from the Talos environment the way OpenShift is, reaches the agent as TALOS_CLUSTER beside OPENSHIFT_CLUSTER / UBUNTU_HOST. * spec.images.talosctl on the deployment config lands on StorageNodesSpec.talosctlImage and reaches the agent as TALOSCTL_IMAGE when stated, so a deployment pins the Job image to its Talos minor (or a mirror) instead of taking the agent's default. Generated CRDs follow in the next commit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore: regenerate CRDs, deepcopy and installer for the Talos fields make manifests generate build-installer + helm-charts/scripts/sync-from-operator.sh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * style: name the true/false strings the agent environment is built from (goconst) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(api): the talosctl slot admits any registry ImageSpec holds every image to simplyblock's own registries, which is right for software simplyblock publishes and wrong for talosctl: Sidero publishes it on ghcr.io and a mirroring site names its own registry. Found on the first Talos deployment, where the draft rejected ghcr.io/siderolabs/talosctl:v1.12.7 against ImageSpec's pattern. ExternalImageSpec keeps the reference-shape check and drops the registry allow-list; the talosctl slot uses it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore: regenerate deepcopy, CRDs and installer for ExternalImageSpec Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(rbac): let the storage-node agent read Job pod logs and check for a Secret Two read-only rules on simplyblock-storage-node-role, both wanted by the Talos huge-page flow and seen as HTTP 403 in the agent log on the first Talos run: * pods/log get: the agent copies a finished talosctl Job's pod log into its own log before deleting the Job, the only record of what happened on the host side. * secrets get: the pre-flight check that the Talos API credentials Secret exists before a Job is created to mount it; without it a missing Secret is a Job pod stuck in ContainerCreating for the whole Job timeout. The operator's own role already holds both, so it can grant them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * style: gofmt and an unshadowed loop variable (lint) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(talos): enable the kubelet configuration on Talos and let the agent read configz With sbcli configuring the kubelet's CPU topology on Talos through machine.kubelet.extraConfig (the Talos API), the Talos environment no longer resolves to "skip the kubelet configuration": it is enabled as on every other distribution, and the node chooses the Talos path by itself. The storage-node ClusterRole gains nodes/proxy get: the agent reads the kubelet's live configuration (nodes/<name>/proxy/configz) before patching, so a kubelet that already runs with the settings is left alone -- Talos restarts the kubelet on every machine-config change, and a restart per SPDK restart would be a regression. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(talos): state who writes the machine config, pre-provisioned unless Managed A Talos deployment so far implied that the node may patch a worker's machine config through the Talos API, which made an os:admin Secret in the namespace a requirement of every Talos cluster and left an operator who prepares the machine config themselves with the wrong error on a short pool. The question is now stated: * ClusterTemplate.talosMachineConfig on the deployment config, valid only with environment Talos (CEL), lands on StorageNodesSpec.talosMachineConfig: Preprovisioned (the default, left unstated when the document is) or Managed. * The agent reads it as TALOS_MACHINE_CONFIG, lowercase, set whenever Talos is stated so the DaemonSet shows what the node will do: with preprovisioned it verifies the pool and the kubelet settings and fails the add with the patch to apply; with managed it runs the talosctl Jobs (sbcli #1491). Generated CRDs, chart and installer follow in the next commit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore: regenerate CRDs, chart and installer for talosMachineConfig controller-gen, the conversion-webhook stanza, the embedded CRD copies, the installer and the chart's CRDs, from the types as they are after the rebase: the talosMachineConfig field on the cluster template and on storageNodes with its enum and the environment rule, and the talosctl image slot in its ExternalImageSpec form. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(talos): a Talos deployment turns the CPU topology on, as OpenShift does The Talos case enabled the kubelet configuration but left the CPU topology unset, i.e. off, and a Talos document cannot state it (the environment decides both). The kubelet patch through the Talos API is what that configuration is for, so the topology is on as on OpenShift. In the pre-provisioned mode a kubelet without the settings refuses the add with the extraConfig patch to apply; in managed mode the node applies it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore: regenerate CRDs, installer and chart for the stated kubelet and CPU topology flags on Talos The flag descriptions said Talos leaves both flags off; a Talos deployment turns both on (the kubelet half through the Talos API), and a document may now state either one off. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(support-tool): a document states the Talos machine-config mode only for a Talos fleet The support tool's admission check carries the same rule as the CRD: spec.cluster.talosMachineConfig is refused unless spec.environment is Talos, and accepted there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Live consistency-group subsystem flip on late label join * fix(cg-join): prepare each host with its own host NQN * fix(swap): drop the source's dead paths before resuming on the target * feat(cg): move a volume between consistency groups on relabel * ran make operator-build-installer * fix(operator): fresh step message and honest colocation count * test(operator): drop the ineffectual ops assignment in the step-message test * fix(volstack): grow the dmLinear mapping when a volume is expanded * feat(driver): add spec.disableDeviceMapperIndirection to SimplyblockDriver * feat(csi-node): turn the dm-linear indirection off on hosts without device-mapper * fix(cg-flip): hold every consumer of a moving namespace, and nothing else touches it * fix(cg-flip): a move that happened is committed, never rolled back after the window * fix(cg-watcher): a member outside its group's subsystem gets the join again * updated design doc design-cg-live-subsystem-flip.md * fix(cg-join): wait out a held plan in Migrating, and plan again before the child * fix(csi): clear the lint the main merge introduced * fix(driver): close the test the main merge left open
…t OpenShift's (#919) * fix(numa): the NUMA device plugin registers with a kubelet that is not OpenShift's The plugin registers by connecting to the kubelet's socket, and SELinux decides that by the type the kubelet runs as. The manifest's container_device_plugin_t may connect to OpenShift's kubelet (kubelet_t). RKE2 runs its kubelet as container_runtime_t, which that type may not connect to: avc: denied { connectto } comm="numa-resource-p" path="/var/lib/kubelet/device-plugins/kubelet.sock" scontext=...:container_device_plugin_t tcontext=...:container_runtime_t The plugin logged "failed to connect to kubelet: context deadline exceeded", exited after 45 s and crash-looped; the node advertised no numa-align, so no storage pod with CPU topology was scheduled there (RKE2 1.36 on Rocky 9.5, SELinux enforcing, 2026-11-13). Off OpenShift the plugin now runs as spc_t. OpenShift keeps container_device_plugin_t. Which one the cluster is comes from the API groups it serves, read once at start; a cluster that cannot be asked stops the operator rather than being guessed at. The plugin stays unprivileged, with every capability dropped and a read-only root filesystem. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * fix(csi-chart): the chart's NUMA device plugin takes the same SELinux type as the operator's The spdk-csi chart carries its own copy of the plugin's DaemonSet with the fixed container_device_plugin_t, which cannot register with a kubelet that does not run as OpenShift's does. It now renders container_device_plugin_t on a cluster that serves config.openshift.io/v1 ClusterVersion, the test validate-tls.yaml already uses for OpenShift, and spc_t elsewhere. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * fix(csi-chart): node-rbac.yaml renders one document separator between its objects The template ended its ServiceAccount block with a separator and began the next block with another, and the whitespace trimming around the two conditions joined them into one line of six dashes. helm lint failed the chart on it ("invalid Yaml document separator"). The blocks now carry one separator each and keep their line breaks. The chart renders the same 29 objects as before, with either condition off as well. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * fix(chart): the control plane's node selector value is rendered as a string controlplane.nodeSelector.value was rendered bare in the operator Deployment, the three observability Deployments and the log collector's affinity, so a value that YAML reads as something else than a string became one: with the label node-role.kubernetes.io/control-plane=true, which k3s sets on its servers, helm upgrade failed on the operator Deployment with "cannot unmarshal bool into Go struct field PodSpec.spec.template.spec.nodeSelector of type string". The ControlPlane the chart bootstraps already quoted it, so that half of the upgrade had gone through and the release was left failed with the selector applied to the control plane and not to the operator. The value is quoted in all five places. A chart that sets no selector renders as before. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * fix(inventory): an RKE2 cluster is concluded to be Rancher's, not K3s RKE2 is built on the K3s code and serves the k3s.cattle.io API group, for its add-ons and its etcd snapshot records. The group was read as K3s outright, and K3s precedes Rancher when a cluster carries both, so the discovery wrote `environment: K3s` for an RKE2 cluster (RKE2 v1.36.6+rke2r1, 2026-11-13). The precedence exists for a Rancher-managed K3s cluster, and its rule is that the distribution that installed the kubelet decides. That rule is now applied to the pair: where the kubelets carry the Rancher family's build suffix and none carries K3s's, a K3s conclusion becomes Rancher. A cluster with a K3s kubelet, with kubelets of both, or with no kubelet version to read keeps the answer it had. The K3s finding stays in the evidence. The two environments expand to the same storage node settings today, so no deployment changes; the draft now names the distribution the cluster is. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
…ers (#932) * fix(ci): run the CSI unit and integration suites on Ubuntu 26.04 runners ubuntu-latest moved to 26.04 between two runs on main (7983a98 passed, 1c4fb97 failed), which brings two changes the tests assumed away. - coreutils comes from uutils, where sleep is a hard link to one multicall binary that picks the utility from its own name. The nfsexport tests copied sleep as rpc.idmapd and nfsdcld to fake a running daemon. The copy refused to run and exited, so the "already running" tests lost a race on every run. The fake is now a copy of the test binary, which sleeps when NFSEXPORT_FAKE_PROCESS is set. - sudo is sudo-rs, which ignores -E and warns on stderr. The integration harness ran talosctl under sudo -E for HOME and KUBECONFIG. talosctl ran with root's HOME, and the warning landed in the JSON the harness parses, so no suite brought its cluster up. Both variables are now passed by name through env, in the harness and in the workflow's diagnostics and cleanup steps. Verified on an ubuntu:26.04 container: TestAFakeProcessKeepsRunning is red before and green after, and the nfsexport package passes three runs under -race -cover. The sudo change is unit-tested on the arguments only; the integration job on this PR is its live check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ci): set talosctl's HOME and KUBECONFIG as sudo arguments, not through env Running talosctl through `sudo env` changed the command sudo authorizes from talosctl to env, which the scoped SETENV grant in the integration README does not cover. An unattended developer host would have been prompted, while the GitHub runner's NOPASSWD: ALL hid it. The variables are now sudo's own VAR=value arguments, which that grant covers. Checked on ubuntu:26.04 with sudo-rs 0.2.13: VAR=value works under both NOPASSWD: ALL and a scoped SETENV grant, and the scoped grant refuses the env form. README and workflow steps updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Exported from simplyblock/sbcli@9699508. Co-authored-by: noctarius <1142801+noctarius@users.noreply.github.com>
…esystem (#936) * fix(devmapper): re-point a dm-linear mapping without freezing the filesystem The volstack heal swaps a volume's dm-linear table onto the namespace the fabric reconnected after total path loss. Swap suspended with dmsetup's default lockfs, which freezes the filesystem on top, and that freeze has to write to the namespace that is gone: ext4's freeze failed with EIO, so NodePublish never restaged the volume, and XFS's blocked in xfs_ail_push_all_sync while holding the superblock lock, so every sync(2) on the node hung behind it. Swap now suspends with --nolockfs, as the live subsystem flip already does. In-flight I/O still drains before the reload. Seen in the lblk e2e run 44463031746: both RECONNECT-FULLLOSS specs, both RECONNECT-GUARDIAN specs, and the PNFS two-writer spec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(e2e): read the filesystem type with df -P so a long device name stays on one line busybox df puts a device name longer than 20 characters on a line of its own, so `awk 'NR==2 {print $2}'` read an empty field once volumes were staged behind /dev/mapper/sb-<uuid>. -P keeps each mount on one line, as the size check in utils.go already does. Seen in the lblk e2e run 44463031746: SPDKCSI-FILESYSTEM XFS and ext4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…937) * ci(security): accept dotted prerelease tags such as v26.4.0-beta6.1 The 26.4 line now resolves to v26.4.0-beta6.1, which the release-tag shape check rejected, failing the scheduled scan on main. * build(operator): bump go.opentelemetry.io/otel to v1.47.0 (GHSA-q7gw-24fr-5p69)
… the unwind (SFAM-2824) (#933) discardMigration returned on any CancelMigration error. A failed migration is what fails the operation, and the control plane answers its cancel with a 400 "is not active", so the unwind stopped on the ordinary failure path: the validation Jobs stayed, no release reached the hosts that connected the target paths, and status.migration kept every connection. An abort, a withdrawal, or an abandonment hit the same refusal on every pass and never finished, holding the volume lock. The refusal wording now lives in the atlas-lib client rather than in the operator. CancelMigration names the "not active" answer of both a volume's migration and a subsystem's group ErrMigrationNotActive; CreateMigration names ErrMigrationBlocked (a 409, or one of the three self-clearing 400 wordings release 26.3 still sends, since only main and the 26.4 betas map them to 409) and ErrMigrationAlreadyOnTarget. discardMigration treats ErrMigrationNotActive and ErrNotFound as taken back and goes on to release; refusedCreate reads the sentinels instead of the body. The operator tests build their control-plane errors through the real client against an httptest server, so they exercise the classification. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#938) * build(operator): embed the CRDs from config/crd/bases instead of a copy The upgrade tool carried a committed copy of every generated CRD under internal/upgrade/crds/manifests, because go:embed cannot reach outside the directory of the file that declares it. Every CRD change therefore showed up twice in a PR in the operator module alone. A small package in config/crd/bases now declares the embed next to the generated files, and internal/upgrade/crds imports it. The crd-embed target, the copy, and the copy's drift check are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(operator): attach dist/install.yaml to releases instead of committing it The consolidated installer repeated every CRD and RBAC change in each PR. Its committed copy also pinned the placeholder image tag 0.1.0, so the raw.githubusercontent.com URL the README pointed at never installed a real release. dist/ is now ignored. The release workflow renders the installer against the release's operator image and uploads it as a release asset, and the README links to that asset. CI still renders and validates it with kubeconform on every PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: collapse generated copies in pull request diffs Marks the chart's synced CRDs, roles, and webhook, the support tool's chart copy, the zz_generated files, and the generated control-plane client as linguist-generated. GitHub collapses them in a PR diff, so a CRD change shows its schema once, in operator/config/crd/bases, which stays expanded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: collapse the generated discovery fixtures in pull request diffs The probe reports, nodes, and OperatorOps under testdata/discovery are written by hack/discoveryfixtures, and a probe-report schema change rewrites all of them (#661 touched 898 files). The builders are what a reviewer reads. The expected files stay expanded, because they are the behavior under review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(operator): summarize discovery expectation changes on pull requests A change to one default rewrites the same line in hundreds of expected files: #805 changed 396 of them, past the 300 files GitHub renders. The new script groups identical added and removed lines, so #805 reads as about 25 rows, with its single enableMetadataJournal: false still on a row of its own. The workflow writes the summary to the job summary and keeps one PR comment updated in place. Fork PRs get the job summary only, since their token cannot comment. It never fails a build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ump (#939) #937 raised go.opentelemetry.io/otel to v1.47.0 and go-logr/logr to v1.4.4 in operator/go.mod. fleet, gateway, and support-tool replace the operator module with ../operator, so the bump raised their minimum versions too, but their go.mod files were not updated. The Docker builds run outside the workspace and failed with "updates to go.mod needed". Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…rt one that stays there (SFAM-2850) (#931) * fix(pnfs): probe a mount whose data goes through the MDS, and report one that stays there (SFAM-2850) A fence from the device brings no reconnect. The client marks the device unusable, and every READ and WRITE of the mount goes through the metadata server. The node plugin probed a staging mount only when first seen, after a reconnect, or after a failed probe, so nothing took the layout again from a namespace that can see the device. In SFAM-2850 the mount also sent all of its I/O through the MDS for the rest of the run, and the only trace was a warning per probe in the node plugin's log. The primer now also probes a primed mount whose READs and WRITEs through the server grew since the last pass, at most every 30 seconds, since some I/O, such as unaligned direct I/O, goes through the server by design. A mount whose probes have failed for 5 minutes is logged as an error and reported once per run of failures with a PNFSDataThroughMDS Warning event on the volume's claim, or on its PersistentVolume when it has no claim. The node plugin's role already allows creating events and listing PersistentVolumes. Together with vela-os patch 0026, which holds a client's layouts again after a fence or a probe through the server, the probe makes the first device lookup once the device is usable again. Tests were written first and failed on the unchanged primer: no probe while data went through the server, no report after 5 minutes, and no event. Test plan rows U-129 to U-133 and F-21. F-21, a failover on a live cluster, is not run yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(pnfs): the MDS guest names the clients whose data goes through it (SFAM-2850) In SFAM-2850 one client sent 6.8 GB through the metadata server in 11 minutes, for the rest of a run, and nothing on the server said which client it was. vela-os patch 0027 has the guest's nfsd count, per client, the bytes of the READs and WRITEs it carries outside the grace period and the layout hold, in /proc/fs/nfsd/clients/<id>/info. The guest agent reads those counts every minute (--mds-data-interval, 0 turns it off) and logs a warning naming each client, by its NFS name and address, that sent 1 MiB or more since the last look. The log reaches the MDS pod log the way the nfsd watchdog's does. The first look names nobody, a client first seen later is counted from zero, and a guest kernel without the counts names nobody. Tests were written first and failed on a watch that did nothing. Test plan rows U-134 and U-135. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(csi): let a manual run build the MDS image from another pnfs-os guest [skip ci] CI built the MDS image only from the shared pnfsos tag, which only main and pnfs-minimal move in vela-os. A guest built from any other vela-os branch publishes only commit-pinned pnfsos-<sha> tags, so testing one meant a local multi-architecture build. A manual run now takes the guest image as guest_image, defaulting to pnfsos, so push and release builds are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnfs): report I/O through the MDS by what is seen, back off failed probes, report a hung probe F-21 on dev-ocp found two defects in the report this branch added. A probe counts as failed when the mount's WRITE count grows under it, which a pod's own writes through the metadata server cause too. A pod writing unaligned O_DIRECT therefore failed every probe: 201 probes in 400 seconds, each with its own LAYOUTGET, and after five minutes a PNFSDataThroughMDS event said the probes could not take the block layout, which they had. - The report is driven by what the node sees: the mount's READs and WRITEs going through the server for five minutes, whatever its probes say. That covers a fenced client, a kernel that cannot open the device (RHEL 9.5 on lab cluster 111), and a pod doing I/O the layout does not serve. The event names those causes and the last probe's outcome, without picking one. - A failed probe is retried after 2s, doubling to 30s, instead of on every pass. A reconnect still retries it at once. - A probe that has not returned for five minutes is reported once, as PNFSLayoutProbeHung. On cluster 111 a probe sat in D state in unlinkat and the mount was never probed again. Test plan: U-131 rewritten, U-136 to U-139 and F-22 added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnfs): send layout reports beside the primer loop, bounded, and retry a failed one Two review findings on #931: - The report ran in the primer's only loop, on a context with no deadline, through a client with no request timeout. An API server that accepted the request and never answered stopped probes, data re-probes, and alias refreshes for every mount on the node. Each report now runs in its own goroutine under a 30s deadline. - overdue() marked a mount reported before the report was recorded, and a failed report was only logged, so a mount that stayed affected never got its event. A failed report clears that mark and is sent again after a minute. The reporter returns its error to the primer instead of logging it. Test plan: U-144 and U-145. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#895) * feat(controlplane): HostMaintenance window takes a FoundationDB host through a node drain * ran make operator-build-installer * feat(controlplane): HostMaintenance window takes a FoundationDB host through a node drain * fix(controlplane): a HostMaintenance window leaves Holding on a budget the controller cannot count * fix(controlplane): AwaitingHost waits for the pods the drain evicted to return * fix(controlplane): a stale pass does not repeat a HostMaintenance step's event * operator: hand FoundationDB roles off before a drain evicts them * ran make operator-build-installer * fix(fleet): tidy go.mod for the operator's OpenTelemetry bump * fix: tidy the gateway, support-tool, and atlas-lib modules
… ControlPlane moves (#924)
) * fix(csi): give the path-loss watch the namespace under the stack, not the device on top Stage and heal registered the top of the volume stack with the path-loss watch. The watch compares its record with the namespace devices sysfs lists, and behind the dmLinear indirection (on by default since #835) or the LVM layers the top is a device-mapper node it never lists. Its first scan after a stage reported the volume as having lost every path, and the guardian marked it broken and restarted opted-in workloads that were healthy. The watch is now given the bottom layer's devices for every plan: the namespace under fabric, or each member under a stripe, as striped volumes already did. They are recorded under watchedDevicePaths; an unstage also reads the stripeDevicePaths key earlier versions wrote. Seen in the lblk e2e runs 44463031746 and 44487645991 ("Device /dev/dm-N is no longer present" right after a stage). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(volstack): remount the filesystem over a dm-linear volume that lost every path The dmLinear indirection keeps the device across total path loss, so the filesystem mounted on it kept what the outage did to it. In the lblk e2e run 44487645991, ext4 got write errors while the namespace was gone, aborted its journal, and failed every write after the heal re-pointed the mapping. Without the indirection the device would have vanished and the heal would have remounted on the new one, replaying the journal. - dmLinear marks its artifact Interrupted when the device it maps was removed. A removed device keeps its number while the mapping holds it, so the check cannot be fooled by a reused minor. - The runner heals an Interrupted layer only after every Yielder above it has let go, top down. The rest of the walk then remounts them over the re-pointed device. - The filesystem layer's Yield unmounts the staging path. It refuses with ErrHeldElsewhere while another mount shares the superblock (the old pod's bind mount), because a later mount would pick up the damaged filesystem again. NodePublish returns that as Unavailable, and kubelet retries after the old pod's unpublish. - FilesystemOps gains OtherMounts, which the CSI driver answers from its mount table by device number. A re-point without loss, the live subsystem flip, is not Interrupted and keeps the mount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(volstack): yield a filesystem only through an ordinary unmount Yield unmounted through clear, which falls back to a lazy unmount when the ordinary one is refused. A lazy unmount detaches the path and nothing else, so a filesystem something still had open survived it. Yield then reported success, the heal re-pointed the mapping and with that erased the interruption, and the next mount picked up the aborted filesystem. Yield now unmounts the ordinary way only and answers a refusal with ErrHeldElsewhere, so the publish retries. The lazy fallback stays in Release, which serves a device that is actually gone. Raised by the Copilot review of #941. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(integration): give the on-node shell filesystem OtherMounts The filesystem layer's ops gained OtherMounts in this branch, and the on-node suite's shellFilesystem, a fourth implementation that only builds on Linux, did not have it. Both integration jobs failed vet on it. It reads /proc/self/mountinfo and matches by device number, as the CSI driver's implementation does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(pnfs): verify a remote control plane by its own hostname when collecting mds-proxy ports On a managed consumer the operator asked the remote control plane for its storage nodes' LVS ports with the in-cluster webappapi client, which pins the TLS server name to simplyblock-webappapi.<ns>.svc and trusts only the in-cluster CA bundle. Every HTTPS endpoint failed verification, the mds-proxy ports stayed empty, and pNFS exports timed out in Assembling. * fix(webapi): verify every remote control plane by its own hostname All HTTPS clients shared one TLS client that pinned the server name to simplyblock-webappapi.<ns>.svc, so any control plane outside the Kubernetes cluster failed verification, not only the mds-proxy port lookup. NewClient now pins the Service name only for the in-cluster address and verifies any other address against its own hostname, trusting the system store plus the mounted CA bundle and presenting the mounted client certificate. NewRemoteClient is removed; the port lookup uses NewClient. * fix(webapi): rename the address variable that shadowed net/url
…942) * test(discovery): record each case as a delta from a shared baseline Each discovery case recorded its whole document and every note, so a change every fleet's draft makes the same way rewrote all 198 cases (#805 touched 396 expected files for one new field). The values more than half of the documents share are now recorded once in testdata/discovery/baseline.yaml, and each case's expected.yaml is a JSON merge patch against it. The comparison stays exact: the delta applied to the baseline must give the document back. A note is recorded only when the field its first word names differs from the baseline in that case. The wording of a note on a field a case shares with the baseline is no longer checked by the case tree. -update rewrites the baseline only when every case ran, so a run narrowed by -run records its cases against the baseline as it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(discovery): describe the baseline the cases are recorded against The case catalog shows baseline.yaml and what a case's expected files hold now. The expectation summary also counts baseline.yaml, says that one of its lines applies to every case, and no longer fails when a branch changes the baseline alone. A misaligned DEV-20 row is aligned so the table gate passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…og (#950) * fix(chart): stop fluent-bit from dropping JSON log lines before Graylog * fix(chart): restart fluent-bit when its config changes
* fix(operator): double the operator memory limit * fix(operator): double the operator memory limit to 256Mi * fix(operator): match the bundle's operator resources to the Helm chart
…952) * fix(controlplane): run the fdb-exporter from a fully qualified image * fix(controlplane): run the fdb-exporter from a fully qualified image
…us.mode Managed keeps the operator's own Prometheus and trusts simplyblock-prometheus as the metrics scraper. External creates ServiceMonitor, PodMonitor, ScrapeConfig, and PrometheusRule objects for a prometheus-operator that already runs, scraping as an operator-owned simplyblock-metrics-scraper account whose token Secret the ScrapeConfig references. Switching modes deletes what the other mode applied. Replaces the ServiceMonitor detection and the SB_EXTRA_METRICS_SERVICE_ACCOUNTS variable.
The chart passes prometheus.mode and prometheus.external.labels to the ControlPlane it bootstraps; the operator now creates the monitors and rules, so prometheus-operator-servicemonitors.yaml goes. kube-prometheus-stack gets its own enabled toggle, valid only in External mode. The Control Center and the metrics API read from the Prometheus the mode selects, and Thanos Query keeps the managed Prometheus's sidecar in Managed mode. The alerting template keeps only the AlertmanagerConfig routing, for External mode.
The storage-node, device, pool, and supercluster capacity controllers read the same flag, so with metricsAPI.enabled=false an External install fell back to the operator's built-in simplyblock-prometheus URL and lost every capacity sample. Adds helm-charts/scripts/test-prometheus-values.sh to the Helm lint workflow.
* feat(atlas): send the metadata journal on cluster create
The data plane can now create an lvstore with or without the metadata
journal (simplyblock/spdk d7561c5c4), and sbcli takes it at cluster
create as md_journal.
CreateClusterParams.MetadataJournal is sent when on and left out when
off, like the other two integrity settings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(api): enableMetadataJournal on StorageCluster and ClusterTemplate
Journals the lvstore metadata pages against torn writes, for devices that
do not complete a 4K write whole. Immutable and defaulting to off, like
the two integrity settings beside it, because the data plane records it
in the lvstore super block at creation.
The expansion copies it from the document, the reconciler sends it on
cluster create, v1alpha1 keeps it in a conversion annotation, and the
support tool edits it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(nodeprobe): report each device's logical block size
The atomic write units say what a device completes whole across a power
failure, but only together with the block size do they say whether a 4K
write does: a 4K-block device always does, and a 512-byte one only when
its atomic write unit reaches 4K. The block size was read by the scan and
dropped on the way into the report.
The support tool shows it beside the device size, with the atomic range
where the kernel published one. The captured-host discovery fixtures pick
up their recorded block sizes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(discovery): derive the metadata journal and atomicity from the devices
A device that does not complete a 4K write whole can tear a metadata page
on a power loss, and checksum validation on such a device works only with
a metadata format or 4K blocks. The draft now judges every device as a
storage node will have it, then derives the settings from that:
- A device writes 4K whole when its blocks are 4K, or when the atomic
write unit it reports runs from at most 4K to at least 4K. With
formatting on, an NVMe namespace not at 4K is judged at the plain 4K
format the control plane reformats it to (format 0 where it has none),
and its probed atomic write unit no longer applies. A device with no
reading counts as one that does not.
- enableMetadataJournal is on unless every device writes 4K whole.
- Checksum validation stays the reviewer's choice. A note says what it
would need: devices that need enableAtomicity4K because their report
covers 4K, devices it cannot check because their report does not, and
devices nothing says anything about. Where an installation states it
on, atomicity is derived the same way, and a device it cannot check
fails the run instead of failing the node add later.
Of the captured hosts, the GCP NVMe fleet (4K blocks) drafts the journal
off, and the GCP block fleet (512-byte disks, atomic only to 512) drafts it
on with a note that checksum validation could not check those disks. The
synthetic cases state no geometry, so they read as unknown and draft it on.
The detection runbook carries the rule and the new readings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(nvme): read each namespace's power-fail atomicity from the drive
The kernel publishes atomic write units only from 6.11 on, so on older
kernels every 512-byte NVMe namespace read as unknown and drafted the
metadata journal on. The drive states the same thing in the Identify data
the probe already reads for the LBA formats:
- Identify Namespace: NSFEAT bit 1 (the namespace states its own values),
NAWUPF, NABSPF, and NABO.
- Identify Controller: AWUPF, which applies where the namespace states no
value of its own.
atlas-lib decodes them into PowerFailAtomicity, with AtomicAt answering
whether an aligned write of a given size completes whole: it has to fit
the unit and must not cross a boundary. The report carries the fields on
each namespace's formats, and the support tool shows the unit and which
field stated it.
Discovery believes the drive over the kernel attributes where both
answer. The 6.11+ kernel ignores AWUPF ("AWUPF ignored, only NAWUPF
accepted") and gives up on any NABO, both of which the specification
allows a drive to rely on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(nvme): decode a real Samsung PM983's power-fail atomicity
Identify Controller and Identify Namespace dumps from a dev-ocp worker's
Samsung PM983 (MZQLB1T9HAJR-00007, firmware EDA5702Q), captured with
nvme-cli through oc debug, serial number and NGUID blanked.
The drive states its own power-fail unit (NSFEAT bit 1) as NAWUPF 0, one
block, and AWUPF is one block too. NAWUN is 127, which covers normal
operation only. At its 4K format a 4K write is atomic, which is what the
host's 6.12 kernel reported (atomic_write_unit_max_bytes 4096). Left at
512 bytes the same drive would not write 4K whole, and would need the
metadata journal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(nvme): the PM983 at its 512-byte format, which writes 4K whole too
The same dev-ocp drive, formatted to LBA format 0 (512 bytes), read, and
formatted back to 4K. It restates every unit in blocks of the format in
use: NAWUPF and AWUPF become 7 (eight blocks) and NABSPF 7 (a boundary
every eight), so it completes an aligned 4K write whole at 512 bytes as
well. The 6.12 kernel agreed: atomic_write_unit_max_bytes 4096 and
atomic_write_boundary_bytes 4096.
This corrects the previous commit, which read the 4K-format values as if
they held at 512 bytes and concluded the drive would tear a 4K write
there. It is also the evidence for judging a reading only at the format
it was taken at. Discovery gains the captured reading as a case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(discovery): a Google Cloud Local SSD is not 4K-atomic, whatever it reports
A controller cannot report less than one block of power-fail atomicity:
AWUPF and NAWUPF are 0's based. On a 4K namespace that floor reads as a 4K
atomic write without the device having promised anything, and the GKE
Local SSD read on 2026-10-27 reports exactly that. Google documents no
torn-write guarantee for Local SSD (only Hyperdisk has one: 128 KB within
1 MB boundaries), and simplyblock's deployment guide rules GCP out of
production for it.
atlas-lib/nvme gains AtomicityDisclaimed, a list of controllers, by PCI
vendor and device ID, whose statement is not believed. It starts with
Google's Local SSD controller (1ae0:001f). Discovery judges such a device
as not 4K-atomic whatever its block size, so the journal is on, and the
note says which devices were overruled and why.
Fallback checksums on 4K blocks assume a 4K write is whole, so a device is
checkable without enforced atomicity only with a metadata format or with
4K blocks that are 4K-atomic. A Local SSD therefore cannot be checked.
The captured GCP NVMe fleet now drafts the journal on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(inventory): a real GKE node reads as GKE
The node of a one-node GKE cluster (n2-standard-2, COS, kernel 6.12, GKE
1.36), created for the device capture and deleted after it, trimmed to its
labels, annotations, and node info without the instance and project
identifiers. Environment detection concludes GKE from the kubelet version
and the cloud.google.com/gke- labels, which is all it has: GKE registers no
API group of its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(transcript): WithClassBlock keeps the root files and machine name
WithClassBlock rebuilt the Host from its sysfs maps alone, so a host
capture that seeds block/ lost its os-release and uname machine on the way
to a fixture and read as an unknown distribution. The captures the
operator's discovery cases replay predate root files, so nothing replayed
there changes; the EKS capture added next was the first to carry them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: what a real EKS node and its AWS devices report
From a one-node EKS cluster (i4i.large, Amazon Linux 2023, kernel 6.18,
EKS 1.37) created for the capture and deleted after it, with an EBS io2
volume attached beside the gp3 root and the instance store. All three
devices are formatted at 512 bytes, and serial numbers, NGUIDs, EUI-64s,
and host names are blanked or replaced.
- The whole host, captured with hack/inventory/capture-host.py. The scan
reads EBS (gp3 and io2) as atomic from 512 to 16384 bytes and the
instance store as atomic to 512 only, which matches the kernel.
- The Identify dumps of an EBS volume and the instance store. EBS states
its own unit (NSFEAT bit 1, NAWUPF 31): 16 KiB within 16 KiB
boundaries, as AWS documents. The i4i instance store states nothing and
its AWUPF is the one-block floor, so a 4K write is not promised whole.
AWS documents torn-write prevention for some I-family instance stores,
and the i4i is not among the devices that claim it.
- The node, which environment detection reads as EKS from the kubelet
version and the eks.amazonaws.com labels.
- Discovery cases for both devices: EBS needs no journal, the instance
store does. No vendor list is involved either way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(atlas): always send md_journal on cluster create
sbcli now journals unless told not to (simplyblock/sbcli#1551), so
leaving md_journal out when it is off, as the other two integrity
settings are, would create a journaled cluster for a StorageCluster that
states enableMetadataJournal: false. The setting is now sent either way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(discovery): pin the draft's SPDK and SPDK proxy images again
discovery.draft.images.spdk and .spdkProxy seeded the draft's spec.images
through the bootstrap ConfigMap until #780 removed it. Since then the
chart accepted both and dropped them, so an installation pinning an SPDK
build got the default one without a word.
The run carries them now: OperatorOps spec.discover.images, copied
unchanged into the draft it writes, and from there onto every
StorageNode. The chart renders the two values into it. Any Discover run
can state images, not only the one the chart creates.
The render check covers both values, one alone, and the other draft
values still reaching nothing. The operator test was shown failing first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(storagecluster): journal the metadata unless the cluster states off
enableMetadataJournal defaulted to false, so a StorageCluster written
without the field created lvstores that a torn write could leave with a
half-written metadata page. It now defaults to true, matching sbcli's own
default, and the create request reads an unset field the same way.
A discovery run is unaffected: it always states the field, and writes false
only when every device is known to complete a 4K write whole.
A cluster that already exists is not changed, since the flag is sent only
at create. A cluster stored before the field existed now reads true in its
spec, while its lvstores keep the off they were created with.
Both tests were shown failing first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011oztNZUjNFk6NYW7UXA1A2 # Conflicts: # .dockerignore # operator/api/v1alpha2/zz_generated.deepcopy.go # operator/dist/install.yaml # operator/internal/upgrade/crds/manifests/storage.simplyblock.io_controlplanes.yaml
Contributor
Discovery expectationsThis branch changes 399 expected files across 199 cases. Identical lines are grouped, most frequent first. - 199 x storage.simplyblock.io/component: nodeprobe
- 199 x approved: false
- 199 x namespace: simplyblock
- 199 x step: {}
- 199 x apiVersion: storage.simplyblock.io/v1alpha2
- 199 x kind: ClusterDeploymentConfig
- 199 x status:
- 198 x enableDriveFormat: true
- 198 x maxSubsystemCount: 30
- 198 x environment: K3s
- 198 x enableDriveFormat is set, so every drive listed here is formatted before a storage node takes it: a drive that carries anything is not usable otherwise. This is the line to remove if any of them shoul...
- 198 x maxSubsystemCount is 30, which is the middle of the range this API accepts and not a reading: how many subsystems a node should serve follows from the workload rather than from the hardware
- 196 x distro: ubuntu
- 196 x family: Debian
- 196 x hostOS:
- 196 x hostOS is ubuntu of the Debian family, read from every worker's os-release. It is what ubuntuHost is decided by, and the one distribution that decides it is Ubuntu, whose NVMe-oF modules are in a pack...
- 195 x spdkOnHostNetwork: true
- 195 x network:
- 193 x vcpuCount: 5
- 181 x parityChunks: 1
- 179 x minHugePagesSize: 16G
- 171 x enableJournalDevice is left unset: worker-01 hands over no single disk smaller than its others, so there is none to dedicate. Setting it would give up a disk the fleet did not set aside, and the field...
- 171 x minHugePagesSize is 16G, the huge pages free on the memory node chosen on worker-01, which is the smallest of the fleet
- 159 x vcpuCount is 5, 10% of the 32 logical CPUs of the memory node chosen on worker-01 (the smallest of the fleet) per storage node, at least 5: the control plane assumes this uniform across a cluster's no...
- 142 x network runs the SPDK pods on the host network, because most workers of the draft have a data interface. Each group names its own NICs. A cluster's network is fixed when it is created, so this is the ...
- 136 x clusterMode: Single
- 116 x dataChunks: 2
- 116 x stripe:
- 91 x stripe is 2+1 as a single-node cluster (clusterMode Single): each chunk of a stripe lands on its own device of the one worker, which hands over 4 for data, so the cluster survives 1 device failure but...
- 53 x network runs the SPDK pods on the host network, on the primary NIC, because no worker has a data interface of its own. A cluster's network is fixed when it is created, so this is the line to correct n...
+ 42 x clusterMode: null
- 14 x vcpuCount is 5, 10% of the 16 logical CPUs of the memory node chosen on worker-01 (the smallest of the fleet) per storage node, at least 5: the control plane assumes this uniform across a cluster's no...
+ 12 x minHugePagesSize: null
- 7 x stripe is 2+1, which needs 4 storage nodes and this fleet of 32 has them: it is the widest stripe to propose without knowing the workload. This fleet could also carry 1+1, 4+1, 1+2, 2+2, 4+2, which tr...
- 6 x enableJournalDevice is left unset: worker-02 hands over no single disk smaller than its others, so there is none to dedicate. Setting it would give up a disk the fleet did not set aside, and the field...
- 6 x minHugePagesSize is 16G, the huge pages free on the memory node chosen on worker-02, which is the smallest of the fleet
- 6 x stripe is 2+1 as a single-node cluster (clusterMode Single): each chunk of a stripe lands on its own device of the one worker, which hands over 3 for data, so the cluster survives 1 device failure but...
- 6 x vcpuCount is the default minimum of 5, because no worker reported the cores of the memory node it was placed on
- 5 x vcpuCount is 5, 10% of the 32 logical CPUs of the memory node chosen on worker-02 (the smallest of the fleet) per storage node, at least 5: the control plane assumes this uniform across a cluster's no...
- 4 x vcpuCount is 5, 10% of the 8 logical CPUs of the memory node chosen on worker-01 (the smallest of the fleet) per storage node, at least 5: the control plane assumes this uniform across a cluster's nod...53 more distinct lines are not shown. Run |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #692. Replaces the
prometheus.enabledHelm value with a ControlPlane field, so OperatorHub and Helm installs choose the Prometheus the same way.API
spec.source.local.observability.prometheus:There is no mode without a Prometheus.
Operator
simplyblock-prometheusSB_K8S_METRICS_SERVICE_ACCOUNTS)simplyblock-prometheussimplyblock-metrics-scraper(operator-owned, token Secret)controlplane/manifests/alert-rules.yaml.prometheusScrapesTheManagementAPI) andSB_EXTRA_METRICS_SERVICE_ACCOUNTS.Chart
prometheus.enabled,prometheus.serviceAccount,prometheus.additionalLabels,prometheus.interval, andprometheus.simplyblock.*are replaced byprometheus.modeandprometheus.external.{labels,url}.kube-prometheus-stack.enabled(default false) is the subchart condition.validate-prometheus.yamlrejects it outside External mode, and keeps the CRD conflict check.prometheus-operator-servicemonitors.yamlis deleted. The operator creates those objects now.prometheus-operator-alerting.yamlkeeps only the AlertmanagerConfig and its Secret, rendered in External mode.--prometheus-urlfollow the mode.metricsAPI.prometheusURLunset now means "follow the mode", and""still means off.--storeback for the managed Prometheus's sidecar in Managed mode.Review findings on #692 this addresses
simplyblock-prometheuswas dropped from the trusted scrapers.prometheus-operatedwith no prometheus-operator installed.authorization.credentialsFile. The ScrapeConfig CRD in kube-prometheus-stack 92.2.0 has onlycredentialsandtype.withoutPrometheusdroppedsimplyblock-prometheus-client-tls, which the mutual-TLS monitors need.Testing
prometheus_mode_test.go: 12 tests.TestControlPlaneCELPrometheusModewas not run red: the API type was added before it.TestExternalScrapeConfigAuthenticatesWithASecretwas not run red because my-runfilter missed it. The old code built no ScrapeConfig at all.go test ./internal/...(with envtest) andmake lintpass.helm templatechecked:kube-prometheus-stack.enabled, labels, and a Slack receiver.helm-charts/scripts/test-prometheus-values.sh(new, in the Helm lint workflow): red on the--prometheus-urlcase below, then green.Live test on cluster .61 (k3s, TLS off, metrics API off), 2026-11-20
Managed → External (bundled kube-prometheus-stack, Slack receiver) → Managed, image
feat-prometheus-mode:SB_K8S_METRICS_SERVICE_ACCOUNTS=simplyblock-prometheus;simplyblock_metricsandfdb_metricstargets upsimplyblock-metrics-scrapersimplyblock-alertsrouteFound and fixed live: with
metricsAPI.enabled=falsethe chart passed no--prometheus-url. The capacity controllers fell back tosimplyblock-prometheus, which External mode had just deleted (bd1d028).Found, not caused by this PR:
helm upgradedoes not install a subchart's CRDs, so enablingkube-prometheus-stackon an existing release needs the monitoring.coreos.com CRDs applied by hand (--server-side) andkube-prometheus-stack.crds.enabled=false.Known gaps
RootFilesystemLowSpace: it needs node-exporter, which the bundled kube-prometheus-stack keeps off.alert-rules.yamlstay in step.