Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Test philosophy: assert external behavior, never internal call ordering or priva
- **License renewal is live** (decided 2026-10-03, organization rule changed 2026-10-05): pods read the license Secret into env at start, so the operator reads it too and `SET DATABASE SETTING`s `organization.name` then `enterprise.license`, each only where it differs, on **every ready pod of both roles**, read-before-write against `SHOW DATABASE SETTINGS` (`settings.License`). It runs in `Reconcile` **before** registration and writes its own `LicenseApplied` condition (`LicenseMatchesSecret` / `LicensePending` / `LicenseRejected`), because an expired license is what stops convergence and every registration return overwrites `Converged`. Secret RBAC is `get;list;watch` only: a metadata-only watch (`WatchesMetadata`) maps a Secret to the clusters naming it, and the value is read uncached through `APIReader`; the value is never logged, evented or put in status. A different organization is SET like the license (Andi's call, no special case). A shorter-expiry replacement is assumed away for now. Core masks `enterprise.license` in its query log (Andi's fix), since Memgraph rejects Bolt parameters in the SET grammar.
- **AWS credentials** (`spec.awsCredentials.secretName`, decided 2026-10-05) carry all four AWS values from one Secret (Andi's call: region and endpoint too, so they left `spec.flags` and the run-time table) and follow the license on both paths, data instances only (coordinators run no S3 queries). Fixed, all-required keys `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`/`AWS_REGION`/`AWS_ENDPOINT_URL` (AWS SDK names; empty endpoint for AWS itself): required because Kubernetes leaves an unresolvable `$(VAR)` as literal text. Startup: env from `secretKeyRef` into `MEMGRAPH_AWS_*`, handed to the flags by Kubernetes `$(VAR)` expansion, appended after the pinned args; a flag is required because all four settings are `kRestore` (persisted across restarts and restored over Memgraph's own `AWS_ACCESS_KEY` env fallback, but not over an explicit flag), and Memgraph's gflags has no `--fromenv`/`--tryfromenv` (verified on 3.13.0). Live: `reconcileAWSCredentials` SETs `aws.access_key`/`aws.secret_key`/`aws.region`/`aws.endpoint_url` read-before-write on every ready data pod in the converged branch beside the run-time flags, reporting on `Converged` (`SettingsPending`/`SettingsRejected`, setting named, value never); the shared metadata Secret watch maps it too. Removing the block issues no SET, like removing a flag. All four `aws-*` flags are denied in `spec.flags`, `MEMGRAPH_AWS_*` in `extraEnv`.
- Storage is never deleted by the operator: no finalizer-based cleanup; PVC retention (deletion *and* scale-down) maps to the StatefulSet PVC retention policy (default `Retain`). The only cluster members the operator removes are the ones a lowered replica count retires; a coordinator removed from Raft keeps running and keeps its state on purpose, which is what makes re-growing onto a retained volume safe.
- Workload pods: non-root uid 101 / gid 103 / fsGroup 103 unless `spec.securityContext` is written, in which case exactly its `runAsUser`/`runAsGroup`/`fsGroup` land and no other (an empty block leaves all three to the platform: OpenShift `restricted-v2` assigns them at admission and rejects named ones; `runAsNonRoot` and seccomp stay unconditional; not pinned, because moving the uid is what the ownership container is for), seccomp RuntimeDefault, all capabilities dropped, `terminationGracePeriodSeconds: 300` (a ceiling, not a delay — an instance killed mid-shutdown recovers from its WAL and lengthens the catch-up a roll waits on). The only privileged containers the operator builds are two init containers that write under `/proc/sys`, both on the cluster's own Memgraph image: `init-sysctl` (`spec.sysctlInitContainer`, the HA chart's block minus its image knobs and its `enabled`: presence-based, absent means off, so the sample and the quickstart example carry it) raises the node's `vm.max_map_count` to `maxMapCount` and only ever raises — the default 524288 is the floor Memgraph itself checks at startup, not the chart's 262144 — and `init-core-pattern` (`coreDumps.configureCorePattern`) sets the core pattern for a role collecting dumps. Both are node properties no namespaced sysctl can set, so a PodSecurity `restricted` namespace must leave the block out and set `configureCorePattern: false`. A third init container, `init-fix-perms` (`spec.fixOwnershipInitContainer`, the HA chart's block minus its image knobs, its `enabled` and its uid/gid knobs; presence-based with no fields, off when absent like the chart), runs last on the same image as root with `CAP_CHOWN` alone, not privileged, and chowns the lib, log and core dumps mounts to the pod's identity (`runAsUser`, and `runAsGroup` or else `fsGroup`) for storage drivers that ignore `fsGroup` (rancher.io/local-path); `restricted` rejects it too since it runs as root, and a spec-level CEL rule rejects it beside a `securityContext` that names no uid or no group, because it cannot chown to an identity the platform assigns later. Every e2e namespace enforces `restricted` (`createClusterNamespace`), so the quickstart carries none of the three and one dedicated scenario in an unlabeled namespace (`test/e2e/init_containers_test.go`) proves all of them.
- Workload pods: non-root uid 101 / gid 103 / fsGroup 103 unless `spec.securityContext` is written (an absent block is schema-defaulted to those three, so the stored object shows the identity; no field inside it has a default), in which case exactly its `runAsUser`/`runAsGroup`/`fsGroup` land and no other (an empty block leaves all three to the platform: OpenShift `restricted-v2` assigns them at admission and rejects named ones; `runAsNonRoot` and seccomp stay unconditional; not pinned, because moving the uid is what the ownership container is for), seccomp RuntimeDefault, all capabilities dropped, `terminationGracePeriodSeconds: 300` (a ceiling, not a delay — an instance killed mid-shutdown recovers from its WAL and lengthens the catch-up a roll waits on). The only privileged containers the operator builds are two init containers that write under `/proc/sys`, both on the cluster's own Memgraph image: `init-sysctl` (`spec.sysctlInitContainer`, the HA chart's block minus its image knobs and its `enabled`: presence-based, absent means off, so the sample and the quickstart example carry it) raises the node's `vm.max_map_count` to `maxMapCount` and only ever raises — the default 524288 is the floor Memgraph itself checks at startup, not the chart's 262144 — and `init-core-pattern` (`coreDumps.configureCorePattern`) sets the core pattern for a role collecting dumps. Both are node properties no namespaced sysctl can set, so a PodSecurity `restricted` namespace must leave the block out and set `configureCorePattern: false`. A third init container, `init-fix-perms` (`spec.fixOwnershipInitContainer`, the HA chart's block minus its image knobs, its `enabled` and its uid/gid knobs; presence-based with no fields, off when absent like the chart), runs last on the same image as root with `CAP_CHOWN` alone, not privileged, and chowns the lib, log and core dumps mounts to the pod's identity (`runAsUser`, and `runAsGroup` or else `fsGroup`) for storage drivers that ignore `fsGroup` (rancher.io/local-path); `restricted` rejects it too since it runs as root, and a spec-level CEL rule rejects it beside a `securityContext` that names no uid or no group, because it cannot chown to an identity the platform assigns later. Every e2e namespace enforces `restricted` (`createClusterNamespace`), so the quickstart carries none of the three and one dedicated scenario in an unlabeled namespace (`test/e2e/init_containers_test.go`) proves all of them.
- **Readiness probe only, no liveness or startup probe**, on both roles. Memgraph recovers every database inside the `DbmsHandler` constructor before any server starts, so during recovery nothing listens and no probe can tell slow from hung; a TCP liveness check could only kill a recovery that outlived a guessed budget (the HA chart's 2h), and a Bolt listener that disappears after startup means the process exited, which restarts the container anyway. A recovering pod is simply unready. The field converged on the same answer (ECK, CockroachDB, MongoDB Community run no liveness; Strimzi and cass-operator make it a process check; Percona deprecated its 7200s startup delay). Do not reintroduce them; a core health endpoint reporting recovery *progress* is the only thing that would justify a liveness check.
- StatefulSet `volumeClaimTemplates` cannot be changed, added or removed on a live cluster. **Sizes grow** (`storage.<role>.libPVCSize`, default 10Gi, `coreDumps.<role>.size` while the role collects dumps; CEL lets them grow, never shrink, compared as quantities; decided 2026-10-05): `internal/volumes` (the fourth pure core) patches every claim carrying the role's selector labels, retained ones included, then once none is left to patch orphan-deletes the StatefulSet (`propagationPolicy: Orphan`) and ends the pass; the next apply recreates it with the grown templates. That restarts nothing because the StatefulSet controller adopts a pod when its volumes name the claims its ordinal gets (`storageMatches`: names, never sizes) and the revision hashes `spec.template` alone — proven on a real StatefulSet controller in Kind. Until then every apply restates the live sizes (`volumes.KeepLiveSizes`), a refused claim patch (StorageClass without `allowVolumeExpansion`; no StorageClass pre-check, no RBAC on it) is `Converged=False/VolumeExpansionRefused` and blocks the recreate, a claim a pod mounts below its request is `VolumeExpansionInProgress` naming claim and pod plus the driver's first `ControllerResizeError`/`NodeResizeError` message, tail kept when it overflows the 1024 cap (found on AKS: `Standard_A2_v2` nodes cannot change an attached disk), an unmounted one counts once only its filesystem is left, `*ResizeInfeasible` is `VolumeExpansionFailed`. While a StatefulSet is missing `replicaCounts` reads its count off the highest pod ordinal + 1, so a count lowered in the window still retires members first. **No pod is ever restarted for a resize** (Andi's call after the Kind proof): `FileSystemResizePending` is a transient state on every online driver (~1 min until the kubelet's sync), and an offline-only driver waits until no pod uses the claim, which a restart does not give it (`docs/storage-resize.md` lists them). Everything else is pinned by a CEL transition rule on the type that carries it: `storage.<role>.{libStorageAccessMode,libStorageClassName}`, the presence of `coreDumps.<role>` (pinned on `CoreDumpsSpec`, because a rule on a block never fires for the block being added or removed whole) and `coreDumps.storageClassName` while a role collects dumps: adding or removing a template changes the pods' volumes, adoption fails, and the StatefulSet controller only recovers adopted pods in ascending ordinal order, MAIN first, which deadlocks the roll. The doc comment on `RoleStorageSpec` carries the argument; read it before touching any claim template knob. The e2e resize scenario runs on the CSI hostpath driver's standard (single-node) deployment, `hack/kind-csi-hostpath.sh`: the distributed one has no resizer.
- **Logs live on the lib claim** (decided 2026-10-07; the separate `log-storage` claim and its four `log*` knobs are gone): `storage.<role>.fileLogging` (bool, default `true`, a default-on toggle like `configureCorePattern`) pins `--log-file=/var/lib/memgraph/logs/memgraph.log`, off pins `--log-file=""`. Never a path knob and `log_file` stays denied in `spec.flags`: Memgraph's `CleanLogsDir` deletes files older than `log_retention_days` from the log file's **parent directory**, so the file must sit in a directory of its own, never the claim root, and any path off a writable mount crash-loops on the read-only root. The empty value is load-bearing: the image's `memgraph.conf` sets `log_file` to `/var/log/memgraph/memgraph.log`. Toggling is an ordinary roll (no CEL pin, no claim template involved); turning it off leaves old log files on the claim, never cleaned by the operator. Logs count toward `libPVCSize`, which is why the defaults moved to 10Gi and `log_level=INFO`.
Expand Down
48 changes: 29 additions & 19 deletions api/v1alpha1/memgraphcluster_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@ const (

DefaultMaxMapCount int64 = 524288

// The identity of the memgraph user baked into the official images, the
// securityContext block's default.
DefaultRunAsUser int64 = 101
DefaultRunAsGroup int64 = 103
DefaultFSGroup int64 = 103

DefaultLibPVCSize = "10Gi"
DefaultFileLogging = true
DefaultStorageAccessMode = corev1.ReadWriteOnce
Expand Down Expand Up @@ -115,10 +121,10 @@ const (
MonitoringPort int32 = 7444
)

// Readiness probe timing defaults, Go constants mirrored by the doc comments
// on ReadinessProbeSpec. A readiness probe that has not succeeded yet only keeps
// the pod unready, so the failure threshold is not a budget anything has to fit
// in.
// Readiness probe timing defaults, declared as CRD schema defaults on
// ReadinessProbeSpec and mirrored here like the defaults above. A readiness
// probe that has not succeeded yet only keeps the pod unready, so the failure
// threshold is not a budget anything has to fit in.
const (
DefaultProbeFailureThreshold int32 = 20
DefaultProbeTimeoutSeconds int32 = 10
Expand Down Expand Up @@ -665,6 +671,7 @@ type CoreDumpsUploaderSpec struct {
// default for it, so a tag belongs here rather than in a separate field.
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=383
// +required
Image string `json:"image"`

// pullPolicy is the image pull policy of the sidecar.
Expand Down Expand Up @@ -839,12 +846,11 @@ type SysctlInitContainerSpec struct {
// memgraphUserId and memgraphGroupId, reshaped so that a platform assigning
// the identity itself can be told to.
//
// The block is presence-based, and presence means the user owns these three
// fields. Absent, the operator writes the identity baked into the Memgraph
// Absent, the block is defaulted to the identity baked into the Memgraph
// images: uid 101, gid 103 and fsGroup 103, which is what makes a claim
// writable to the non-root process on a driver that honors fsGroup. Present,
// exactly the fields written land on the pod and no other: an empty block
// writes none of the three, which is what OpenShift's restricted-v2 SCC
// writable to the non-root process on a driver that honors fsGroup. No field
// inside it has a default, so exactly the fields written land on the pod and
// no other: an empty block writes none of the three, which is what OpenShift's restricted-v2 SCC
// wants, since it assigns runAsUser and fsGroup from the namespace's range at
// admission and rejects a pod naming values outside it. Naming values covers
// a uid inside that range or a service account granted anyuid. Memgraph
Expand Down Expand Up @@ -921,8 +927,8 @@ type FixOwnershipInitContainerSpec struct{}
//
// Until the probe first succeeds — which for a data instance is after every
// database has been recovered — the pod is unready and nothing more, so none of
// these timings is a budget a recovery has to fit in. Every field defaults to
// the value named in its doc comment.
// these timings is a budget a recovery has to fit in. Every field has a CRD
// schema default, so the stored object shows the timings the pods run with.
//
// There is deliberately no liveness and no startup probe. Memgraph recovers its
// databases before it opens any port, so during recovery nothing distinguishes
Expand All @@ -939,19 +945,21 @@ type FixOwnershipInitContainerSpec struct{}
// which knows the cluster's state, not to the kubelet, which does not.
type ReadinessProbeSpec struct {
// failureThreshold is how many consecutive failures flip a pod that was
// ready to unready. Defaults to 20.
// ready to unready.
// +kubebuilder:validation:Minimum=1
// +kubebuilder:default=20
// +optional
FailureThreshold *int32 `json:"failureThreshold,omitempty"`

// timeoutSeconds is how long a single probe attempt may take. Defaults to
// 10.
// timeoutSeconds is how long a single probe attempt may take.
// +kubebuilder:validation:Minimum=1
// +kubebuilder:default=10
// +optional
TimeoutSeconds *int32 `json:"timeoutSeconds,omitempty"`

// periodSeconds is how often the probe runs. Defaults to 5.
// periodSeconds is how often the probe runs.
// +kubebuilder:validation:Minimum=1
// +kubebuilder:default=5
// +optional
PeriodSeconds *int32 `json:"periodSeconds,omitempty"`
}
Expand Down Expand Up @@ -2043,10 +2051,11 @@ type MemgraphClusterSpec struct {
SysctlInitContainer *SysctlInitContainerSpec `json:"sysctlInitContainer,omitempty"`

// securityContext is the uid, gid and fsGroup every pod of both roles runs
// under. Absent, the operator writes the Memgraph images' 101, 103 and 103;
// present, exactly the fields written and no other, so an empty block
// leaves all three to the platform, which is what OpenShift's restricted-v2
// SCC requires.
// under. Absent, it is defaulted to the Memgraph images' 101, 103 and 103;
// written, exactly the fields written and no other land on the pod, so an
// empty block leaves all three to the platform, which is what OpenShift's
// restricted-v2 SCC requires.
// +kubebuilder:default={runAsUser: 101, runAsGroup: 103, fsGroup: 103}
// +optional
SecurityContext *PodSecurityContextSpec `json:"securityContext,omitempty"`

Expand All @@ -2069,6 +2078,7 @@ type MemgraphClusterSpec struct {

// readinessProbe tunes the readiness probe timings of every pod, the one
// probe the pods carry.
// +kubebuilder:default={}
// +optional
ReadinessProbe ReadinessProbeSpec `json:"readinessProbe,omitzero"`

Expand Down
2 changes: 1 addition & 1 deletion charts/memgraph-operator/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Kubernetes operator for Memgraph high-availability clusters
type: application
# version is the chart version, appVersion the operator version it installs.
# Both move together with the git tag at release time.
version: 1.0.1-rc.1
version: 1.0.1-rc.2
appVersion: "1.0.1-rc.1"
# The CustomResourceDefinition uses CEL validation rules
# (x-kubernetes-validations), available from Kubernetes 1.25.
Expand Down
Loading
Loading