From 54d33cd86031ffaa71c4ac95a23c6e2bd87862ea Mon Sep 17 00:00:00 2001 From: as51340 Date: Wed, 14 Oct 2026 08:35:13 +0200 Subject: [PATCH] feat!: Replace the two spec booleans with enums storage..fileLogging and coreDumps.configureCorePattern were the only booleans in the spec, and both answer a question with more than two answers. Where the log files go already lost one, the separate log claim, and could grow another (an emptyDir). Who sets the node's core_pattern could be a per-node DaemonSet instead of a privileged init container in every pod. - storage..fileLogging (bool, default true) becomes storage..logFile: LibClaim | None, default LibClaim. - coreDumps.configureCorePattern (bool, default true) becomes coreDumps.corePattern: Operator | Node, default Operator. The pods are unchanged: the builders derive the same switches from the enums, treating anything but None and Node as the default so a spec that never passed admission behaves like an admitted one. Admission rejects a value it does not know. BREAKING CHANGE: the old fields are gone from the schema, so the API server prunes them, and a cluster that set either to false reads the new default once the CRD is upgraded: its pods roll onto file logging, or gain the privileged core pattern init container, which a restricted namespace rejects. Stop the operator, upgrade the CRD, set logFile: None or corePattern: Node on those clusters, then start the new operator. --- CLAUDE.md | 6 +- README.md | 2 +- api/v1alpha1/memgraphcluster_types.go | 88 +++++++++---- api/v1alpha1/zz_generated.deepcopy.go | 10 -- charts/memgraph-operator/Chart.yaml | 2 +- .../crds/memgraph.com_memgraphclusters.yaml | 118 ++++++++++-------- .../bases/memgraph.com_memgraphclusters.yaml | 118 ++++++++++-------- config/samples/v1alpha1_memgraphcluster.yaml | 25 ++-- docs/backup-restore.md | 2 +- docs/flags.md | 2 +- docs/rolling-restart-pacing.md | 2 +- .../memgraphcluster_validation_test.go | 28 +++-- internal/resources/resources.go | 19 ++- internal/resources/statefulset_test.go | 6 +- test/e2e/external_access_test.go | 4 +- test/e2e/gateway_access_test.go | 4 +- test/e2e/memgraphcluster_test.go | 4 +- test/e2e/tls_test.go | 4 +- test/e2e/volume_resize_test.go | 2 +- 19 files changed, 249 insertions(+), 197 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ba4fe1f..384622f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -63,7 +63,7 @@ Test philosophy: assert external behavior, never internal call ordering or priva ## Conventions - Spec knob names mirror the HA Helm chart's vocabulary where the concept carries over (e.g. the `secrets.name` / `secrets.licenseKey` / `secrets.organizationKey` block) — check the chart before inventing a name. -- Every optional block is presence-based and there is no `enabled` anywhere in the API (decided 2026-09-28): `externalAccess`, `monitoring.*`, `tls.*`, `scheduling.podAntiAffinity`, `sysctlInitContainer`, `fixOwnershipInitContainer`, `securityContext` and `coreDumps.` are on when written and off when absent. A behavior that should be on for most users lives in the sample and the quickstart example, never in a schema default, because a schema default on a block makes it impossible to turn off by leaving it out. The one default-on toggle left is `coreDumps.configureCorePattern`, a bool inside a block that is itself presence-based. +- Every optional block is presence-based and there is no `enabled` anywhere in the API (decided 2026-09-28): `externalAccess`, `monitoring.*`, `tls.*`, `scheduling.podAntiAffinity`, `sysctlInitContainer`, `fixOwnershipInitContainer`, `securityContext` and `coreDumps.` are on when written and off when absent. A behavior that should be on for most users lives in the sample and the quickstart example, never in a schema default, because a schema default on a block makes it impossible to turn off by leaving it out. There is no bool in the spec either (decided 2026-10-14): a choice that starts as on/off is an enum, because those grow a third answer (`storage..logFile: LibClaim | None` already lost one, the separate log claim); the default-on choice inside a presence-based block is `coreDumps.corePattern` (`Operator | Node`, default `Operator`). - **Memgraph flags** (`spec.flags.`, `map[string]string`, flag name without dashes to value; replaced the `extraArgs` list 2026-10-02) never travel as container args: each role's flags are rendered into a gflags flag file in a ConfigMap `--flags` (canonical underscore spelling, sorted, the operator's logging defaults `log_level=INFO`, `also_log_to_stderr=true`, `log_retention_days=35` as the base the map merges over) that the container loads with `--flag-file` as its **first** argument; the command line carries exactly the pinned flags admission denies in the map (ports, `bolt-address`, `monitoring-address`, `monitoring-port`, `metrics-port`, `metrics-format`, coordinator identity, `data-directory`, `log-file`, the TLS file flags) plus, on data instances with `spec.awsCredentials`, `--aws-access-key`/`--aws-secret-key`/`--aws-region`/`--aws-endpoint-url` as `$(MEMGRAPH_AWS_*)` references (see AWS credentials below). gflags processes `--flag-file` in argument order and takes the last occurrence, so a pinned flag always wins; an unknown name inside a flag file is silently ignored, a bad value is fatal. The pod template carries `memgraph.com/flags-restart-hash` over the **startup-only** lines only, so a startup-only change rolls through `internal/rollout` unchanged and a run-time change rolls nothing. Run-time flags (the sixteen in `internal/settings`, flag name ≠ setting name, copied from `src/flags/run_time_configurable.cpp` and pinned by a test) are applied with `SET DATABASE SETTING` on **every ready pod of both roles** in the converged branch before the roll picks a pod, each pod dialed over pod DNS (`resources.DataEndpoints` joined `CoordinatorEndpoints`), read-before-write, only keys present in the file. Memgraph refuses Bolt parameters in that grammar, so values are rendered as Cypher string literals (`memgraph.cypherString`). Failures ride `Converged`: `SettingsPending` (ready pod did not answer), `SettingsRejected` (pod, command, Memgraph's error verbatim); neither fails the pass or escalates to a restart. Removing a key issues no `SET` (Andi's call until core gains a reset): a non-persisted setting reverts on the next restart, a `kRestore` one keeps its last value; a removed override of an operator default does revert, because the file then says the default. - **What a key is, the operator asks Memgraph** (Andi's corrections 2026-10-02): there is no `coordinatorSettings` block. Coordinator settings (`enabled_reads_on_main`, `instance_down_timeout_sec`, ...) go under `flags.coordinators`, and a key is classified per pass by `settings.Classify` against `SHOW CONFIG` (flags, read from a pod on the current template; Memgraph hides `also_log_to_stderr`, `scheduler`, `license_key`, `organization_name` from it, the first two allowed, the last two denied) and `SHOW COORDINATOR SETTINGS` (coordinator settings), the coordinator settings read over the leader connection. Coordinator settings are `SET COORDINATOR SETTING` once (`settings.Diff`); unknown names are `Converged=False/UnknownFlags` and applied by nothing. The pod template's `memgraph.com/flags` annotation is per-key digests of every non-run-time key and `memgraph.com/template-hash` (`resources.PodTemplateHash`) hashes everything else, both pure functions of the spec; the **restart** decision is per pod: a pod is considered for exemption only when its template hash matches its role's (the flags are the one thing that changed), then `resources.ChangedFlags` between the pod's flags annotation and the template's, and a pod whose changed keys are all non-flags is marked `rollout.Pod.Current` and not restarted. `SHOW CONFIG` is local, not forwarded, so it is read from a ready pod already on its role's template (data first), the leader only as fallback: during an image upgrade the leader is the last pod on the old binary. Views unavailable with changed keys means `Updated=False/FlagsUnclassified` and no restart: never restart on a guess. Admission keeps shape checks for the eight known coordinator setting names and rejects them under `flags.data`. - **User containers** (`spec.userContainers.`, the HA chart's block) are raw `core/v1` Containers appended after the operator's own (Memgraph, then the uploader), stored schemaless like `extraVolumes` because a full Container schema per role would push the CRD past the client-side apply limit. The builder fills the restricted security context on a container that names none, so the chart's busybox example runs under PodSecurity `restricted`; a container naming its own keeps it. Name collisions and malformed entries surface as `ApplyFailed`, not at admission. **Init containers** (`spec.initContainers.`, the chart's block) are the same passthrough, with the same schemaless storage and the same security context fill, appended after the operator's own init containers (`init-sysctl`, `init-core-pattern`, `init-fix-perms`) in the chart's order, so a user's runs on volumes the ownership container already fixed. @@ -71,10 +71,10 @@ Test philosophy: assert external behavior, never internal call ordering or priva - **License renewal is live** (decided 2026-10-03, organization rule changed 2026-10-05): pods read the license Secret into env at start, so the operator reads it too and `SET DATABASE SETTING`s `organization.name` then `enterprise.license`, each only where it differs, on **every ready pod of both roles**, read-before-write against `SHOW DATABASE SETTINGS` (`settings.License`). It runs in `Reconcile` **before** registration and writes its own `LicenseApplied` condition (`LicenseMatchesSecret` / `LicensePending` / `LicenseRejected`), because an expired license is what stops convergence and every registration return overwrites `Converged`. Secret RBAC is `get;list;watch` only: a metadata-only watch (`WatchesMetadata`) maps a Secret to the clusters naming it, and the value is read uncached through `APIReader`; the value is never logged, evented or put in status. A different organization is SET like the license (Andi's call, no special case). A shorter-expiry replacement is assumed away for now. Core masks `enterprise.license` in its query log (Andi's fix), since Memgraph rejects Bolt parameters in the SET grammar. - **AWS credentials** (`spec.awsCredentials.secretName`, decided 2026-10-05) carry all four AWS values from one Secret (Andi's call: region and endpoint too, so they left `spec.flags` and the run-time table) and follow the license on both paths, data instances only (coordinators run no S3 queries). Fixed, all-required keys `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`/`AWS_REGION`/`AWS_ENDPOINT_URL` (AWS SDK names; empty endpoint for AWS itself): required because Kubernetes leaves an unresolvable `$(VAR)` as literal text. Startup: env from `secretKeyRef` into `MEMGRAPH_AWS_*`, handed to the flags by Kubernetes `$(VAR)` expansion, appended after the pinned args; a flag is required because all four settings are `kRestore` (persisted across restarts and restored over Memgraph's own `AWS_ACCESS_KEY` env fallback, but not over an explicit flag), and Memgraph's gflags has no `--fromenv`/`--tryfromenv` (verified on 3.13.0). Live: `reconcileAWSCredentials` SETs `aws.access_key`/`aws.secret_key`/`aws.region`/`aws.endpoint_url` read-before-write on every ready data pod in the converged branch beside the run-time flags, reporting on `Converged` (`SettingsPending`/`SettingsRejected`, setting named, value never); the shared metadata Secret watch maps it too. Removing the block issues no SET, like removing a flag. All four `aws-*` flags are denied in `spec.flags`, `MEMGRAPH_AWS_*` in `extraEnv`. - Storage is never deleted by the operator: no finalizer-based cleanup; PVC retention (deletion *and* scale-down) maps to the StatefulSet PVC retention policy (default `Retain`). The only cluster members the operator removes are the ones a lowered replica count retires; a coordinator removed from Raft keeps running and keeps its state on purpose, which is what makes re-growing onto a retained volume safe. -- Workload pods: non-root uid 101 / gid 103 / fsGroup 103 unless `spec.securityContext` is written (an absent block is schema-defaulted to those three, so the stored object shows the identity; no field inside it has a default), in which case exactly its `runAsUser`/`runAsGroup`/`fsGroup` land and no other (an empty block leaves all three to the platform: OpenShift `restricted-v2` assigns them at admission and rejects named ones; `runAsNonRoot` and seccomp stay unconditional; not pinned, because moving the uid is what the ownership container is for), seccomp RuntimeDefault, all capabilities dropped, `terminationGracePeriodSeconds: 300` (a ceiling, not a delay — an instance killed mid-shutdown recovers from its WAL and lengthens the catch-up a roll waits on). The only privileged containers the operator builds are two init containers that write under `/proc/sys`, both on the cluster's own Memgraph image: `init-sysctl` (`spec.sysctlInitContainer`, the HA chart's block minus its image knobs and its `enabled`: presence-based, absent means off, so the sample and the quickstart example carry it) raises the node's `vm.max_map_count` to `maxMapCount` and only ever raises — the default 524288 is the floor Memgraph itself checks at startup, not the chart's 262144 — and `init-core-pattern` (`coreDumps.configureCorePattern`) sets the core pattern for a role collecting dumps. Both are node properties no namespaced sysctl can set, so a PodSecurity `restricted` namespace must leave the block out and set `configureCorePattern: false`. A third init container, `init-fix-perms` (`spec.fixOwnershipInitContainer`, the HA chart's block minus its image knobs, its `enabled` and its uid/gid knobs; presence-based with no fields, off when absent like the chart), runs last on the same image as root with `CAP_CHOWN` alone, not privileged, and chowns the lib, log and core dumps mounts to the pod's identity (`runAsUser`, and `runAsGroup` or else `fsGroup`) for storage drivers that ignore `fsGroup` (rancher.io/local-path); `restricted` rejects it too since it runs as root, and a spec-level CEL rule rejects it beside a `securityContext` that names no uid or no group, because it cannot chown to an identity the platform assigns later. Every e2e namespace enforces `restricted` (`createClusterNamespace`), so the quickstart carries none of the three and one dedicated scenario in an unlabeled namespace (`test/e2e/init_containers_test.go`) proves all of them. +- Workload pods: non-root uid 101 / gid 103 / fsGroup 103 unless `spec.securityContext` is written (an absent block is schema-defaulted to those three, so the stored object shows the identity; no field inside it has a default), in which case exactly its `runAsUser`/`runAsGroup`/`fsGroup` land and no other (an empty block leaves all three to the platform: OpenShift `restricted-v2` assigns them at admission and rejects named ones; `runAsNonRoot` and seccomp stay unconditional; not pinned, because moving the uid is what the ownership container is for), seccomp RuntimeDefault, all capabilities dropped, `terminationGracePeriodSeconds: 300` (a ceiling, not a delay — an instance killed mid-shutdown recovers from its WAL and lengthens the catch-up a roll waits on). The only privileged containers the operator builds are two init containers that write under `/proc/sys`, both on the cluster's own Memgraph image: `init-sysctl` (`spec.sysctlInitContainer`, the HA chart's block minus its image knobs and its `enabled`: presence-based, absent means off, so the sample and the quickstart example carry it) raises the node's `vm.max_map_count` to `maxMapCount` and only ever raises — the default 524288 is the floor Memgraph itself checks at startup, not the chart's 262144 — and `init-core-pattern` (`coreDumps.corePattern: Operator`) sets the core pattern for a role collecting dumps. Both are node properties no namespaced sysctl can set, so a PodSecurity `restricted` namespace must leave the block out and set `corePattern: Node`. A third init container, `init-fix-perms` (`spec.fixOwnershipInitContainer`, the HA chart's block minus its image knobs, its `enabled` and its uid/gid knobs; presence-based with no fields, off when absent like the chart), runs last on the same image as root with `CAP_CHOWN` alone, not privileged, and chowns the lib, log and core dumps mounts to the pod's identity (`runAsUser`, and `runAsGroup` or else `fsGroup`) for storage drivers that ignore `fsGroup` (rancher.io/local-path); `restricted` rejects it too since it runs as root, and a spec-level CEL rule rejects it beside a `securityContext` that names no uid or no group, because it cannot chown to an identity the platform assigns later. Every e2e namespace enforces `restricted` (`createClusterNamespace`), so the quickstart carries none of the three and one dedicated scenario in an unlabeled namespace (`test/e2e/init_containers_test.go`) proves all of them. - **Readiness probe only, no liveness or startup probe**, on both roles. Memgraph recovers every database inside the `DbmsHandler` constructor before any server starts, so during recovery nothing listens and no probe can tell slow from hung; a TCP liveness check could only kill a recovery that outlived a guessed budget (the HA chart's 2h), and a Bolt listener that disappears after startup means the process exited, which restarts the container anyway. A recovering pod is simply unready. The field converged on the same answer (ECK, CockroachDB, MongoDB Community run no liveness; Strimzi and cass-operator make it a process check; Percona deprecated its 7200s startup delay). Do not reintroduce them; a core health endpoint reporting recovery *progress* is the only thing that would justify a liveness check. - StatefulSet `volumeClaimTemplates` cannot be changed, added or removed on a live cluster. **Sizes grow** (`storage..libPVCSize`, default 10Gi, `coreDumps..size` while the role collects dumps; CEL lets them grow, never shrink, compared as quantities; decided 2026-10-05): `internal/volumes` (the fourth pure core) patches every claim carrying the role's selector labels, retained ones included, then once none is left to patch orphan-deletes the StatefulSet (`propagationPolicy: Orphan`) and ends the pass; the next apply recreates it with the grown templates. That restarts nothing because the StatefulSet controller adopts a pod when its volumes name the claims its ordinal gets (`storageMatches`: names, never sizes) and the revision hashes `spec.template` alone — proven on a real StatefulSet controller in Kind. Until then every apply restates the live sizes (`volumes.KeepLiveSizes`), a refused claim patch (StorageClass without `allowVolumeExpansion`; no StorageClass pre-check, no RBAC on it) is `Converged=False/VolumeExpansionRefused` and blocks the recreate, a claim a pod mounts below its request is `VolumeExpansionInProgress` naming claim and pod plus the driver's first `ControllerResizeError`/`NodeResizeError` message, tail kept when it overflows the 1024 cap (found on AKS: `Standard_A2_v2` nodes cannot change an attached disk), an unmounted one counts once only its filesystem is left, `*ResizeInfeasible` is `VolumeExpansionFailed`. While a StatefulSet is missing `replicaCounts` reads its count off the highest pod ordinal + 1, so a count lowered in the window still retires members first. **No pod is ever restarted for a resize** (Andi's call after the Kind proof): `FileSystemResizePending` is a transient state on every online driver (~1 min until the kubelet's sync), and an offline-only driver waits until no pod uses the claim, which a restart does not give it (`docs/storage-resize.md` lists them). Everything else is pinned by a CEL transition rule on the type that carries it: `storage..{libStorageAccessMode,libStorageClassName}`, the presence of `coreDumps.` (pinned on `CoreDumpsSpec`, because a rule on a block never fires for the block being added or removed whole) and `coreDumps.storageClassName` while a role collects dumps: adding or removing a template changes the pods' volumes, adoption fails, and the StatefulSet controller only recovers adopted pods in ascending ordinal order, MAIN first, which deadlocks the roll. The doc comment on `RoleStorageSpec` carries the argument; read it before touching any claim template knob. The e2e resize scenario runs on the CSI hostpath driver's standard (single-node) deployment, `hack/kind-csi-hostpath.sh`: the distributed one has no resizer. -- **Logs live on the lib claim** (decided 2026-10-07; the separate `log-storage` claim and its four `log*` knobs are gone): `storage..fileLogging` (bool, default `true`, a default-on toggle like `configureCorePattern`) pins `--log-file=/var/lib/memgraph/logs/memgraph.log`, off pins `--log-file=""`. Never a path knob and `log_file` stays denied in `spec.flags`: Memgraph's `CleanLogsDir` deletes files older than `log_retention_days` from the log file's **parent directory**, so the file must sit in a directory of its own, never the claim root, and any path off a writable mount crash-loops on the read-only root. The empty value is load-bearing: the image's `memgraph.conf` sets `log_file` to `/var/log/memgraph/memgraph.log`. Toggling is an ordinary roll (no CEL pin, no claim template involved); turning it off leaves old log files on the claim, never cleaned by the operator. Logs count toward `libPVCSize`, which is why the defaults moved to 10Gi and `log_level=INFO`. +- **Logs live on the lib claim** (decided 2026-10-07; the separate `log-storage` claim and its four `log*` knobs are gone): `storage..logFile` (`LibClaim | None`, default `LibClaim`; a bool until 2026-10-14) pins `--log-file=/var/lib/memgraph/logs/memgraph.log`, `None` pins `--log-file=""`. Never a path knob and `log_file` stays denied in `spec.flags`: Memgraph's `CleanLogsDir` deletes files older than `log_retention_days` from the log file's **parent directory**, so the file must sit in a directory of its own, never the claim root, and any path off a writable mount crash-loops on the read-only root. The empty value is load-bearing: the image's `memgraph.conf` sets `log_file` to `/var/log/memgraph/memgraph.log`. Changing it is an ordinary roll (no CEL pin, no claim template involved); moving to `None` leaves old log files on the claim, never cleaned by the operator. Logs count toward `libPVCSize`, which is why the defaults moved to 10Gi and `log_level=INFO`. - Both StatefulSets use `updateStrategy: OnDelete`, so **nothing but the operator ever restarts a workload pod**. A pod-template change no reconcile acts on takes effect never, which is what the `Updated` condition exists to report. A run-time flag change is deliberately *not* a pod-template change (see Memgraph flags above). `RollingUpdate` cannot express the required order (it sweeps highest ordinal to lowest, and `partition` is a descending cutoff, not a set), so a MAIN on any ordinal but 0 would be restarted mid-sweep and each such restart buys another failover. - The operator never promotes a MAIN outside bootstrap and the scale-down handover: a roll deletes the MAIN's pod and lets the Raft coordinators promote. That is only safe on a Memgraph reporting an unreachable MAIN as `role=main, health=down` — a release that vacates the `main` row makes `planner.Plan` believe there is no MAIN and race the failover. - **The shrink is the one write with an expectation** (`internal/expectations`, decided 2026-10-09). The operator reads its StatefulSets from the informer cache, and `replicaCounts` applies `max(declared, cached spec.replicas)`, so a pass reading the StatefulSet from before `shedRetiredPods` lowered it would grow it back (and shrink it again once it found the member gone, the pod recreated in between). `shedRetiredPods` records the UID and generation the apply returned; `replicaCounts` marks a role whose cached StatefulSet is the same UID at a lower generation, and the pass ends before applying anything, waiting on the StatefulSet event (`Owns`). It is in-memory on purpose: a restarted operator lists into a fresh cache that already holds every write it made, so only the writing process can be ahead of its cache. Every other cached read is safe without one (the apply restates desired state, pod deletes carry a UID precondition, `rollout`'s `Stale` covers status, membership is read live over Bolt); a new write whose stale read would undo it, or a decision pushed into Memgraph from a cached count, needs one too. diff --git a/README.md b/README.md index f1ccece..27e9400 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,7 @@ OpenShift's default `restricted-v2` SCC assigns every pod's `runAsUser` and `fsG - On the cluster, write `securityContext: {}` so the pods name no uid, gid or fsGroup and take the assigned ones. Memgraph needs no particular uid, only to own the data directory it creates, and OpenShift's storage honors `fsGroup`, so `fixOwnershipInitContainer` is neither needed nor admitted there (it runs as root). - On the operator chart, drop the manager's own uid and gid: `--set podSecurityContext.runAsUser=null --set podSecurityContext.runAsGroup=null`. -`sysctlInitContainer` and `coreDumps.configureCorePattern` need privileged containers, which `restricted-v2` forbids as well; leave the first out and set the second to `false`. +`sysctlInitContainer` and `coreDumps.corePattern: Operator` need privileged containers, which `restricted-v2` forbids as well; leave the first out and set the second to `Node`. ## External access diff --git a/api/v1alpha1/memgraphcluster_types.go b/api/v1alpha1/memgraphcluster_types.go index 830e76f..cf66757 100644 --- a/api/v1alpha1/memgraphcluster_types.go +++ b/api/v1alpha1/memgraphcluster_types.go @@ -52,8 +52,8 @@ const ( AWSRegionSecretKey = "AWS_REGION" AWSEndpointURLSecretKey = "AWS_ENDPOINT_URL" - DefaultCoreDumpsSize = "10Gi" - DefaultConfigureCorePattern = true + DefaultCoreDumpsSize = "10Gi" + DefaultCorePattern = CorePatternOperator DefaultMaxMapCount int64 = 524288 @@ -64,7 +64,7 @@ const ( DefaultFSGroup int64 = 103 DefaultLibPVCSize = "10Gi" - DefaultFileLogging = true + DefaultLogFile = LogFileLibClaim DefaultStorageAccessMode = corev1.ReadWriteOnce DefaultStorageRetention = RetentionPolicyRetain @@ -179,6 +179,39 @@ const ( RetentionPolicyDelete StorageRetentionPolicy = "Delete" ) +// LogFileLocation is where every pod of a role writes Memgraph's log files. +// It is an enum rather than a switch because the answers have changed before: +// the log files had a claim of their own until they moved onto the lib claim. +// +kubebuilder:validation:Enum=LibClaim;None +type LogFileLocation string + +const ( + // LogFileLibClaim writes the log files to /var/lib/memgraph/logs on the lib + // storage claim, so they survive the pod and count toward libPVCSize. + LogFileLibClaim LogFileLocation = "LibClaim" + + // LogFileNone writes no log file: stderr, and whatever collects it, is the + // only log sink. + LogFileNone LogFileLocation = "None" +) + +// CorePatternOwner is who points the node's kernel.core_pattern at the core +// dumps volume. It is an enum rather than a switch because who sets a node +// property is a question with more answers than two, a per-node DaemonSet +// among them. +// +kubebuilder:validation:Enum=Operator;Node +type CorePatternOwner string + +const ( + // CorePatternOperator has the operator set it, from a privileged init + // container in every pod of a role that collects dumps. + CorePatternOperator CorePatternOwner = "Operator" + + // CorePatternNode leaves it to the node or the platform: the operator only + // provisions and mounts the volumes, trusting the node to point at them. + CorePatternNode CorePatternOwner = "Node" +) + // Condition types reported on MemgraphCluster status. All four use // normal-True polarity: True is the healthy state. Converged, Updated and // LicenseApplied each answer one question; Ready is their summary, the one @@ -557,8 +590,8 @@ type AWSCredentialsSpec struct { // claim template changes the pods' volumes, which the StatefulSet controller // only repairs in ascending ordinal order, MAIN first, which deadlocks the // rolling restart. A size given in other units is not a change: quantities are -// compared as quantities. fileLogging is not part of a claim template and -// changes freely. +// compared as quantities. logFile is not part of a claim template and changes +// freely. // // The has() guards keep every rule evaluable against the block's empty object // default, which the API server checks before nested field defaults apply. @@ -569,8 +602,8 @@ type AWSCredentialsSpec struct { type RoleStorageSpec struct { // libPVCSize is the requested size of the lib storage claim, which backs // Memgraph's data directory (snapshots, WAL, and durability metadata). It - // can grow on a live cluster and never shrink. With fileLogging on it holds - // the role's log files too. + // can grow on a live cluster and never shrink. With logFile LibClaim it + // holds the role's log files too. // +kubebuilder:validation:XValidation:rule="quantity(string(self)).isGreaterThan(quantity('0'))",message="libPVCSize must be greater than zero" // +kubebuilder:default="10Gi" // +optional @@ -591,22 +624,22 @@ type RoleStorageSpec struct { // +optional LibStorageClassName *string `json:"libStorageClassName,omitempty"` - // fileLogging decides whether Memgraph writes log files. With it on, every - // pod of the role logs to /var/lib/memgraph/logs/memgraph_.log on the - // lib storage claim, so the logs survive the pod and count toward - // libPVCSize: size the claim for snapshots, WAL and log-retention-days of - // logs, or lower log-level or log-retention-days in flags. With it off the - // operator passes an empty --log-file, so stderr and `kubectl logs` (plus - // whatever collects them, such as monitoring.vectorRemote) are the only - // log sink. + // logFile is where Memgraph writes its log files. LibClaim, the default, + // has every pod of the role log to + // /var/lib/memgraph/logs/memgraph_.log on the lib storage claim, so + // the logs survive the pod and count toward libPVCSize: size the claim for + // snapshots, WAL and log-retention-days of logs, or lower log-level or + // log-retention-days in flags. None has the operator pass an empty + // --log-file, so stderr and `kubectl logs` (plus whatever collects them, + // such as monitoring.vectorRemote) are the only log sink. // - // It changes on a live cluster with a rolling restart. Turning it off + // It changes on a live cluster with a rolling restart. Moving to None // leaves the log files already written where they are: Memgraph only // prunes old log files while it writes them, and the operator never // deletes storage. - // +kubebuilder:default=true + // +kubebuilder:default=LibClaim // +optional - FileLogging *bool `json:"fileLogging,omitempty"` + LogFile LogFileLocation `json:"logFile,omitempty"` } // StorageSpec configures persistence for both roles plus what happens to a @@ -782,21 +815,22 @@ type CoreDumpsSpec struct { // +optional StorageClassName *string `json:"storageClassName,omitempty"` - // configureCorePattern lets the operator point the kernel at - // /var/core/memgraph by running a privileged init container that writes - // /proc/sys/kernel/core_pattern. It uses the cluster's own Memgraph image, - // so no second image has to be pulled. + // corePattern is who points the kernel at /var/core/memgraph. Operator, + // the default, has the operator run a privileged init container that + // writes /proc/sys/kernel/core_pattern, on the cluster's own Memgraph + // image so no second image has to be pulled. Node leaves it to the node + // or the platform: the operator only provisions and mounts the volumes, + // trusting the node to already point at them. // // It is cluster-wide rather than per role for two reasons: core_pattern is a // property of the **node**, so it applies to every process that crashes // there regardless of which role asked for it, and what really decides this // is whether the namespace tolerates a privileged container at all. - // PodSecurity "restricted" does not — set this to false there, or wherever - // the platform manages core_pattern itself, and the operator only provisions - // and mounts the volumes, trusting the node to already point at them. - // +kubebuilder:default=true + // PodSecurity "restricted" does not — set Node there, or wherever the + // platform manages core_pattern itself. + // +kubebuilder:default=Operator // +optional - ConfigureCorePattern *bool `json:"configureCorePattern,omitempty"` + CorePattern CorePatternOwner `json:"corePattern,omitempty"` // uploader is an optional sidecar that ships collected dumps off the volume // — to object storage, a debug host, wherever. Any image and destination diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index edeb2b2..af34767 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -89,11 +89,6 @@ func (in *CoreDumpsSpec) DeepCopyInto(out *CoreDumpsSpec) { *out = new(string) **out = **in } - if in.ConfigureCorePattern != nil { - in, out := &in.ConfigureCorePattern, &out.ConfigureCorePattern - *out = new(bool) - **out = **in - } if in.Uploader != nil { in, out := &in.Uploader, &out.Uploader *out = new(CoreDumpsUploaderSpec) @@ -927,11 +922,6 @@ func (in *RoleStorageSpec) DeepCopyInto(out *RoleStorageSpec) { *out = new(string) **out = **in } - if in.FileLogging != nil { - in, out := &in.FileLogging, &out.FileLogging - *out = new(bool) - **out = **in - } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RoleStorageSpec. diff --git a/charts/memgraph-operator/Chart.yaml b/charts/memgraph-operator/Chart.yaml index 6b8a32d..75ae856 100644 --- a/charts/memgraph-operator/Chart.yaml +++ b/charts/memgraph-operator/Chart.yaml @@ -4,7 +4,7 @@ description: Kubernetes operator for Memgraph high-availability clusters type: application # version is the chart version, appVersion the operator version it installs. # Both move together with the git tag at release time. -version: 1.0.1-rc.2 +version: 1.0.1-rc.3 appVersion: "1.0.1-rc.1" # The CustomResourceDefinition uses CEL validation rules # (x-kubernetes-validations), available from Kubernetes 1.25. diff --git a/charts/memgraph-operator/crds/memgraph.com_memgraphclusters.yaml b/charts/memgraph-operator/crds/memgraph.com_memgraphclusters.yaml index e718ff0..462ccce 100644 --- a/charts/memgraph-operator/crds/memgraph.com_memgraphclusters.yaml +++ b/charts/memgraph-operator/crds/memgraph.com_memgraphclusters.yaml @@ -122,22 +122,6 @@ spec: coreDumps collects crash dumps of the roles whose blocks are present onto a volume of their own. properties: - configureCorePattern: - default: true - description: |- - configureCorePattern lets the operator point the kernel at - /var/core/memgraph by running a privileged init container that writes - /proc/sys/kernel/core_pattern. It uses the cluster's own Memgraph image, - so no second image has to be pulled. - - It is cluster-wide rather than per role for two reasons: core_pattern is a - property of the **node**, so it applies to every process that crashes - there regardless of which role asked for it, and what really decides this - is whether the namespace tolerates a privileged container at all. - PodSecurity "restricted" does not — set this to false there, or wherever - the platform manages core_pattern itself, and the operator only provisions - and mounts the volumes, trusting the node to already point at them. - type: boolean coordinators: description: |- coordinators, when present, makes every coordinator pod collect dumps @@ -170,6 +154,26 @@ spec: rule: has(self.size) == has(oldSelf.size) && (!has(self.size) || quantity(string(self.size)).compareTo(quantity(string(oldSelf.size))) >= 0) + corePattern: + default: Operator + description: |- + corePattern is who points the kernel at /var/core/memgraph. Operator, + the default, has the operator run a privileged init container that + writes /proc/sys/kernel/core_pattern, on the cluster's own Memgraph + image so no second image has to be pulled. Node leaves it to the node + or the platform: the operator only provisions and mounts the volumes, + trusting the node to already point at them. + + It is cluster-wide rather than per role for two reasons: core_pattern is a + property of the **node**, so it applies to every process that crashes + there regardless of which role asked for it, and what really decides this + is whether the namespace tolerates a privileged container at all. + PodSecurity "restricted" does not — set Node there, or wherever the + platform manages core_pattern itself. + enum: + - Operator + - Node + type: string data: description: |- data, when present, makes every data instance pod collect dumps onto a @@ -2842,23 +2846,6 @@ spec: description: coordinators configures the storage of every coordinator pod. properties: - fileLogging: - default: true - description: |- - fileLogging decides whether Memgraph writes log files. With it on, every - pod of the role logs to /var/lib/memgraph/logs/memgraph_.log on the - lib storage claim, so the logs survive the pod and count toward - libPVCSize: size the claim for snapshots, WAL and log-retention-days of - logs, or lower log-level or log-retention-days in flags. With it off the - operator passes an empty --log-file, so stderr and `kubectl logs` (plus - whatever collects them, such as monitoring.vectorRemote) are the only - log sink. - - It changes on a live cluster with a rolling restart. Turning it off - leaves the log files already written where they are: Memgraph only - prunes old log files while it writes them, and the operator never - deletes storage. - type: boolean libPVCSize: anyOf: - type: integer @@ -2867,8 +2854,8 @@ spec: description: |- libPVCSize is the requested size of the lib storage claim, which backs Memgraph's data directory (snapshots, WAL, and durability metadata). It - can grow on a live cluster and never shrink. With fileLogging on it holds - the role's log files too. + can grow on a live cluster and never shrink. With logFile LibClaim it + holds the role's log files too. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: @@ -2893,6 +2880,26 @@ spec: PersistentVolume. maxLength: 253 type: string + logFile: + default: LibClaim + description: |- + logFile is where Memgraph writes its log files. LibClaim, the default, + has every pod of the role log to + /var/lib/memgraph/logs/memgraph_.log on the lib storage claim, so + the logs survive the pod and count toward libPVCSize: size the claim for + snapshots, WAL and log-retention-days of logs, or lower log-level or + log-retention-days in flags. None has the operator pass an empty + --log-file, so stderr and `kubectl logs` (plus whatever collects them, + such as monitoring.vectorRemote) are the only log sink. + + It changes on a live cluster with a rolling restart. Moving to None + leaves the log files already written where they are: Memgraph only + prunes old log files while it writes them, and the operator never + deletes storage. + enum: + - LibClaim + - None + type: string type: object x-kubernetes-validations: - message: 'libStorageClassName cannot be changed on a live cluster: @@ -2921,23 +2928,6 @@ spec: description: data configures the storage of every data instance pod. properties: - fileLogging: - default: true - description: |- - fileLogging decides whether Memgraph writes log files. With it on, every - pod of the role logs to /var/lib/memgraph/logs/memgraph_.log on the - lib storage claim, so the logs survive the pod and count toward - libPVCSize: size the claim for snapshots, WAL and log-retention-days of - logs, or lower log-level or log-retention-days in flags. With it off the - operator passes an empty --log-file, so stderr and `kubectl logs` (plus - whatever collects them, such as monitoring.vectorRemote) are the only - log sink. - - It changes on a live cluster with a rolling restart. Turning it off - leaves the log files already written where they are: Memgraph only - prunes old log files while it writes them, and the operator never - deletes storage. - type: boolean libPVCSize: anyOf: - type: integer @@ -2946,8 +2936,8 @@ spec: description: |- libPVCSize is the requested size of the lib storage claim, which backs Memgraph's data directory (snapshots, WAL, and durability metadata). It - can grow on a live cluster and never shrink. With fileLogging on it holds - the role's log files too. + can grow on a live cluster and never shrink. With logFile LibClaim it + holds the role's log files too. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: @@ -2972,6 +2962,26 @@ spec: PersistentVolume. maxLength: 253 type: string + logFile: + default: LibClaim + description: |- + logFile is where Memgraph writes its log files. LibClaim, the default, + has every pod of the role log to + /var/lib/memgraph/logs/memgraph_.log on the lib storage claim, so + the logs survive the pod and count toward libPVCSize: size the claim for + snapshots, WAL and log-retention-days of logs, or lower log-level or + log-retention-days in flags. None has the operator pass an empty + --log-file, so stderr and `kubectl logs` (plus whatever collects them, + such as monitoring.vectorRemote) are the only log sink. + + It changes on a live cluster with a rolling restart. Moving to None + leaves the log files already written where they are: Memgraph only + prunes old log files while it writes them, and the operator never + deletes storage. + enum: + - LibClaim + - None + type: string type: object x-kubernetes-validations: - message: 'libStorageClassName cannot be changed on a live cluster: diff --git a/config/crd/bases/memgraph.com_memgraphclusters.yaml b/config/crd/bases/memgraph.com_memgraphclusters.yaml index f12a86d..2da0312 100644 --- a/config/crd/bases/memgraph.com_memgraphclusters.yaml +++ b/config/crd/bases/memgraph.com_memgraphclusters.yaml @@ -120,22 +120,6 @@ spec: coreDumps collects crash dumps of the roles whose blocks are present onto a volume of their own. properties: - configureCorePattern: - default: true - description: |- - configureCorePattern lets the operator point the kernel at - /var/core/memgraph by running a privileged init container that writes - /proc/sys/kernel/core_pattern. It uses the cluster's own Memgraph image, - so no second image has to be pulled. - - It is cluster-wide rather than per role for two reasons: core_pattern is a - property of the **node**, so it applies to every process that crashes - there regardless of which role asked for it, and what really decides this - is whether the namespace tolerates a privileged container at all. - PodSecurity "restricted" does not — set this to false there, or wherever - the platform manages core_pattern itself, and the operator only provisions - and mounts the volumes, trusting the node to already point at them. - type: boolean coordinators: description: |- coordinators, when present, makes every coordinator pod collect dumps @@ -168,6 +152,26 @@ spec: rule: has(self.size) == has(oldSelf.size) && (!has(self.size) || quantity(string(self.size)).compareTo(quantity(string(oldSelf.size))) >= 0) + corePattern: + default: Operator + description: |- + corePattern is who points the kernel at /var/core/memgraph. Operator, + the default, has the operator run a privileged init container that + writes /proc/sys/kernel/core_pattern, on the cluster's own Memgraph + image so no second image has to be pulled. Node leaves it to the node + or the platform: the operator only provisions and mounts the volumes, + trusting the node to already point at them. + + It is cluster-wide rather than per role for two reasons: core_pattern is a + property of the **node**, so it applies to every process that crashes + there regardless of which role asked for it, and what really decides this + is whether the namespace tolerates a privileged container at all. + PodSecurity "restricted" does not — set Node there, or wherever the + platform manages core_pattern itself. + enum: + - Operator + - Node + type: string data: description: |- data, when present, makes every data instance pod collect dumps onto a @@ -2840,23 +2844,6 @@ spec: description: coordinators configures the storage of every coordinator pod. properties: - fileLogging: - default: true - description: |- - fileLogging decides whether Memgraph writes log files. With it on, every - pod of the role logs to /var/lib/memgraph/logs/memgraph_.log on the - lib storage claim, so the logs survive the pod and count toward - libPVCSize: size the claim for snapshots, WAL and log-retention-days of - logs, or lower log-level or log-retention-days in flags. With it off the - operator passes an empty --log-file, so stderr and `kubectl logs` (plus - whatever collects them, such as monitoring.vectorRemote) are the only - log sink. - - It changes on a live cluster with a rolling restart. Turning it off - leaves the log files already written where they are: Memgraph only - prunes old log files while it writes them, and the operator never - deletes storage. - type: boolean libPVCSize: anyOf: - type: integer @@ -2865,8 +2852,8 @@ spec: description: |- libPVCSize is the requested size of the lib storage claim, which backs Memgraph's data directory (snapshots, WAL, and durability metadata). It - can grow on a live cluster and never shrink. With fileLogging on it holds - the role's log files too. + can grow on a live cluster and never shrink. With logFile LibClaim it + holds the role's log files too. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: @@ -2891,6 +2878,26 @@ spec: PersistentVolume. maxLength: 253 type: string + logFile: + default: LibClaim + description: |- + logFile is where Memgraph writes its log files. LibClaim, the default, + has every pod of the role log to + /var/lib/memgraph/logs/memgraph_.log on the lib storage claim, so + the logs survive the pod and count toward libPVCSize: size the claim for + snapshots, WAL and log-retention-days of logs, or lower log-level or + log-retention-days in flags. None has the operator pass an empty + --log-file, so stderr and `kubectl logs` (plus whatever collects them, + such as monitoring.vectorRemote) are the only log sink. + + It changes on a live cluster with a rolling restart. Moving to None + leaves the log files already written where they are: Memgraph only + prunes old log files while it writes them, and the operator never + deletes storage. + enum: + - LibClaim + - None + type: string type: object x-kubernetes-validations: - message: 'libStorageClassName cannot be changed on a live cluster: @@ -2919,23 +2926,6 @@ spec: description: data configures the storage of every data instance pod. properties: - fileLogging: - default: true - description: |- - fileLogging decides whether Memgraph writes log files. With it on, every - pod of the role logs to /var/lib/memgraph/logs/memgraph_.log on the - lib storage claim, so the logs survive the pod and count toward - libPVCSize: size the claim for snapshots, WAL and log-retention-days of - logs, or lower log-level or log-retention-days in flags. With it off the - operator passes an empty --log-file, so stderr and `kubectl logs` (plus - whatever collects them, such as monitoring.vectorRemote) are the only - log sink. - - It changes on a live cluster with a rolling restart. Turning it off - leaves the log files already written where they are: Memgraph only - prunes old log files while it writes them, and the operator never - deletes storage. - type: boolean libPVCSize: anyOf: - type: integer @@ -2944,8 +2934,8 @@ spec: description: |- libPVCSize is the requested size of the lib storage claim, which backs Memgraph's data directory (snapshots, WAL, and durability metadata). It - can grow on a live cluster and never shrink. With fileLogging on it holds - the role's log files too. + can grow on a live cluster and never shrink. With logFile LibClaim it + holds the role's log files too. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: @@ -2970,6 +2960,26 @@ spec: PersistentVolume. maxLength: 253 type: string + logFile: + default: LibClaim + description: |- + logFile is where Memgraph writes its log files. LibClaim, the default, + has every pod of the role log to + /var/lib/memgraph/logs/memgraph_.log on the lib storage claim, so + the logs survive the pod and count toward libPVCSize: size the claim for + snapshots, WAL and log-retention-days of logs, or lower log-level or + log-retention-days in flags. None has the operator pass an empty + --log-file, so stderr and `kubectl logs` (plus whatever collects them, + such as monitoring.vectorRemote) are the only log sink. + + It changes on a live cluster with a rolling restart. Moving to None + leaves the log files already written where they are: Memgraph only + prunes old log files while it writes them, and the operator never + deletes storage. + enum: + - LibClaim + - None + type: string type: object x-kubernetes-validations: - message: 'libStorageClassName cannot be changed on a live cluster: diff --git a/config/samples/v1alpha1_memgraphcluster.yaml b/config/samples/v1alpha1_memgraphcluster.yaml index e9475e8..41660d4 100644 --- a/config/samples/v1alpha1_memgraphcluster.yaml +++ b/config/samples/v1alpha1_memgraphcluster.yaml @@ -42,7 +42,7 @@ spec: # after themselves. retentionPolicy: Retain # Each pod of a role gets a lib claim: Memgraph's data directory and, with - # fileLogging on, its log files under /var/lib/memgraph/logs. Size it for + # logFile LibClaim, its log files under /var/lib/memgraph/logs. Size it for # snapshots, WAL and log-retention-days of logs (see flags below). The # storage class names have no default and are commented out on purpose: # left unset, each claim uses the cluster's default StorageClass, while the @@ -53,30 +53,29 @@ spec: # StatefulSet volumeClaimTemplates, which Kubernetes treats as immutable, so # admission refuses changing them on a live cluster: delete the # MemgraphCluster (the claims are retained) and recreate it with the new - # values. Setting fileLogging to false passes an empty --log-file, leaving - # stderr (and so `kubectl logs`) as the only log sink; it changes on a live - # cluster with a rolling restart, and turning it off leaves the log files - # already written in place. + # values. Setting logFile to None passes an empty --log-file, leaving stderr + # (and so `kubectl logs`) as the only log sink; it changes on a live cluster + # with a rolling restart, and moving to None leaves the log files already + # written in place. coordinators: libPVCSize: 10Gi libStorageAccessMode: ReadWriteOnce # libStorageClassName: standard - fileLogging: true + logFile: LibClaim data: libPVCSize: 10Gi libStorageAccessMode: ReadWriteOnce # libStorageClassName: standard - fileLogging: true + logFile: LibClaim # A role collects core dumps when its block below is present. That provisions # a second claim per pod, mounts it at /var/core/memgraph — a fixed path, since # only the kernel writes it and only the uploader reads it, and the operator - # points both at it — and, unless configureCorePattern is false, runs a + # points both at it — and, unless corePattern is Node, runs a # privileged init container that sets the node's core pattern. Two things that # catch people: core_pattern is a property of the node, so it applies to # everything that crashes there, and the privileged init container is rejected - # in a namespace running PodSecurity "restricted". Set configureCorePattern: - # false there and have the platform point core_pattern at that directory - # itself. Whether a role collects dumps is decided when the cluster is + # in a namespace running PodSecurity "restricted". Set corePattern: Node + # there and have the platform point core_pattern at that directory itself. Whether a role collects dumps is decided when the cluster is # created: the claim is a StatefulSet volumeClaimTemplate, which Kubernetes # forbids adding or removing in place, so adding or removing a role's block on # a live cluster is rejected. To change it, delete the MemgraphCluster (the @@ -93,7 +92,7 @@ spec: # size: 10Gi # The rest is one decision for the cluster, so it is stated once. # storageClassName: standard - # configureCorePattern: true + # corePattern: Operator # An optional sidecar that ships the dumps somewhere durable, joining the # pods of every role that collects them. The operator mounts the dumps into # it read-only, passes the directory as CORE_DUMPS_DIR, mounts a writable @@ -136,7 +135,7 @@ spec: # and the value its docs recommend up to 64 GB of RAM; larger nodes want about # one map area per 128 KB of memory. Unlike the chart, the block is off when # absent: leave it out in a namespace running PodSecurity "restricted" (as - # with configureCorePattern above) or on nodes the platform tunes itself. + # with corePattern above) or on nodes the platform tunes itself. sysctlInitContainer: maxMapCount: 524288 # The identity every pod of both roles runs under, the HA chart's diff --git a/docs/backup-restore.md b/docs/backup-restore.md index 7a491f6..77bcf94 100644 --- a/docs/backup-restore.md +++ b/docs/backup-restore.md @@ -22,7 +22,7 @@ Three properties of the operator make restore possible, and all three are delibe ## Taking a backup -Back up the `lib-storage` claims; the core-dump claims hold nothing a restore needs. With `storage..fileLogging` on (the default) a `lib-storage` claim also holds Memgraph's log files under `logs/`, which ride along in the snapshot and are harmless on restore. Two things distinguish a database cluster from a stateless workload: +Back up the `lib-storage` claims; the core-dump claims hold nothing a restore needs. With `storage..logFile: LibClaim` (the default) a `lib-storage` claim also holds Memgraph's log files under `logs/`, which ride along in the snapshot and are harmless on restore. Two things distinguish a database cluster from a stateless workload: - **Consistency across members.** Snapshot all `lib-storage` claims of the cluster as one group, at one instant, so the coordinators' view of the cluster and the data instances' storage agree. With Stork that is a `GroupVolumeSnapshot` selecting `app.kubernetes.io/instance=`; Velero and Kasten have their own grouping. Snapshots taken one claim at a time, seconds apart, can disagree about which instance was MAIN and how far each replica had caught up. - **Application consistency.** A snapshot is crash-consistent: Memgraph recovers from it as it would from a power loss, replaying the WAL since its last own snapshot. Running `CREATE SNAPSHOT;` on the MAIN just before the volume snapshot (a Stork pre-exec `Rule`, a Velero pre-hook) shortens that replay to seconds. It is an optimization, not a requirement. diff --git a/docs/flags.md b/docs/flags.md index 7abd2c4..0cac63e 100644 --- a/docs/flags.md +++ b/docs/flags.md @@ -133,7 +133,7 @@ Admission rejects these keys, in either spelling: - **`bolt-port`, `management-port`, `coordinator-port`, `bolt-address`, `monitoring-address`, `monitoring-port`, `metrics-port`**: the Services, probes, registered addresses, metrics scrapers and the Vector sidecar reach every instance at the fixed ports and on every interface. - **`coordinator-id`, `coordinator-hostname`**: a coordinator's identity, derived from its pod ordinal. -- **`data-directory`, `log-file`**: both on the lib claim. `log-file` is `/var/lib/memgraph/logs/memgraph.log`, in a directory of its own because Memgraph prunes files older than `log-retention-days` from the log file's directory, and empty when `storage..fileLogging` is false, which is what keeps the image's own `log_file` default from crash-looping a read-only root filesystem. The log files count toward `libPVCSize`. +- **`data-directory`, `log-file`**: both on the lib claim. `log-file` is `/var/lib/memgraph/logs/memgraph.log`, in a directory of its own because Memgraph prunes files older than `log-retention-days` from the log file's directory, and empty when `storage..logFile` is `None`, which is what keeps the image's own `log_file` default from crash-looping a read-only root filesystem. The log files count toward `libPVCSize`. - **`bolt-cert-file`, `bolt-key-file`, `cluster-cert-file`, `cluster-key-file`, `cluster-ca-file`**: derived from `spec.tls`, see [TLS](tls.md). - **`metrics-format`**: always OpenMetrics, see [monitoring](monitoring.md). - **`license-key`, `organization-name`**: the CR carries no secret material; the license comes from the `secrets` block. diff --git a/docs/rolling-restart-pacing.md b/docs/rolling-restart-pacing.md index 9501288..9d0fd2d 100644 --- a/docs/rolling-restart-pacing.md +++ b/docs/rolling-restart-pacing.md @@ -111,7 +111,7 @@ the signature; the operator's default is INFO, so set `log-level: TRACE` in `spe StatefulSet's `status.updateRevision`; `kubectl get controllerrevisions` shows the args each revision carried and when it was created. 4. Memgraph's own logs persist on the `lib-storage` claim at - `/var/lib/memgraph/logs/memgraph_.log` (with `fileLogging` on, the default), so they + `/var/lib/memgraph/logs/memgraph_.log` (with `logFile: LibClaim`, the default), so they survive the pod being recreated. Look there for `Name or service not known` (DNS), `Replica up to date` (the catch-up the roll waits for), and `PromoteToMainReq` / `DemoteMainToReplicaReq` (coordinator-driven failovers). diff --git a/internal/controller/memgraphcluster_validation_test.go b/internal/controller/memgraphcluster_validation_test.go index 76bb591..e373bd9 100644 --- a/internal/controller/memgraphcluster_validation_test.go +++ b/internal/controller/memgraphcluster_validation_test.go @@ -49,7 +49,7 @@ func defaultRoleStorage() memgraphcomv1alpha1.RoleStorageSpec { return memgraphcomv1alpha1.RoleStorageSpec{ LibPVCSize: ptr.To(resource.MustParse(memgraphcomv1alpha1.DefaultLibPVCSize)), LibStorageAccessMode: memgraphcomv1alpha1.DefaultStorageAccessMode, - FileLogging: ptr.To(memgraphcomv1alpha1.DefaultFileLogging), + LogFile: memgraphcomv1alpha1.DefaultLogFile, } } @@ -58,7 +58,7 @@ func defaultRoleStorage() memgraphcomv1alpha1.RoleStorageSpec { // the cluster-wide toggle is filled in. func defaultCoreDumps() memgraphcomv1alpha1.CoreDumpsSpec { return memgraphcomv1alpha1.CoreDumpsSpec{ - ConfigureCorePattern: ptr.To(memgraphcomv1alpha1.DefaultConfigureCorePattern), + CorePattern: memgraphcomv1alpha1.DefaultCorePattern, } } @@ -179,7 +179,7 @@ var _ = Describe("MemgraphCluster CRD validation", func() { Expect(stored.Spec.Secrets.LicenseKey).To(Equal(memgraphcomv1alpha1.DefaultLicenseSecretKey)) Expect(stored.Spec.Secrets.OrganizationKey).To(Equal(memgraphcomv1alpha1.DefaultOrganizationSecretKey)) Expect(stored.Spec.Storage.Data.LibPVCSize).To(Equal(ptr.To(resource.MustParse("100Gi")))) - Expect(stored.Spec.Storage.Data.FileLogging).To(HaveValue(BeTrue())) + Expect(stored.Spec.Storage.Data.LogFile).To(Equal(memgraphcomv1alpha1.LogFileLibClaim)) Expect(stored.Spec.Storage.RetentionPolicy).To(Equal(memgraphcomv1alpha1.DefaultStorageRetention)) Expect(stored.Spec.Storage.Coordinators).To(Equal(defaultRoleStorage())) }) @@ -421,7 +421,7 @@ var _ = Describe("MemgraphCluster CRD validation", func() { Expect(dumps.Data).To(HaveValue(Equal(memgraphcomv1alpha1.RoleCoreDumpsSpec{ Size: ptr.To(resource.MustParse("200Gi")), }))) - Expect(dumps.ConfigureCorePattern).To(HaveValue(BeTrue())) + Expect(dumps.CorePattern).To(Equal(memgraphcomv1alpha1.CorePatternOperator)) Expect(dumps.Uploader.PullPolicy).To(Equal(memgraphcomv1alpha1.DefaultImagePullPolicy)) // Whether a role collects at all is its block being present: the // coordinators wrote none, and no schema default may conjure one. @@ -694,6 +694,18 @@ var _ = Describe("MemgraphCluster CRD validation", func() { }, }, "coreDumps size must be greater than zero"), + Entry("a log file location the operator does not know", "invalid-log-file", + memgraphcomv1alpha1.MemgraphClusterSpec{ + Storage: memgraphcomv1alpha1.StorageSpec{ + Data: memgraphcomv1alpha1.RoleStorageSpec{LogFile: "EmptyDir"}, + }, + }, + `Unsupported value: "EmptyDir"`), + Entry("a core pattern owner the operator does not know", "invalid-core-pattern", + memgraphcomv1alpha1.MemgraphClusterSpec{ + CoreDumps: memgraphcomv1alpha1.CoreDumpsSpec{CorePattern: "DaemonSet"}, + }, + `Unsupported value: "DaemonSet"`), Entry("an uploader Secret that is not a Secret name", "invalid-uploader-env-from-secret", memgraphcomv1alpha1.MemgraphClusterSpec{ CoreDumps: memgraphcomv1alpha1.CoreDumpsSpec{ @@ -1313,7 +1325,7 @@ var _ = Describe("MemgraphCluster CRD validation", func() { Expect(update("core-dumps-unchanged", func(c *memgraphcomv1alpha1.MemgraphCluster) { c.Spec.Image.Tag = customImageTag - c.Spec.CoreDumps.ConfigureCorePattern = ptr.To(false) + c.Spec.CoreDumps.CorePattern = memgraphcomv1alpha1.CorePatternNode })).To(Succeed()) }) @@ -1397,14 +1409,14 @@ var _ = Describe("MemgraphCluster CRD validation", func() { Entry("turning file logging off", "claims-file-logging-off", memgraphcomv1alpha1.MemgraphClusterSpec{}, func(c *memgraphcomv1alpha1.MemgraphCluster) { - c.Spec.Storage.Data.FileLogging = ptr.To(false) + c.Spec.Storage.Data.LogFile = memgraphcomv1alpha1.LogFileNone }), Entry("turning file logging on", "claims-file-logging-on", memgraphcomv1alpha1.MemgraphClusterSpec{Storage: memgraphcomv1alpha1.StorageSpec{ - Coordinators: memgraphcomv1alpha1.RoleStorageSpec{FileLogging: ptr.To(false)}, + Coordinators: memgraphcomv1alpha1.RoleStorageSpec{LogFile: memgraphcomv1alpha1.LogFileNone}, }}, func(c *memgraphcomv1alpha1.MemgraphCluster) { - c.Spec.Storage.Coordinators.FileLogging = ptr.To(true) + c.Spec.Storage.Coordinators.LogFile = memgraphcomv1alpha1.LogFileLibClaim }), // A role without a block has no size to edit, so the class is the one // core dumps knob that backs no claim here. diff --git a/internal/resources/resources.go b/internal/resources/resources.go index 0f1d1b5..3168d33 100644 --- a/internal/resources/resources.go +++ b/internal/resources/resources.go @@ -587,10 +587,9 @@ func normalizeStorage(spec memgraphcomv1alpha1.RoleStorageSpec) normalizedStorag libSize: resource.MustParse(memgraphcomv1alpha1.DefaultLibPVCSize), libAccessMode: spec.LibStorageAccessMode, libClass: spec.LibStorageClassName, - fileLogging: memgraphcomv1alpha1.DefaultFileLogging, - } - if spec.FileLogging != nil { - n.fileLogging = *spec.FileLogging + // Anything but None is the default, LibClaim, so a spec that never + // passed admission logs where an admitted one does. + fileLogging: spec.LogFile != memgraphcomv1alpha1.LogFileNone, } if spec.LibPVCSize != nil { n.libSize = *spec.LibPVCSize @@ -705,18 +704,16 @@ func normalizeCoreDumps( role *memgraphcomv1alpha1.RoleCoreDumpsSpec, ) normalizedCoreDumps { n := normalizedCoreDumps{ - enabled: role != nil, - size: resource.MustParse(memgraphcomv1alpha1.DefaultCoreDumpsSize), - class: shared.StorageClassName, - configurePattern: memgraphcomv1alpha1.DefaultConfigureCorePattern, + enabled: role != nil, + size: resource.MustParse(memgraphcomv1alpha1.DefaultCoreDumpsSize), + class: shared.StorageClassName, + // Anything but Node is the default, Operator. + configurePattern: shared.CorePattern != memgraphcomv1alpha1.CorePatternNode, uploader: shared.Uploader, } if role != nil && role.Size != nil { n.size = *role.Size } - if shared.ConfigureCorePattern != nil { - n.configurePattern = *shared.ConfigureCorePattern - } return n } diff --git a/internal/resources/statefulset_test.go b/internal/resources/statefulset_test.go index 94f966e..76c43b5 100644 --- a/internal/resources/statefulset_test.go +++ b/internal/resources/statefulset_test.go @@ -741,7 +741,7 @@ func TestStatefulSetStorageOverrides(t *testing.T) { func TestStatefulSetFileLoggingOff(t *testing.T) { cluster := minimalCluster() cluster.Spec.Storage = memgraphcomv1alpha1.StorageSpec{ - Coordinators: memgraphcomv1alpha1.RoleStorageSpec{FileLogging: ptr.To(false)}, + Coordinators: memgraphcomv1alpha1.RoleStorageSpec{LogFile: memgraphcomv1alpha1.LogFileNone}, // The data instances keep logging to a file: the knob is per role. Data: memgraphcomv1alpha1.RoleStorageSpec{}, } @@ -852,8 +852,8 @@ func TestStatefulSetCoreDumps(t *testing.T) { func TestStatefulSetCoreDumpsWithoutCorePattern(t *testing.T) { cluster := minimalCluster() cluster.Spec.CoreDumps = memgraphcomv1alpha1.CoreDumpsSpec{ - Data: &memgraphcomv1alpha1.RoleCoreDumpsSpec{}, - ConfigureCorePattern: ptr.To(false), + Data: &memgraphcomv1alpha1.RoleCoreDumpsSpec{}, + CorePattern: memgraphcomv1alpha1.CorePatternNode, } sts := dataStatefulSet(cluster) diff --git a/test/e2e/external_access_test.go b/test/e2e/external_access_test.go index 3234f39..5d440c5 100644 --- a/test/e2e/external_access_test.go +++ b/test/e2e/external_access_test.go @@ -81,9 +81,9 @@ spec: name: %s storage: coordinators: - fileLogging: false + logFile: None data: - fileLogging: false + logFile: None resources: coordinators: requests: diff --git a/test/e2e/gateway_access_test.go b/test/e2e/gateway_access_test.go index d456556..254ccdd 100644 --- a/test/e2e/gateway_access_test.go +++ b/test/e2e/gateway_access_test.go @@ -80,9 +80,9 @@ spec: name: %s storage: coordinators: - fileLogging: false + logFile: None data: - fileLogging: false + logFile: None resources: coordinators: requests: diff --git a/test/e2e/memgraphcluster_test.go b/test/e2e/memgraphcluster_test.go index 54fd618..66d593c 100644 --- a/test/e2e/memgraphcluster_test.go +++ b/test/e2e/memgraphcluster_test.go @@ -1107,9 +1107,9 @@ spec: name: %s storage: coordinators: - fileLogging: false + logFile: None data: - fileLogging: false + logFile: None resources: coordinators: requests: diff --git a/test/e2e/tls_test.go b/test/e2e/tls_test.go index 3c0bdf2..388a907 100644 --- a/test/e2e/tls_test.go +++ b/test/e2e/tls_test.go @@ -110,9 +110,9 @@ spec: name: %s storage: coordinators: - fileLogging: false + logFile: None data: - fileLogging: false + logFile: None resources: coordinators: requests: diff --git a/test/e2e/volume_resize_test.go b/test/e2e/volume_resize_test.go index 8913f0d..151537a 100644 --- a/test/e2e/volume_resize_test.go +++ b/test/e2e/volume_resize_test.go @@ -88,7 +88,7 @@ spec: libStorageClassName: %[6]s coreDumps: storageClassName: %[6]s - configureCorePattern: false + corePattern: Node coordinators: size: 1Gi data: