Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ Test philosophy: assert external behavior, never internal call ordering or priva
- Both StatefulSets use `updateStrategy: OnDelete`, so **nothing but the operator ever restarts a workload pod**. A pod-template change no reconcile acts on takes effect never, which is what the `Updated` condition exists to report. A run-time flag change is deliberately *not* a pod-template change (see Memgraph flags above). `RollingUpdate` cannot express the required order (it sweeps highest ordinal to lowest, and `partition` is a descending cutoff, not a set), so a MAIN on any ordinal but 0 would be restarted mid-sweep and each such restart buys another failover.
- The operator never promotes a MAIN outside bootstrap and the scale-down handover: a roll deletes the MAIN's pod and lets the Raft coordinators promote. That is only safe on a Memgraph reporting an unreachable MAIN as `role=main, health=down` — a release that vacates the `main` row makes `planner.Plan` believe there is no MAIN and race the failover.
- **The shrink is the one write with an expectation** (`internal/expectations`, decided 2026-10-09). The operator reads its StatefulSets from the informer cache, and `replicaCounts` applies `max(declared, cached spec.replicas)`, so a pass reading the StatefulSet from before `shedRetiredPods` lowered it would grow it back (and shrink it again once it found the member gone, the pod recreated in between). `shedRetiredPods` records the UID and generation the apply returned; `replicaCounts` marks a role whose cached StatefulSet is the same UID at a lower generation, and the pass ends before applying anything, waiting on the StatefulSet event (`Owns`). It is in-memory on purpose: a restarted operator lists into a fresh cache that already holds every write it made, so only the writing process can be ahead of its cache. Every other cached read is safe without one (the apply restates desired state, pod deletes carry a UID precondition, `rollout`'s `Stale` covers status, membership is read live over Bolt); a new write whose stale read would undo it, or a decision pushed into Memgraph from a cached count, needs one too.
- **Conditions** (decided 2026-10-14): four, all normal-True. `Converged` (membership, storage, settings), `Updated` (the roll) and `LicenseApplied` each answer one question; `Ready` is their summary, because kstatus (and with it Flux health checks) decides on `Ready` alone for a custom resource. `Ready` is True only when the cluster serves (a MAIN behind a reachable leader) and the other three are True **for the current generation**; otherwise it carries the reason of the first that is not, in the order not serving, `LicenseApplied`, `Converged`, `Updated`, the message prefixed with the condition's name, and a True decided for an older generation makes it `Unknown/Reconciling`. Whether the cluster serves is a pass-local verdict (`conditionServing`, set where `Ready` used to be) that `summarizeReady` folds into `Ready` at the end of `Reconcile` and removes, so it never reaches the API; a pass that did not observe the cluster keeps a not-serving `Ready`. `status.main` is what says a MAIN is serving through a roll, scale or resize. Detail goes into reasons, never a new condition per activity (cass-operator's fourteen is the shape avoided).
- Log messages follow Kubernetes style: capital first letter, no trailing period, past tense, object type named (see AGENTS.md for examples).
- **External access** (`spec.externalAccess`, `type: LoadBalancer | Gateway`) is operator-owned on both sides: the external objects *and* the announced `bolt_server` of every exposed member. Both modes build the same Services, one shared by the coordinators plus one per data instance (selected by the StatefulSet pod-name label); with `LoadBalancer` they are the way in, with `Gateway` they are ClusterIPs behind one operator-owned Gateway (`gatewayClassName` required, listeners `coordinators-bolt` on 7687 and `data-<n>-bolt` on `dataPortBase + n`, default 9000) fed by one TCPRoute each. The address is discovered, never declared: external-dns hostname annotation (on the Service, or in Gateway mode on the TCPRoute, then the Gateway), then the reported hostname, then the IP, pod DNS until any exists (`Converged=False/ExternalAddressPending`). `{ordinal}` in per-instance annotation values is substituted with the pod ordinal and CEL requires it on a data hostname annotation. Only the bolt address ever leaves the cluster network; the operator itself always dials coordinators on pod DNS (`resources.CoordinatorEndpoints`), never through the LoadBalancer or Gateway. External objects follow the data pods the operator *runs* (applied count), are pruned by label once the spec stops describing them, and removing the block reverts every `bolt_server` to pod DNS. The Gateway API CRDs are never bundled: `main.go` discovers once at start (`controller.GatewayAPIServed`) whether `Gateway` and `TCPRoute` are served at `v1`, watches the kinds only if so, and a cluster asking for `Gateway` without that gets `Converged=False/ApplyFailed` naming what is missing. TCPRoute is `v1` only from Gateway API v1.6 (Envoy Gateway ≥ v1.9); older releases serve it as `v1alpha2` alone and v1.6 stops serving `v1alpha2`, so no single version works on both sides — the operator requires ≥ v1.6. The e2e suite gets LoadBalancer addresses from MetalLB and a Gateway from Envoy Gateway (`hack/kind-metallb.sh`, `hack/kind-envoy-gateway.sh`, run by `make setup-test-e2e`); envtest loads the Gateway API CRDs from the `sigs.k8s.io/gateway-api` module.
- **TLS** (`spec.tls`) is presence-based per mode and takes one shared `kubernetes.io/tls` Secret per mode for both roles, fixed keys, no key-name knobs: a StatefulSet has one pod template, and Memgraph verifies no member hostname, so per-pod identities buy nothing. `bolt` (`secretName`; `tls.crt`/`tls.key` mounted read-only at `/etc/memgraph/ssl`, `--bolt-cert-file`/`--bolt-key-file`) also moves 9091 to https because Memgraph serves metrics from the Bolt server context. Everything downstream is derived, never a knob: the operator dials `bolt+ssc` (TLS, unverified — verification would force pod-DNS SANs and a CA onto a user-facing cert and buys nothing while Bolt is unauthenticated; Bolt auth is the trigger to revisit) and falls back to the other scheme once, because a roll turning TLS on or off leaves coordinators in the old mode while the spec says the new one (`memgraph.dialSchemes`); the ServiceMonitor switches to https with `insecureSkipVerify`. Probes are TCP and untouched. Rotation is not the operator's: the Secret is mounted without `subPath` so the kubelet refreshes it, and Memgraph picks it up on `RELOAD BOLT SERVER TLS` until the core reloads by mtime (planned upstream); the operator watches no Secret and never restarts for an in-place update, and a `secretName` change is an ordinary roll. `intraCluster` (`secretName`; `tls.crt`/`tls.key`/`ca.crt` mounted read-only at `/etc/memgraph/intra_cluster_tls`, the three `--cluster-*-file` flags always together) is mTLS proving membership, not identity: Memgraph verifies the chain and demands a client cert but checks no hostname, so one shared cert is the shape (nine of thirteen surveyed operators do the same). Its presence is pinned by a CEL transition rule on `MemgraphClusterSpec`, not on `TLSSpec`, because a rule on the block does not fire when the block is added or removed whole: a member with it cannot talk to one without, so a one-pod-at-a-time roll deadlocks on lag. `secretName` changes roll normally; a CA cut-over needs a bundled `ca.crt` in both Secrets, documented, not detected. Renewal is `RELOAD INTRA_CLUSTER TLS` per instance until the core reloads by mtime (Raft already does).
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,15 +83,15 @@ memgraph 3 2 instance_0 True True 4m12s
```

- **`MAIN`** is the data instance the coordinators elected as MAIN — the one that accepts writes. It is observed, not decided by the operator, so it changes on failover.
- **`READY`** is True once a MAIN is elected, i.e. the cluster serves writes.
- **`READY`** is True once the cluster both serves and runs its spec: a MAIN is elected, and `Converged`, `Updated` and `LicenseApplied` are all True. Otherwise its reason says what is unfinished, the cluster not serving first; a MAIN keeps serving through a roll, a scale or a resize, so `MAIN` is what says whether writes are accepted.
- **`CONVERGED`** is True once every declared coordinator and data instance is registered and reported healthy, and both StatefulSets run the declared number of replicas.

`kubectl get mgc -n memgraph -o wide` adds how many of each role's declared members the cluster actually has registered (`REGISTERED-COORDINATORS`, `REGISTERED-DATA`), which is what a scale is watched through.

To block a script or a GitOps step on the cluster being usable:
To block a script or a GitOps step on the cluster being done (Flux and other tools built on kstatus read the same condition):

```sh
kubectl wait --namespace memgraph --for=condition=Converged \
kubectl wait --namespace memgraph --for=condition=Ready \
memgraphcluster/memgraph --timeout=10m
```

Expand Down
40 changes: 24 additions & 16 deletions api/v1alpha1/memgraphcluster_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -179,16 +179,22 @@ const (
RetentionPolicyDelete StorageRetentionPolicy = "Delete"
)

// Condition types reported on MemgraphCluster status. Both use normal-True
// polarity: True is the healthy state. Ready answers "is the cluster serving"
// (a MAIN is elected and reachable); Converged answers "does the cluster match
// the declared topology" (every declared coordinator and data instance is
// registered, and both StatefulSets run the declared number of replicas). A
// cluster can be Ready but not Converged — a MAIN still serves while a lost
// replica registration is being restored, or while a scale finishes.
// Condition types reported on MemgraphCluster status. All four use
// normal-True polarity: True is the healthy state. Converged, Updated and
// LicenseApplied each answer one question; Ready is their summary, the one
// condition tools that know nothing about Memgraph read (kstatus, and with it
// Flux health checks, decides on Ready alone for a custom resource).
const (
// ConditionReady is True when a MAIN data instance is elected and the
// coordinator leader is reachable.
// ConditionReady is True when the cluster serves and runs its spec: a
// data instance is MAIN behind a reachable coordinator leader, and
// LicenseApplied, Converged and Updated are all True for the current
// generation. Otherwise it carries the reason of the first thing that is
// not, in that order: the cluster not serving (NoMainElected,
// WorkloadsNotReady, CoordinatorUnreachable, NoCoordinatorLeader,
// ApplyFailed), then LicenseApplied, Converged and Updated, with the
// message naming the condition the reason came from. A MAIN keeps serving
// through most of what makes Ready False — a roll, a scale, a resize — and
// status.main names it throughout.
ConditionReady = "Ready"

// ConditionConverged is True when the observed cluster matches the declared
Expand Down Expand Up @@ -265,9 +271,8 @@ const (
// registered but an exposed one still announces its in-cluster address,
// because the LoadBalancer in front of it has not been given an external
// address yet. The message names the Service being waited on. Registration
// itself is complete and in-cluster clients are served, so Ready is
// unaffected; what is missing is the cloud provisioning the operator cannot
// hurry. It persisting means the LoadBalancer is not being provisioned at
// itself is complete and in-cluster clients are served; what is missing
// is the cloud provisioning the operator cannot hurry. It persisting means the LoadBalancer is not being provisioned at
// all — no cloud controller or no address pool answers the Service.
ReasonExternalAddressPending = "ExternalAddressPending"

Expand Down Expand Up @@ -395,16 +400,19 @@ const (
// clears it.
ReasonLicenseRejected = "LicenseRejected"

// ReasonMainElected is set when a data instance is observed as MAIN.
ReasonMainElected = "MainElected"

// ReasonNoMainElected is set when the cluster is reachable but no data
// instance has yet been promoted to MAIN.
ReasonNoMainElected = "NoMainElected"

// ReasonReconciling is set on a condition no pass has decided yet, so
// every condition is present from the first status write on.
// every condition is present from the first status write on. Ready also
// carries it while one of the conditions it summarizes still describes an
// earlier generation of the spec.
ReasonReconciling = "Reconciling"

// ReasonReconciled is set on Ready when the cluster serves and every other
// condition is True for the current generation.
ReasonReconciled = "Reconciled"
)

// InitializeConditions sets every condition the resource reports to Unknown,
Expand Down
2 changes: 1 addition & 1 deletion docs/external-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ kubectl get mgc memgraph -n memgraph -o jsonpath='{.status.externalAccess}' | jq
| `False` | `ApplyFailed` | The exposure cannot be served on this cluster at all; the message says what to install. Waiting does not clear it. |
| `True` | `AllInstancesRegistered` | Every member is registered at the address in `status.externalAccess`. |

So `kubectl wait --for=condition=Converged` on an exposed cluster means "reachable at the addresses in status", which is what a client outside the cluster wants to wait for. `Ready` is untouched by any of this: a MAIN is serving and in-cluster clients work throughout.
So `kubectl wait --for=condition=Converged` on an exposed cluster means "reachable at the addresses in status", which is what a client outside the cluster wants to wait for. `Ready` carries the same reason until then, since it summarizes `Converged`, while a MAIN serves and in-cluster clients work throughout (`status.main` names it).

## Lifecycle

Expand Down
2 changes: 1 addition & 1 deletion docs/flags.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Run-time settings are applied on the pass that finds the cluster registered, bef
- `Converged=False` with reason **`SettingsPending`** names the ready pods that did not answer. The usual cause is a Bolt endpoint lagging its pod's readiness by a few seconds; the pass retries on a delay. A pod that is not ready is not dialed and not reported — it is the roll's or the kubelet's business, and the flag file has its settings for when it comes up.
- `Converged=False` with reason **`SettingsRejected`** names the pod, the exact `SET DATABASE SETTING` and Memgraph's error, verbatim: a value the setting's validator refuses (`Boolean value supports only 'false' or 'true'`, `Unsupported log level`), or `Unknown setting name` on a Memgraph that does not have the setting at run time. It is retried every pass and only a changed flag clears it. The operator never escalates to a restart on its own: a value an instance refuses at run time it would refuse at startup too, and crash-loop on.

`Ready` is unaffected by either: a refused log level does not make a MAIN stop serving. `Updated` is the roll's, as before.
`Ready` carries the same reason, since it summarizes `Converged`, but a refused log level does not make a MAIN stop serving: `status.main` still names it. `Updated` is the roll's, as before.

## Removing a flag

Expand Down
2 changes: 1 addition & 1 deletion docs/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Converged=False reason=ApplyFailed
restart the operator
```

while `Ready` and everything else about the cluster are unaffected. Installing the CRD later takes an operator restart, which is documented here rather than detected, exactly as for the Gateway API. Dropping the block clears the condition.
and `Ready` the same reason, while the cluster itself serves as before. Installing the CRD later takes an operator restart, which is documented here rather than detected, exactly as for the Gateway API. Dropping the block clears the condition.

## `spec.monitoring.grafanaDashboard`

Expand Down
Loading
Loading