The StatefulSet built by internal/controller/resources.go gets the startup path right — startupProbe with FailureThreshold: 30 (~5m, enough for WAL replay on a large dir) and PodManagementPolicy: Parallel. The shutdown and disruption paths are unhandled.
1. No terminationGracePeriodSeconds
The pod spec doesn't set it, so it defaults to 30s. On SIGTERM an oteldb node has to flush the unflushed head and close the WAL cleanly (store.Close). For a node carrying a large head that can exceed 30s, after which the kubelet SIGKILLs it. That turns every rollout, drain and eviction into an unclean shutdown — recoverable via WAL replay, but it means paying replay cost on every restart and relying on the recovery path far more often than necessary.
Proposal: set a generous default (120s or more) and expose it as spec.terminationGracePeriodSeconds for tuning against head size.
2. No PodDisruptionBudget
Nothing constrains voluntary disruption. A node drain — or an unlucky multi-node maintenance window — can take out more than RF-1 ring owners at once, which is exactly the condition the replication factor exists to survive.
Since the operator already knows replicas and cluster.replicationFactor, it can derive a correct budget rather than making the user reason about it: maxUnavailable: RF-1 (clamped so it can never reach a value that permits quorum loss), or the equivalent minAvailable.
Proposal: create a PDB alongside the StatefulSet, derived from RF, with an opt-out for users who manage disruption themselves.
Both are small and independent of each other; happy to split if preferred.
The StatefulSet built by
internal/controller/resources.gogets the startup path right —startupProbewithFailureThreshold: 30(~5m, enough for WAL replay on a large dir) andPodManagementPolicy: Parallel. The shutdown and disruption paths are unhandled.1. No
terminationGracePeriodSecondsThe pod spec doesn't set it, so it defaults to 30s. On SIGTERM an oteldb node has to flush the unflushed head and close the WAL cleanly (
store.Close). For a node carrying a large head that can exceed 30s, after which the kubelet SIGKILLs it. That turns every rollout, drain and eviction into an unclean shutdown — recoverable via WAL replay, but it means paying replay cost on every restart and relying on the recovery path far more often than necessary.Proposal: set a generous default (120s or more) and expose it as
spec.terminationGracePeriodSecondsfor tuning against head size.2. No PodDisruptionBudget
Nothing constrains voluntary disruption. A node drain — or an unlucky multi-node maintenance window — can take out more than
RF-1ring owners at once, which is exactly the condition the replication factor exists to survive.Since the operator already knows
replicasandcluster.replicationFactor, it can derive a correct budget rather than making the user reason about it:maxUnavailable: RF-1(clamped so it can never reach a value that permits quorum loss), or the equivalentminAvailable.Proposal: create a PDB alongside the StatefulSet, derived from RF, with an opt-out for users who manage disruption themselves.
Both are small and independent of each other; happy to split if preferred.