Repository navigation
Report container restarts in ReplicaObservation status - #1
dennis-upbound wants to merge 1 commit into
Conversation
A ReplicaObservation reports whether its pods are ready, but not whether their containers restarted. An engine that keeps getting OOMKilled or crashing shows up only as Ready flapping, or as CrashLoopBackOff while the kubelet backs off. The control plane can't otherwise see restarts: pod restart counts and termination reasons come from kube-state-metrics, which Modelplane's collector doesn't scrape, so today they're inferred from engine Prometheus counters resetting. This commit adds two fields. pods.restarts sums the restart counts of the replica's pods' containers, init containers included, as the agent already does for waiting reasons. lastTermination reports the most recent last termination state across those containers: the pod, container, reason (like OOMKilled or Error), exit code and finishedAt. Both only cover the replica's current pods, like a pod's own restart counts, and are omitted when nothing has restarted. The agent already watches these pods, so it needs no new RBAC. Signed-off-by: Dennis Ramdass <dennis@upbound.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @dennis-upbound! Is this in service modelplaneai/modelplane#498? If so, I think it's worth a small one-pager. modelplaneai/modelplane#498 looks like it gets into passing errors up to parent resources, which is harder than it looks. e.g. What should a ModelDeployment say when two replicas are failing for different reasons? Will an error from a pod on a workload cluster make sense to someone looking at a ModelDeployment? Kubernetes usually has the parent count failed children and leaves you to go look at them. I've always taken this stance in Crossplane, and it's been one of my more controversial decisions in the community. People find it frustrating because the children are hard to find. It'll be harder still across cluster boundaries in Modelplane. So I think this is worth solving, but I'd like us to spend a bit more time exploring how first. |
A ReplicaObservation reports whether its pods are ready, but not whether their containers restarted. An engine that keeps getting OOMKilled or crashing shows up only as
Readyflapping, or asCrashLoopBackOffwhile the kubelet backs off. Pod restart counts and termination reasons normally come from kube-state-metrics, which Modelplane's collector doesn't scrape, so today we infer engine restarts from Prometheus counters resetting. The agent already watches these pods, so it can report them directly.This PR adds two optional status fields:
pods.restartssumsrestartCountover the pods' containers and init containers. It includes init containers because the agent already reports their waiting reasons.lastTerminationis the latestlastState.terminatedbyfinishedAt, with the pod and container it came from.nix run .#generate.Testing:
OOMKilledcase ininternal/replicahas an engine container that was OOMKilled after 3 restarts and is now in CrashLoopBackOff, plus an init container that restarted once after anErrorat an earlier time. The case checksRestarts: 4and that the engine's OOMKilled termination wins. I wrote it inline like the other synthetic cases rather than adding it totestdata, which holds objects captured from a real cluster.Restartedcase in the controller'sTestStatuschecks that restarts are summed across pods and that the latest termination across pods is reported.nix flake checkpasses: tests, golangci-lint (0 issues), the generated-files check, and helm lint.🤖 Generated with Claude Code