Skip to content

Report container restarts in ReplicaObservation status - #1

Open
dennis-upbound wants to merge 1 commit into
modelplaneai:mainfrom
dennis-upbound:replica-restarts
Open

dennis-upbound wants to merge 1 commit into
modelplaneai:mainfrom
dennis-upbound:replica-restarts

Conversation

@dennis-upbound

Copy link
Copy Markdown

A ReplicaObservation reports whether its pods are ready, but not whether their containers restarted. An engine that keeps getting OOMKilled or crashing shows up only as Ready flapping, or as CrashLoopBackOff while the kubelet backs off. Pod restart counts and termination reasons normally come from kube-state-metrics, which Modelplane's collector doesn't scrape, so today we infer engine restarts from Prometheus counters resetting. The agent already watches these pods, so it can report them directly.

This PR adds two optional status fields:

status:
  pods:
    total: 1
    scheduled: 1
    ready: 0
    restarts: 4               # Sum of the pods' container restart counts.
  lastTermination:            # Most recent lastState.terminated across them.
    pod: llama-0
    container: engine
    reason: OOMKilled
    exitCode: 137
    finishedAt: "2026-10-20T10:01:00Z"
  • pods.restarts sums restartCount over the pods' containers and init containers. It includes init containers because the agent already reports their waiting reasons.
  • lastTermination is the latest lastState.terminated by finishedAt, with the pod and container it came from.
  • Both cover only the replica's current pods, the same as a pod's own restart counts. Both are omitted when no container has restarted.
  • No new RBAC or watches. The deepcopy and CRD changes come from nix run .#generate.

Testing:

  • A new OOMKilled case in internal/replica has an engine container that was OOMKilled after 3 restarts and is now in CrashLoopBackOff, plus an init container that restarted once after an Error at an earlier time. The case checks Restarts: 4 and that the engine's OOMKilled termination wins. I wrote it inline like the other synthetic cases rather than adding it to testdata, which holds objects captured from a real cluster.
  • A new Restarted case in the controller's TestStatus checks that restarts are summed across pods and that the latest termination across pods is reported.
  • nix flake check passes: tests, golangci-lint (0 issues), the generated-files check, and helm lint.

🤖 Generated with Claude Code

A ReplicaObservation reports whether its pods are ready, but not whether
their containers restarted. An engine that keeps getting OOMKilled or
crashing shows up only as Ready flapping, or as CrashLoopBackOff while
the kubelet backs off. The control plane can't otherwise see restarts:
pod restart counts and termination reasons come from kube-state-metrics,
which Modelplane's collector doesn't scrape, so today they're inferred
from engine Prometheus counters resetting.

This commit adds two fields. pods.restarts sums the restart counts of
the replica's pods' containers, init containers included, as the agent
already does for waiting reasons. lastTermination reports the most
recent last termination state across those containers: the pod,
container, reason (like OOMKilled or Error), exit code and finishedAt.
Both only cover the replica's current pods, like a pod's own restart
counts, and are omitted when nothing has restarted. The agent already
watches these pods, so it needs no new RBAC.

Signed-off-by: Dennis Ramdass <dennis@upbound.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@negz

negz commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Thanks @dennis-upbound! Is this in service modelplaneai/modelplane#498? If so, I think it's worth a small one-pager.

modelplaneai/modelplane#498 looks like it gets into passing errors up to parent resources, which is harder than it looks. e.g. What should a ModelDeployment say when two replicas are failing for different reasons? Will an error from a pod on a workload cluster make sense to someone looking at a ModelDeployment? Kubernetes usually has the parent count failed children and leaves you to go look at them. I've always taken this stance in Crossplane, and it's been one of my more controversial decisions in the community. People find it frustrating because the children are hard to find. It'll be harder still across cluster boundaries in Modelplane.

So I think this is worth solving, but I'd like us to spend a bit more time exploring how first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants