Skip to content

fix(server): keep old runtime status from failing sandbox restarts - #5237

Open
shiju-nv wants to merge 2 commits into
NVIDIA:fix/stopped-driver-watch-fence-c92/shiju-nvfrom
shiju-nv:fix/4932-lifecycle-observation-complete/shiju-nv
Open

shiju-nv wants to merge 2 commits into
NVIDIA:fix/stopped-driver-watch-fence-c92/shiju-nvfrom
shiju-nv:fix/4932-lifecycle-observation-complete/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

During a sandbox restart, a periodic status check can still see the previous run's exited container or no container at all. That stale observation can mark the new start as failed. Protect the start until its driver operation settles, then retain the replacement's actual readiness or failure.

Related Issue

Closes #4932. Related: #4981 and #5114. Depends on #5204.

Continues @prekshivyas's published candidate in #4932 and shares reconciliation guards and lifecycle coverage with @TaylorMutch's #4983. This also handles atomic start ownership and legacy retries. Resolve the shared scope and landing order before merging; after #4983 lands, reduce this PR to the remaining changes.

Changes

  • Save the new start and its driver ownership together; retries retain the same generation and session.
  • Defer stale observations while preserving readiness checks and genuine later failures.
  • Settle ownership on preparation failure and keep deletion cleanup and unrelated sandboxes responsive.
  • Carry the upstream fix(tests): pin the k3s installer to a release #5223 K3s installer pin so hosted qualification uses the fixed installer source.

Release Notes

Sandbox restarts no longer fail because a periodic check applies the previous run's state during the new start.

Testing

At head 7a6b2767, Branch Checks passed. The focused lifecycle regressions, server Clippy and formatting also passed; the lifecycle source is unchanged by the CI installer backport.

The Docker runtime job and Podman runtime job passed on this head. Both ran the stop/start and retained-workspace cases; Docker also passed the SSH host-identity restart/recreation case.

The full E2E run is still running; its aggregate result remains pending.

  • Checks appropriate to the affected code and behavior pass: remaining E2E aggregate pending
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv
shiju-nv marked this pull request as draft November 4, 2026 05:31
@copy-pr-bot

copy-pr-bot Bot commented Nov 4, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@shiju-nv shiju-nv changed the title fix(server): prevent stale observations from failing sandbox restarts fix(server): keep old runtime status from failing sandbox restarts Nov 4, 2026
get.k3s.io serves the latest install.sh from k3s master, so the pinned
checksum broke when upstream changed the script. Download install.sh from
the release tag matching k3s_version instead and pin its checksum.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv shiju-nv added the test:e2e Requires end-to-end coverage label Nov 4, 2026
@shiju-nv
shiju-nv marked this pull request as ready for review November 4, 2026 11:09
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/5237 is at fe8eaf2 while the PR head is 7a6b276. A maintainer needs to comment /ok to test 7a6b276729a83403776bb80b5e48b2765f7e4c32 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:platform test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants