Skip to content

fix(server): keep stopped sandboxes stopped after late driver updates - #5204

Open
shiju-nv wants to merge 1 commit into
mainfrom
fix/stopped-driver-watch-fence-c92/shiju-nv
Open

shiju-nv wants to merge 1 commit into
mainfrom
fix/stopped-driver-watch-fence-c92/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A driver bootstrap event can be queued while sandbox stop is still running, then arrive after the sandbox reaches Stopped. The gateway now waits for that stop and checks current driver state before applying the event, so an older observation cannot make the sandbox appear to start again.

Related Issue

Part of #5195. Complements #5196, which corrects the Kubernetes driver's stopped-pod snapshot. This change handles delayed observations at the gateway for every backend. #5195 tracks both layers; this PR covers the gateway behavior.

Changes

  • Recheck a bootstrap observation against the lifecycle operation and a fresh driver read when the sandbox has already stopped.
  • Preserve Stopped when the driver is unavailable, returns no sandbox, or returns an invalid snapshot.
  • Keep recovery through an accepted start and a current bootstrap observation.
  • Use one deterministic stop/watch regression, with existing recovery coverage updated to use an authoritative driver response.

Release Notes

A completed stop stays visible as Stopped when an older driver event arrives late. A later start still reports its current progress.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Wait for the active lifecycle operation and revalidate delayed bootstrap observations against current driver state. Preserve a completed stop when the read is unavailable or invalid, while retaining progress from an accepted start.

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv shiju-nv added the test:e2e Requires end-to-end coverage label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Label test:e2e applied for 760396e. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant