Skip to content

fix(server): skip the reconcile poll of a sandbox whose start is in progress - #5265

Open
matthewgrossman wants to merge 4 commits into
mainfrom
fix/5258-podman-poll-fence/matthewgrossman
Open

matthewgrossman wants to merge 4 commits into
mainfrom
fix/5258-podman-poll-fence/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes #5258. The gateway's reconcile sweep no longer polls a sandbox while its
StartSandbox call is in progress. A poll that lands mid-start could move the
sandbox from Starting to Error (ContainerExited) even though its
containers came up.

Root cause

StartSandbox holds the sandbox's lifecycle gate for the whole driver call. On
Podman that call stops the supervisor, copies restart metadata, starts the
workload, then starts the supervisor, which takes roughly 500 ms. Until the
driver starts the workload again, it still shows its previous exited state.

The two paths that apply driver snapshots treat this differently:

  • Watch events: apply_sandbox_update sees a Starting sandbox, waits on the
    lifecycle gate, and re-reads the driver before applying anything.
  • Reconcile sweep: reconcile_snapshot_sandbox polls the driver and applies
    the result under only the local mutation lock. It never looks at the gate.

A sweep tick inside the start window therefore applied a stale Exited
snapshot. The phase merge lets Error replace Starting, and the SDK's
wait_ready stops at the first Error.

Fix

If the stored phase is Starting, the sweep takes the lifecycle gate with
try_lock_for. If a start holds the gate, the sweep skips that sandbox. It
never waits, so a slow start cannot stall the sweep. The next tick or watch
event reports the settled state, so a genuine exit of the restarted workload is
still reported, just after the start returns. The sweep holds the gate while it
polls, so a start cannot begin between the check and the poll. The same check is
applied to the prune path.

How the PR got here

This PR first fixed the problem in the Podman driver, which was the wrong layer:

  1. 1984a72 added start-in-progress tracking to the driver's lifecycle fences.
    get_sandbox and list_sandboxes inspected stopped containers during a
    start and rewrote a snapshot that still showed the previous FinishedAt as
    ContainerStarting.
  2. c15cece documented that fence in the crate README.
  3. 03162db added a second fence in inspect_workload so a restart would not
    contain its own workload while the supervisor was still exited.

That approach worked but was more complex than it needed to be:

The gateway already had the right mechanism for the watch path. The sweep was
the only path that skipped it, so the final commit reverts the Podman driver
changes and applies the gate in the sweep instead. The net diff against the
base is crates/openshell-server/src/compute/mod.rs only.

Follow-up

Whether a restart can contain its own workload in inspect_workload is
unconfirmed and is tracked separately. It is out of scope here.

Testing

  • New: reconcile_sweep_skips_a_sandbox_while_its_start_is_in_progress. A sweep
    runs during a blocked start while the driver reports ContainerExited. It
    asserts the sandbox stays Starting. It fails without the fix (phase 8
    instead of Starting).
  • cargo test -p openshell-server --lib compute:: passes (403 passed, 3 ignored).
  • Clippy is clean.
  • The Podman conformance test persistent_workspace_survives_stop_start has not
    been run.

The gateway's reconcile sweep applies polled snapshots without waiting
for an in-progress start. During a stop/start restart, get_sandbox and
list_sandboxes built the stopped workload from its list entry, so the
previous run's exit was reported as ContainerExited and moved the
sandbox from Starting to Error.

While a start is in progress, inspect a stopped workload before
reporting it. If it is running again, or still shows the exit recorded
before the start, report it as ContainerStarting. A new exit, and any
exit once the start has returned, is still reported. The start is
tracked by a guard that clears the in-progress mark on every return
path.

Closes #5258

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Label test:e2e applied for c15cece. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

A restart starts the workload before its exited supervisor. Watch
reconciliation, start events, and polled snapshots could inspect the
workload in that gap, see a running workload beside an exited
supervisor, and stop it.

Record the supervisor exit that the start observed. While that start is
in progress, an exited supervisor with the recorded FinishedAt reports
the workload as starting instead of containing it. A missing
supervisor, a new supervisor exit, and any exit after the start returns
are still contained.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…rogress

StartSandbox holds the sandbox lifecycle gate for the whole driver call,
and the workload can still show its previous exit until the driver starts
it. The watch path already waits on that gate for Starting sandboxes; the
reconcile sweep did not, so a poll could move a restarting sandbox from
Starting to Error. Skip such a sandbox and let the next tick or event
report the settled state. This replaces the Podman-specific fence of
polled snapshots.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman changed the title fix(podman): keep a polled stale exit from failing an in-progress start fix(server): skip the reconcile poll of a sandbox whose start is in progress Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(podman): polled snapshot moves a restarting sandbox from Starting to Error (ContainerExited)

1 participant