bug(kubernetes): stopped sandbox regresses to Provisioning after stop completes #5195

User Story

I run the testkit conformance suite against the Kubernetes gateway in CI. In the workspace-managed lane, lifecycle::restart_stopped_sandbox stops a sandbox, sees it Stopped, and then a second (idempotent) StopSandbox is rejected because the gateway now reports the sandbox as Provisioning. Nothing started it.

Problem Statement

After a stop has settled, the gateway can move a sandbox from Stopped to Provisioning on a late driver watch snapshot. There was no StartSandbox. While it shows Provisioning, StopSandbox is rejected with FAILED_PRECONDITION:

sandbox must be Ready, Completed, Suspended, Error, or in an automatic restart to stop (current phase: Provisioning)

Mechanism, as read from the code at cc0642fa8:

1. During a stop the Kubernetes driver writes openshell.ai/sandbox-runtime-bootstrapping: "true" with bootstrap-operation: "stop" (phase Releasing, then Suspending). It keeps that annotation until sandbox_runtime_suspension_completion_patch clears it.
   • sandbox_runtime_stop_begin_patch / sandbox_runtime_suspension_patch
   • sandbox_runtime_suspension_completion_patch
2. The watch converter checks only whether that annotation is "true" and ignores the operation. If it is, mark_sandbox_runtime_bootstrapping strips the Suspended condition and adds Bootstrapping=True ("replacement sandbox-runtime generation is starting"). As a result, every snapshot of a CR mid-stop looks like a start.
   • sandbox_runtime_bootstrap_in_progress: sandbox_runtime_bootstrap_operation exists right below it but isn't consulted.
   • sandbox_from_object_with_sandbox_runtime_readiness / mark_sandbox_runtime_bootstrapping
   • Watch events are converted from the event's own object, after awaiting several readiness API calls per event (watch loop), so a mid-stop object can reach the gateway after the stop RPC has returned.
3. In the gateway, apply_driver_snapshot allows Stopped to be overridden by any snapshot that satisfies driver_snapshot_confirms_starting (Bootstrapping=True and no Suspended=True). The comment assumes "a genuine stop includes Suspended=True", which is false after step 2. apply_sandbox_update only revalidates against live driver state for Starting or a Ready exit, not for Stopped.
   • apply_driver_snapshot Stopped arm
   • driver_snapshot_confirms_starting
   • apply_sandbox_update
   • stop admission

Whether the stale snapshot lands before or after the gateway settles Stopped is timing. It was observed once, in the workspace-managed lane only, but the code path is shared by every Kubernetes mode.

Impact / Why This Matters

• A stopped sandbox is shown as Provisioning with no start in flight, which misreports its state to users and the TUI.
• While it's in that state, stop (and anything else gated on Stopped) is rejected. Clients that retry until the phase settles may wait on a transition that only happens by accident.
• It makes the conformance restart_stopped_sandbox scenario flaky on Kubernetes.

Acceptance Criteria

• Once a sandbox has reached Stopped, driver snapshots that describe the stop itself (bootstrap operation stop) never move it out of Stopped.
• StopSandbox on a sandbox that has been reported Stopped succeeds (it's idempotent) until a StartSandbox or delete is accepted.
• The real start path still recovers a stale Stopped view: a snapshot with bootstrap operation create/restart still moves it forward.
• There's a regression test for a stop-time snapshot delivered after the gateway has settled Stopped.

Possible directions (for discussion):

• Driver: only call mark_sandbox_runtime_bootstrapping when the bootstrap operation isn't stop, or preserve Suspended during a stop.
• Gateway: only honor driver_snapshot_confirms_starting for a Stopped sandbox while a StartSandbox is in flight, or revalidate against the driver as apply_sandbox_update already does for Starting.

Reproduction Steps

This is timing dependent. Seen with the conformance suite in the e2e:kubernetes:workspace-managed lane (deploy/helm/openshell/ci/values-workspace-managed.yaml, workspace_mode: managed, agent-sandbox v1beta1):

1. Create a sandbox running sleep infinity and wait for Ready.
2. StopSandbox and wait for Stopped.
3. Within about 2 seconds, call StopSandbox again.
4. Occasionally, step 3 fails with FAILED_PRECONDITION ... (current phase: Provisioning), and the gateway logs Sandbox phase changed old_phase=Stopped new_phase=Provisioning.

A deterministic reproduction is a unit test: feed apply_driver_snapshot a Stopped sandbox and a snapshot built from a CR that carries the stop-begin annotations.

Environment

• main at cc0642fa8 (first seen on PR feat(conformance): add the openshell-conformance program #5138 head 2fecde62c; the driver and server code are unchanged between them).
• Kubernetes e2e, workspace-managed mode, agent-sandbox v1beta1, kubernetes-secrets credential driver.

Logs

Failing job: https://github.com/NVIDIA/OpenShell/actions/runs/41525881380/job/125578174853 (attempt 1; the rerun and the other Kubernetes lanes passed).

Sandbox test-gbo65w4f-rs, id bfd99b26-4e62-4286-aa4e-be69b9489fd6:

22:07:57.856  supervisor session: accepted (sandbox Ready; test proceeds)
22:07:58.858  StopSandbox -> kubernetes.stop_sandbox: Prepared Kubernetes sandbox stop sandbox_api_version=v1beta1
22:07:58.881  supervisor session: stream closed by supervisor
              (first stop settles Stopped; exec on the stopped sandbox is rejected as expected)
22:08:00.819  driver_watch.sandbox_updated: Sandbox phase changed old_phase=Stopped new_phase=Provisioning
22:08:00.864  second StopSandbox rejected: gateway error (9): sandbox must be Ready, Completed,
              Suspended, Error, or in an automatic restart to stop (current phase: Provisioning)

The rejected stop never reached the driver: there's only one kubernetes.stop_sandbox span for this sandbox.

Related but different: #4371 (stuck in Provisioning with replicas: 0, no recovery) and #3525 (closed; stale boundary reconciliation for suspended sandboxes).