[vineethsai7]

Summary

Keep sandboxes created by another compute driver when a gateway's compute_driver changes. The gateway records each sandbox's creating driver, leaves foreign records in their persisted phase, and shows why they are unavailable until the original driver is selected again.

Related Issue

Closes #4769

Supersedes #5071 (auto-closed before the vouch)

Changes

• Record internal.openshell.ai/created-by-compute-driver for every new sandbox. The existing internal.openshell.ai/compute-driver annotation remains the sandbox authentication binding and is used as a fallback for older authenticated records. Records with neither annotation retain their existing reconciliation behavior.
• Skip foreign records in missing-resource reconciliation, startup, provisioning deadline expiry and cleanup, automatic restart, shutdown, lifecycle recovery, and pending configuration update recovery. These background paths do not call the newly selected driver for foreign IDs.
• Reject start, stop, suspend, exec, and configuration updates for foreign sandboxes with a precondition error naming both drivers. Deleting one removes its gateway record without sending a delete request to the active driver.
• Add a read-only Ready=False condition with reason ComputeDriverMismatch to get, list, and watch responses. The CLI list displays the reason. The stored phase and conditions remain unchanged, so switching back restores the sandbox's prior view.
• Document driver changes in gateway configuration and link to that guidance from the sandbox runtimes page.

Review findings from #5071

• P1, DCO and commit text: Replace the unsigned commit with a conventional commit signed off by Vineeth Sai Narajala; omit attribution footers.
• P1, provisioning deadlines: Check ownership before deadline refresh, timeout claim, and timeout reclamation. Add coverage for Provisioning, Starting, and an expired record.
• P1, automatic restart: Filter the due-restart sweep and re-check the record under the lifecycle gate before claiming or stopping it. Add a due-restart regression test.
• P2, shutdown and recovery: Skip foreign records before shutdown stop calls and before recovery of quiesce transitions. Add coverage for Ready, Stopping, and Suspending records.
• P2, legacy docs: Specify that the current-driver fallback applies only when both driver annotations are absent.

Release Notes

Changing a gateway's compute driver keeps sandboxes created by its previous driver. openshell sandbox list shows the mismatch; start, stop, exec, and configuration updates refuse those sandboxes until you switch back. Deleting a foreign sandbox removes only its gateway record.

Testing

• Checks appropriate to the affected code and behavior pass.

• Unit tests added/updated.

• E2E tests added/updated.

• cargo fmt -p openshell-server -p openshell-cli -- --check: passed on dc-build-1.

• cargo clippy -p openshell-server -p openshell-cli --features vendored-z3 --all-targets -- -D warnings: passed on dc-build-1.

• cargo test -p openshell-server -p openshell-cli --features vendored-z3 --lib -- --test-threads=8: passed on dc-build-1 (400 CLI tests and 2,617 server tests; zero failures, 38 server tests ignored).

• python3 tasks/scripts/check_docs_nav.py: passed.

• npx --yes fern-api@5.112.0 check --warnings from fern/: zero errors, three existing warnings (MDX syntax elsewhere on the configuration page, unauthenticated redirect check, and theme contrast).

No driver-switch E2E run was performed for this branch.

Checklist

• Follows Conventional Commits.
• Commits are signed off (DCO).
• Architecture docs updated (if applicable).

Review follow-up

• P1, foreign supervisor sessions and callbacks (4445828231): Fixed in 1621b43efccd322afcc2ae9303e2bc452052ec7e. The gateway checks driver ownership before accepting a supervisor reconnect and before session state, exit/finalize, configuration, policy, endpoint, or provider readiness callbacks can change state. The reconnect/exit regression fails on the original code and passes with the fix.
• P1, foreign delete with a pending driver operation (4445828245): Fixed in 1621b43efccd322afcc2ae9303e2bc452052ec7e. Explicit record-only deletion bypasses the old driver's pending-operation interlocks at both the Deleting transition and final record removal. Request identity and resource-version preconditions remain enforced; the active driver receives no delete call. The pending-operation regression fails on the original code and passes with the fix.
• P1, SSH and TCP relays for foreign sandboxes (4445828258): Fixed in 1621b43efccd322afcc2ae9303e2bc452052ec7e. CreateSshSession and ForwardTcp check driver ownership before opening a relay. The relay regression fails on the original code and passes with the fix.

Follow-up checks on the build box: cargo fmt -p openshell-server -- --check passed; cargo clippy -p openshell-server --features vendored-z3 --all-targets -- -D warnings passed; cargo test -p openshell-server --features vendored-z3 --lib -- --test-threads=4 passed (2,619 passed, 38 ignored). An 8-thread repeat had one unrelated intermittent workspace-authorization audit test failure; that test passed alone and the complete 4-thread run passed.

The driver-switch E2E scenarios were not run on the build box because it has no KVM, Docker, or mise. The lead runs manual E2E on a real host separately.

Manual verification

PR: NVIDIA/OpenShell #5133 (fix/4769-keep-other-driver-sandboxes-v2), head 1621b43efc.
Before: upstream main be5e49499 (openshell 0.1.7-dev.40+gbe5e49499). After: the PR head.

Environment

• Host: RHEL 9.8 x86_64 (EC2), Docker Engine 29.8.1 (root daemon) and rootless Podman 5.8.2, both present. One disposable local account, private HOME, one gateway process with its own SQLite database on a non-default loopback port (plaintext local-development config, allow_unauthenticated_users). No host OpenShell service was involved.
• Both runs use the same procedure and the same gateway database; only the gateway/CLI build differs. Binaries were built on an Ubuntu 24.04 box (cargo build --release); supervisor and sandbox runtime are local images set as supervisor_image and sandbox_runtime_image for both drivers. Gateway and CLI ran on RHEL 9 through an Ubuntu 24.04 glibc loader.

Steps

# driver A = docker
openshell-gateway                       # gateway.toml: compute_driver = "docker"
openshell gateway add http://127.0.0.1:<port> --name <name>
openshell sandbox create --name osp-keep --no-tty -- sleep 3000     # becomes Ready
openshell sandbox stop osp-keep                                     # phase Stopped
# switch to driver B = podman: stop the gateway, set compute_driver = "podman" in the same TOML, restart
openshell sandbox list                                              # repeat for ~10 minutes
# after only: switch back to docker, restart, then
openshell sandbox start osp-keep; openshell sandbox exec osp-keep -- echo ...

Before (main)

The gateway restarted with the podman driver listed osp-keep as Stopped at first. The periodic store/backend reconciliation then did not find it in the podman driver and removed the record:

18:45:45  openshell sandbox stop osp-keep      -> Stopped
18:45:51  gateway restarted with compute_driver = "podman"; sandbox list: osp-keep  Stopped
18:50:51  WARN reconcile{driver.name=podman backend_count=0 store_count=1}: Retained sandbox resource disappeared from the compute driver sandbox_id=93051319-... sandbox_name=osp-keep phase=Stopped
18:51:51  INFO reconcile{driver.name=podman backend_count=0 store_count=1}: Removing sandbox from store after it disappeared from the compute driver snapshot sandbox_id=93051319-... sandbox_name=osp-keep age_secs=382
18:54:37  openshell sandbox list   -> No sandboxes found.
          docker ps -a: openshell-default--osp-keep-...  Exited (1) 9 minutes ago      <- the docker container and its workspace volume are still there, now unmanaged

The stopped sandbox from driver A was deleted from the gateway about six minutes after the switch (one reconcile sweep after the 5-minute orphan grace), leaving its container behind with no record.

After (PR head)

19:12:22  openshell -g osp-d sandbox create --name osp-keep ...   (docker driver)  Ready
19:12:38  openshell -g osp-d sandbox stop osp-keep                Stopped
19:12:45  gateway restarted with compute_driver = "podman"
NAME      CREATED              PHASE  REASON
osp-keep  2026-11-02 19:12:22  Stopped  Created on compute driver 'docker'; this gateway runs 'podman'

$ openshell sandbox get osp-keep           (excerpt)
  Annotations: internal.openshell.ai/created-by-compute-driver: docker
  Conditions:  - Ready: False (ComputeDriverMismatch) - Created on compute driver 'docker'; this gateway runs 'podman'

$ openshell sandbox start osp-keep
  precondition: sandbox was created by compute driver 'docker', but this gateway uses 'podman'; switch the gateway back to 'docker' to use it, or delete it

19:21:53  (9 minutes after the switch, past the grace period and several reconcile sweeps)
osp-keep  2026-11-02 19:12:22  Stopped  Created on compute driver 'docker'; this gateway runs 'podman'
          docker ps -a: openshell-default--osp-keep-...  Exited (1) 9 minutes ago
          log lines "Removing sandbox from store": 0

# switch back to docker
osp-keep  2026-11-02 19:12:22  Stopped  -
$ openshell sandbox start osp-keep   -> Started sandbox osp-keep
osp-keep  2026-11-02 19:12:22  Ready  -
$ openshell sandbox exec osp-keep -- echo restored-after-driver-switch-back
restored-after-driver-switch-back

Result

PASS. With the PR, the stopped sandbox created by the docker driver is kept after the gateway is switched to podman: it is listed with a clear ComputeDriverMismatch reason, refuses start under the wrong driver, survived nine minutes of reconciliation, and starts and runs a command again after the gateway is switched back to docker. On main it was deleted from the gateway after about six minutes.

Only the docker-then-podman direction was run (a stopped sandbox, the case in the issue). Podman-then-docker was not run.

Second run: macOS, VM driver to Docker driver

Result: PASS (with a limit: the second driver was Docker on a forwarded remote daemon, and no Docker sandbox was created)

Environment

• Host: macOS 15.8 (Apple silicon, arm64), disposable local account, private HOME and state dirs, gateway on a non-default loopback port, plaintext. The first driver is the VM driver (Hypervisor.framework). The host has no Docker or Podman of its own.
• Second driver: the in-tree Docker driver, pointed (socket_path) at a private forwarded unix socket of a real Docker daemon (Docker 29.1.3, linux/amd64) on another machine. The driver connected, pulled its runtime images and reported Compute driver connected configured_driver=docker. No Docker sandbox container was created or run (the daemon is amd64 and the host is arm64); the test only needs the gateway to run with the other driver selected.
• Before: upstream main at be5e49499. After: PR head 1621b43efc (fix(server): fence foreign sandbox callbacks and pending deletes). Gateway, VM driver and CLI built on the host (Rust 1.95.0, release); for the PR the CLI was built from the PR head too, since the list REASON column is a CLI change.

Steps (same before and after)

1. Fresh state; gateway with compute_driver = "vm"; openshell sandbox create --name v1 --from ubuntu:24.04 -- sleep 3000 in the background; wait until v1 is Ready.
2. Stop the gateway cleanly (SIGTERM to its exact PID; the shutdown sweep stops v1: stopped=1 failed=0).
3. Start the gateway again on the same database with compute_driver = "docker". Run openshell sandbox list and sandbox get v1; for the PR also try sandbox start, exec and stop on v1.
4. Stop it and start the gateway again with compute_driver = "vm"; list and exec into v1.

Before (main, be5e49499)

Starting the Docker-driver gateway treats the VM sandbox as a lost Docker resource and writes that into its record:

WARN openshell_server::compute: Cannot start sandbox: backend resource is missing sandbox_name=v1
INFO openshell_server::compute: Sandbox start sweep complete started=0 recovered=0 missing_backend=1 failed=0

NAME  CREATED              PHASE
v1    2026-11-02 19:39:30  Error
  Ready: False (BackendResourceMissing) - Sandbox compute resource disappeared while the gateway was offline

Switching back to the VM driver does not recover it: after the second start v1 is still Phase: Error, BackendResourceMissing, and the VM driver does not start it.

After (PR head, 1621b43efc)

With the Docker driver selected the sandbox is kept, left in its stored phase, and marked as unavailable with a clear reason; nothing is sent to the Docker driver for it:

INFO openshell_server::compute: Not starting sandbox created by another compute driver sandbox_name=v1 owner_driver="vm" driver="docker"

NAME  CREATED              PHASE         REASON
v1    2026-11-02 19:43:06  Provisioning  Created on compute driver 'vm'; this gateway runs 'docker'
  Ready: False (ComputeDriverMismatch) - Created on compute driver 'vm'; this gateway runs 'docker'

Start and stop are refused with a precondition error that names both drivers, and exec is refused while the sandbox is not Ready:

$ openshell sandbox start v1
Error: code: 'The system is not in a state required ...', message: "sandbox was created by compute driver 'vm', but this gateway uses 'docker'; switch the gateway back to 'vm' to use it, or delete it"
$ openshell sandbox exec -n v1 -- true
Error: sandbox 'v1' is not ready (phase: Provisioning); wait for it to reach Ready state
$ openshell sandbox stop v1
Error: ... sandbox was created by compute driver 'vm', but this gateway uses 'docker' ...

Switching back to the VM driver restores it with no manual step: the start sweep restarts it (started=1 missing_backend=0 failed=0), v1 lists as Ready with no reason, and exec works:

NAME  CREATED              PHASE  REASON
v1    2026-11-02 19:43:06  Ready  -
BACK-ON-VM
aarch64

Result

PASS. Main marks the VM sandbox Error (BackendResourceMissing) when the driver changes and never recovers it; the PR head keeps it, shows ComputeDriverMismatch, refuses operations with a clear message, and the sandbox is fully usable after switching back.

Not covered: a Docker-created sandbox seen from a VM-driver gateway (the reverse direction; the Docker driver could not run sandboxes here), the Podman and Kubernetes drivers, deleting a foreign sandbox, and the other background paths the PR lists (deadline expiry, automatic restart); those are unit-tested in the PR. Suggested for the Linux lane if it has Docker and a VM driver on the same host.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-02T17:56:12.499372Z	1ec6696	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[vineethsai7]

I have read the DCO document and I hereby sign the DCO.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ec6696ec2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".