bug(docker): daemon restart misclassifies a broken wait stream as supervisor exit #4878

User Story

I use OpenShell to run isolated agent workloads and need routine Docker maintenance not to destroy otherwise surviving sandboxes. I am reporting a colleague's controlled Linux staging reproduction and my source review, not claiming that I independently repeated the daemon restart.

Problem Statement

When Docker restarts with live-restore enabled while the OpenShell gateway remains running, the supervisor container can survive but the gateway's Docker wait connection breaks. In the reported reproduction, OpenShell classified that transport failure as ControlSupervisorExited, moved the sandbox from Ready to Error, and stopped the workload.

Losing the monitoring connection does not itself establish that the supervisor exited.

Impact / Why This Matters

A routine Docker daemon restart left the sandbox unusable. In this reproduction, start and stop rejected the Error state, and restarting the gateway did not recover it.

A comparison run that stopped/restarted the gateway along with Docker recovered to Ready with the same container ID. That comparison is limited evidence, not a general recovery guarantee.

Acceptance Criteria

• A Docker wait transport failure is distinguished from a confirmed supervisor exit.
• With live-restore enabled, a surviving, correctly identified supervisor whose required authenticated control/enforcement connection remains or becomes healthy can be monitored again without falsely terminating the workload.
• Recovery attempts are bounded; actual supervisor failure or inability to restore the required security boundary remains fail-closed. Container running status alone is not sufficient readiness evidence.
• Intentional stop/delete and concurrent lifecycle changes remain safe, without duplicate monitors or stale observations affecting a newer runtime.
• Tests cover daemon restart with and without live-restore, confirmed supervisor exit, prolonged daemon unavailability, and concurrent intentional shutdown.
• Successful recovery preserves sandbox identity/workspace and is followed by verified execution and policy enforcement.

Reproduction Steps

These describe the reported trigger. A minimal stock-image reproducer and exact runtime versions still need confirmation.

Use a disposable Linux Docker-backed OpenShell deployment, not a host with unrelated workloads:

1. Enable Docker live-restore before the test and record the effective setting.
2. Create a long-running sandbox with a working workload image and policy; wait for Ready and verify execution.
3. Confirm the OpenShell gateway will remain running across a Docker daemon restart. Record the gateway PID and exact owned workload/supervisor container identities.
4. Restart the Docker daemon on that disposable host.
5. Observe the gateway's wait-connection failure, sandbox phase, and container state.
6. Check whether sandbox start/stop and a gateway restart can recover the retained sandbox without deleting it or modifying its state database.

Observed sequence: gateway PID unchanged; Docker live-restore preserved containers through the daemon restart; a wait connection error was followed by Ready -> Error and a gateway-initiated workload stop. The workload then exited with code 137, which does not establish an OOM cause.

Environment

• Reported runtime: Linux host, Docker driver, Docker live-restore enabled, gateway kept running across daemon restart.
• Exact OpenShell CLI/gateway/supervisor versions, Docker version, OS release, and image digests for this reproduction: pending confirmation. No version is inferred from repository pins.
• Latest stable checked while filing: v0.1.5. This report does not claim a fresh reproduction on that release.
• Source inspected: 72e4eaa89c892688a59843096268a3c302c23797. This is source-review evidence, not the reproduced runtime's asserted commit.

Logs

Minimal sanitized excerpts from the reported reproduction:

Docker supervisor container exited unexpectedly
reason=ControlSupervisorExited
failed to wait for Docker supervisor container: error reading a body from connection

Sandbox phase changed ... old_phase=Ready new_phase=Error
Stopped Docker sandbox after control supervisor failure

Recovery failures on that retained runtime:

sandbox must be Stopped, Completed, or a failed main-process Error to start (current phase: Error)
sandbox must be Ready to stop (current phase: Error)

Source finding / proposed approach for maintainer confirmation

At the inspected revision, docker_supervisor_wait_failure() maps generic wait errors and unexpected stream termination to ControlSupervisorExited. The monitoring task then invokes the runtime-failure handler.

Please confirm the intended contract: on a transport failure, reconnect with bounded retries, inspect the exact owned supervisor, and safely reattach monitoring only when identity and control/enforcement health are verified. Do not treat a real exit as a transient disconnect or weaken containment while recovery is uncertain.

Related work and scope

• bug(docker): sandbox stays in Error after its container restarts, and stop/start cannot recover it #4828 covers a backing-container restart followed by supervisor reconnection and a persistent Error state.
• bug(docker): recovery stop aborts when supervisor-failure watch restores Error #4727 covers a recovery-stop race that restores Error while stopping; it also documents newer recovery behavior, so the rejected start/stop commands above must not be generalized to every current build.

This issue focuses on preventing false supervisor-exit classification when the Docker daemon disconnects but live-restore preserves the supervisor. Recovery for sandboxes already in Error should coordinate with those existing issues rather than duplicate them. No installer workaround or broad lifecycle redesign is requested here.