Skip to content

bug(windows): sandbox processes outlive a killed gateway and keep its port bound #5298

Open

Description

@ashwind-nvidia

User Story

I work on OpenShell's native Windows support and run the gateway with the built-in MXC driver and real sandboxes on Windows 11 x64. I directly encountered this when I hard-killed a gateway that had three running sandboxes: the sandboxes' processes kept running and a new gateway could not start on the same port. This affects anyone who restarts or loses the gateway while sandboxes are running.

Problem Statement

With the MXC driver built into the gateway, the gateway starts wxc-exec, openshell-supervisor and openshell-windows-sandbox for each sandbox. If the gateway process is killed (crash, taskkill, logoff), those processes are not ended. They keep running with no owner, and they keep the gateway's listening socket open, so the port stays bound. A new gateway then fails to start with "failed to bind ... Only one usage of each socket address (os error 10048)".

Two things combine to cause it:

  1. Nothing ties the lifetime of the sandbox processes to the gateway process.
  2. The gateway's listening sockets are created with plain socket() (mio 1.2.0 on Windows), which leaves the handle inheritable. Child processes are started with handle inheritance enabled, so each sandbox process holds its own copy of the socket and keeps the port bound after the gateway dies.

Impact / Why This Matters

After a gateway crash or forced stop, the operator has to find and kill the leftover sandbox processes by hand before a new gateway can bind its port. Until then the sandboxes run ungoverned: no gateway enforces their policy or can delete them, and the host cannot reconcile them.

Acceptance Criteria

  • After the gateway is hard-killed, none of the sandbox processes it started (wxc-exec, openshell-supervisor, openshell-windows-sandbox) are still running.
  • Immediately after the kill, the gateway port can be bound again and a new gateway starts on the same port.
  • The gateway's main, health and metrics listeners are not inheritable by child processes.
  • Creating, running and deleting sandboxes behaves as before.

Reproduction Steps

  1. On Windows with the MXC driver, start the gateway on port 17680 with the built-in MXC driver (OPENSHELL_COMPUTE_DRIVER=mxc).
  2. Create three long-running sandboxes with openshell sandbox create --detach and confirm all three are Ready.
  3. Hard-kill the gateway: Stop-Process -Id <gateway pid> -Force.
  4. After 5 and again after 25 seconds, list openshell-supervisor, wxc-exec and openshell-windows-sandbox processes, and try to bind port 17680.
  5. Start the gateway again on port 17680.

Actual: nine sandbox processes (three of each) are still running after 25 seconds, the port cannot be bound, and the new gateway exits with the bind error. Killing all openshell-windows-sandbox processes releases the port. Observed in 5 of 5 runs.

Expected: no sandbox process survives, the port can be bound, and the new gateway starts.

Environment

  • OpenShell: openshell 0.1.5-dev.40+g5b9e5b91a (windows branch at 5b9e5b9)
  • OS: Windows 11 x64, native
  • Deployment: gateway with the built-in MXC driver, backend = "process_container", plaintext gateway on loopback

Logs

Error: x transport error
  `-> failed to bind to 127.0.0.1:17680: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions