Skip to content

bug(sandbox): agent Wait does not return when shutdown cancels the reaper #5329

Open

Description

@elezar

User Story

I operate OpenShell sandboxes on Kubernetes and delete them regularly.
I directly encountered openshell-sandbox always spending its full 2 s shutdown bound after SIGTERM when the supervisor holds an agent Wait exchange open.
This affects how quickly sandbox deletion and shutdown complete.

Problem Statement

The supervisor keeps an agent Wait exchange open for the life of the sandbox. ManagedProcess::wait blocks on a condition variable that only the agent's async reaper task wakes. When runtime shutdown cancels the reaper before it records the agent's exit, Wait never returns.

#5279 bounds runtime shutdown to 2 s so the boundary exits instead of hanging until SIGKILL. That stopgap leaves the lost wake-up in place, so in this situation the boundary always uses the full 2 s bound before exiting.

Impact / Why This Matters

Deleting a Kubernetes sandbox takes about 4-6 s with #5279, of which 2 s is the shutdown bound, instead of exiting as soon as the workload is gone. The wait is deterministic, not occasional. Without a bound (before #5279) the same condition left the process alive until SIGKILL at the 30 s grace period.

Acceptance Criteria

  • When the supervisor holds an agent Wait exchange open and the boundary receives SIGTERM, the agent's exit is recorded and the Wait returns, without relying on the runtime shutdown timeout.
  • The boundary exits well inside the 2 s bound in that situation, covered by a test.

Reproduction

  1. Run a Kubernetes sandbox with a running agent so the supervisor holds an open Wait exchange.
  2. Delete the sandbox and send SIGTERM to openshell-sandbox.
  3. Observe that the boundary exits only after the 2 s runtime shutdown bound elapses (gdb on a hung run, pre-fix(sandbox): bound runtime shutdown after SIGTERM #5279, showed the thread in serve_one -> dispatch -> BoundaryRuntime::wait -> ManagedProcess::wait).

Additional Context

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:sandboxSandbox runtime and isolation work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions