Skip to content

bug(docker): containerized gateways reuse helper container names openshell-supervisor-extract-1-<seq>, so concurrent starts on one engine can collide #5094

Open

Description

@cv

User Story

As an operator running more than one OpenShell gateway on a Docker engine, I want each gateway's helper containers to have unique names, so that two gateways starting, or starting sandboxes, at the same time don't fail.

Problem Statement

temp_extract_container_name() in crates/openshell-core/src/driver_utils.rs builds helper container names from the process ID and a per-process counter:

format!("openshell-supervisor-extract-{pid}-{seq}")

Its comment says this avoids collisions across concurrent gateway starts. That holds only for gateways running directly on the host. A gateway running in a container, as the published ghcr.io/nvidia/openshell/gateway image does, is PID 1 in its own PID namespace. So every containerized gateway on an engine generates the same names, starting at openshell-supervisor-extract-1-0.

The Docker driver uses these names in three places in crates/openshell-driver-docker/src/lib.rs, at ac7f615c and on main:

  • supervisor binary extraction at gateway start (extract_runtime_path_archive): openshell-supervisor-extract-1-<seq>;
  • workload identity resolution for each sandbox image: …-<seq>-identity;
  • host-gateway address resolution: …-<seq>-host-gateway.

Each one calls create_container with a fixed name and returns an error if creation fails. Nothing retries, and nothing falls back to another name. The Podman driver uses the same function.

Impact / Why This Matters

If two gateways on one engine start, or start sandboxes, within the same fraction of a second, the second create_container fails with a name conflict. The gateway start or the sandbox creation then fails with:

  • failed to create extractor container from '<image>', or
  • create Docker identity resolver container: ….

This hits two deployments applied at the same time on one host, a gateway restarting while another starts, and any test suite that runs gateways in parallel. NVIDIA/NemoClaw keeps its Docker live tests serial for this reason.

Acceptance Criteria

  • Helper container names are unique per call across processes and containers, for example with a random suffix or a UUID instead of the PID. Alternatively, creation retries with a fresh name when it returns a conflict.
  • The -identity and -host-gateway variants get the same fix.
  • Two containerized gateways starting at the same moment on one engine both succeed.

Reproduction Steps

Observed: on one Docker engine, the Docker events for a run of NemoClaw's Docker live suite show every gateway container creating openshell-supervisor-extract-1-0 within about 0.3 s of starting, and creating openshell-supervisor-extract-1-1-identity (or -1-2-identity) when it starts a sandbox. In that run the gateways started one after another, so the names didn't overlap, and no collision was triggered.

To trigger a collision:

  1. On one Docker engine, start two containerized gateways at the same moment, or have two running gateways create sandboxes at the same moment.
  2. One of the overlapping helper-container creations fails with a name conflict.

The failure above follows from the code; it hasn't been triggered deliberately yet.

Environment

OpenShell v0.1.6 (ac7f615cd12d5cc49cb16eb1f5f10ffeb1e90f47); the same code is on main. Docker driver, gateway image ghcr.io/nvidia/openshell/gateway@sha256:f0b36dba…, linux_arm64.

Activity

  1. harsh543 commented on Oct 31, 2026

    @harsh543

    Hi @cv — I saw NVIDIA/NemoClaw#14043 merge, and its live-test note still keeps Docker tests serial because of this issue. It looks like #5094 itself remains open, so I would like to help with a focused, non-overlapping OpenShell contribution.

    My proposed scope is:

    • make helper-container names unique in the shared temp_extract_container_name() path so Docker and Podman, including the -identity and -host-gateway variants, inherit the fix;
    • add deterministic unit coverage for uniqueness across simulated gateway processes or namespaces;
    • add or extend a multi-gateway Docker regression that starts gateways or sandboxes concurrently on one engine.

    Before I start, is anyone already implementing the naming change? If so, I am happy to take a clearly separate slice such as the Docker/Podman regression coverage or retry-on-name-conflict hardening. Which split would be most useful to you?

  2. cv commented on Oct 31, 2026

    @cv
    ContributorAuthor

    Thanks, @harsh543, that would be very welcome. Nobody is working on it on our side, so please take the whole scope you listed:

    • unique names in temp_extract_container_name(), so the Docker and Podman drivers and the -identity and -host-gateway variants all inherit the fix;
    • deterministic unit coverage across simulated gateway processes or namespaces;
    • a multi-gateway Docker regression that starts gateways or sandboxes at the same time on one engine.

    Retry on a name conflict is optional once names are unique; skip it unless it falls out naturally.

    When it lands in an OpenShell release, NemoClaw can run its Docker live tests two at a time again. Feel free to mention me on the PR.

  3. harsh543 commented on Nov 1, 2026

    @harsh543

    Thanks @cv. I have a working change for the full scope you listed and wanted to share where it stands before opening a PR.

    What it does: temp_extract_container_name() now appends a per-process random nonce, openshell-supervisor-extract-{pid}-{seq}-{nonce}, so the Docker (including -identity and -host-gateway) and Podman helpers stay unique across PID namespaces. No call-site changes. Retry-on-conflict is left out as you suggested.

    Validation so far (local, macOS arm64, Docker Desktop):

    • fmt, clippy -D warnings, and the full openshell-core, docker, and podman driver suites pass.
    • Unit tests simulate gateways that share the same pid and counter.
    • A live multi-gateway test creates 24 helper containers concurrently on one engine and passes. With the old naming forced back in, the same test fails with Docker 409 Conflict, so it does reproduce the issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions