Skip to content

feat(observability): expose supervisor version and workload image IDs in sandbox status - #5140

Open
vineethsai7 wants to merge 2 commits into
NVIDIA:mainfrom
vineethsai7:feat/4272-v1
Open

vineethsai7 wants to merge 2 commits into
NVIDIA:mainfrom
vineethsai7:feat/4272-v1

Conversation

@vineethsai7

@vineethsai7 vineethsai7 commented Nov 2, 2026 •

Copy link
Copy Markdown

Summary

Draft proposal: expose per-sandbox supervisor build information and the workload image ID observed by Docker or Podman in GetSandbox and openshell sandbox get --output json. This is a first slice of the runtime evidence requested in #4272; the issue remains state:triage-needed, and design feedback is welcome.

Related Issue

Part of #4272

Changes

  • Add a read-only SandboxStatus.runtime block with independent supervisor and workload image fields.
  • Have the dedicated supervisor report its own build version over its authenticated control session. Preserve the version reported by the active supervisor across driver status refreshes.
  • Carry Docker's container image ID and Podman's pinned workload image ID through the existing driver status path. Keep an observed ID across lightweight updates only for the same container and image reference.
  • Show the fields in CLI JSON/YAML output and the curated Go SDK, and document their meaning and absence rules.

Design

The driver already observes the container image ID, and the supervisor already has its build version. The gateway stores those observations in sandbox status through the existing driver snapshot and supervisor session paths. digest_kind: image_id distinguishes a container image ID from a registry manifest digest. The reference is the effective template image and can be a moving tag.

This uses the current per-sandbox status path rather than adding a new RPC, gateway interceptor, or supervisor middleware. An interceptor sees the requested image reference but cannot prove what the driver booted; middleware observes network activity, not the runtime. GetGatewayInfo describes the gateway rather than the supervisor serving a particular sandbox. The VM driver's resolved_identity.resource_digest currently identifies its rootfs cache layout, so exposing it as a source image digest would give the field the wrong meaning.

Example

openshell sandbox get my-sandbox --output json | jq '.runtime'
{
  "supervisor": {"version": "0.1.6", "build_commit": null},
  "workload_image": {
    "reference": "example/agent:latest",
    "digest": "sha256:...",
    "digest_kind": "image_id"
  }
}

The same block is available as Sandbox.status.runtime from GetSandbox. No configuration change is needed.

Compatibility

The protobuf additions are optional and preserve existing field numbers. Older supervisors and drivers leave their respective fields absent; CLI output shows null when no runtime evidence is available. Existing policy hash and gateway version fields remain in their current locations. The supervisor version is reported by the process on an authenticated session; it is not an independent binary attestation. No default runtime behavior changes.

Follow-ups

Review follow-up

  • F1 (P1): Fixed in 12bbf17. Podman status and retained-workload refresh now use the top-level immutable Image ID; the regression separates it from Config.Image and verifies absence when no ID is reported.
  • F2 (P2): Fixed in 12bbf17. A stopped image change clears the old ID atomically, and delayed snapshots of a still-stopped container cannot relabel it with the new reference.
  • C1: Not applicable. The supervisor build is deliberately last-observed evidence after disconnect; a new authenticated session replaces or clears it.
  • GitHub feedback: No inline review comments or submitted reviews. The runner-validation notice and DCO acknowledgment require no code change.

Release Notes

openshell sandbox get --output json and the Go SDK now report the dedicated supervisor's version and the Docker or Podman workload image ID when those components supply them.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated for driver IDs, status refresh and replacement, supervisor reconnects, CLI JSON, protobuf inventory, and Go SDK conversion and copying
  • E2E tests added/updated (no live runtime test host was provided)

All commands ran on the shared build box through dcb at nice level 15:

  • cargo fmt --all -- --check: passed.
  • Clippy with --all-targets -- -D warnings for the six touched Rust crates: passed.
  • Tests for the six touched Rust crates: passed. The server library passed 2,607 tests with 38 ignored. The supervisor-process library passed 132 tests in a serial rerun after three reconnect tests stalled in a parallel run.
  • go test -race ./... in sdk/go: passed.
  • buf lint and the generated Go protobuf comparison: passed.
  • Docs navigation check: passed. Fern check: 0 errors, 3 warnings.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Manual verification

Result: PASS. Repeated the Docker and rootless Podman cases on upstream main be5e49499 and PR head 12bbf172d1. Both main sandboxes reached Ready without a runtime field. Both PR sandboxes reported the supervisor version and the workload container's actual image ID in sandbox get -o json and -o yaml.

Environment

  • RHEL 9.8 x86_64 stack host, Docker 29.8.1 and rootless Podman 5.8.2. All runs used one disposable local user, a private HOME, and private loopback gateway ports. The installed DefenseClaw and host OpenShell 0.1.5 service on port 17670 were untouched.
  • Each commit's openshell, openshell-gateway, openshell-supervisor, and openshell-sandbox binaries were built on Ubuntu 24.04 through dcb with nice -n 15 cargo build --release --locked -p openshell-cli -p openshell-gateway -p openshell-supervisor -p openshell-sandbox --features openshell-gateway/vendored-z3. Main was built from a detached checkout at be5e49499; the PR build used 12bbf172d1. The CLI, gateway, supervisor, and sandbox for each case came from the same commit.
  • Each supervisor binary was packaged in a local image based on the distroless Debian 13 supervisor image, with libgcc_s.so.1; each sandbox binary was packaged in a scratch image. These match the relevant binary paths and entrypoints in deploy/docker/Dockerfile.supervisor and Dockerfile.sandbox. The gateway config selected those local images and docker.io/library/ubuntu:24.04 as the workload image, with compute_driver = "docker" or "podman", plaintext on loopback, and a private database. The Ubuntu-built gateway and CLI used an Ubuntu 24.04 glibc loader on RHEL.
  • Rootless Podman used its user socket and a temporary, per-user cpu cgroup delegation. The delegation was removed during cleanup.
  • The synced PR build context had no usable Git tag metadata, so openshell-supervisor --version reported 0.0.0. The status reports that actual supervisor value; it is not a missing version field.

Steps

For each commit, start a gateway with the selected driver and its matching local supervisor and sandbox images, then create a sandbox. Run the status commands from a second shell while the sandbox is Ready:

openshell gateway add http://<loopback>:<private-port> --name <case>
openshell -g <case> sandbox create --name <sandbox> --no-tty -- sleep 600
openshell -g <case> sandbox get <sandbox> -o json
openshell -g <case> sandbox get <sandbox> -o yaml
docker inspect --format '{{.Image}}' <docker-workload-container>
podman inspect --format '{{.Image}}' <podman-workload-container>

The inspect commands target the workload containers, not their supervisor containers. The same procedure was run for main/Docker, main/Podman, PR/Docker, and PR/Podman.

Before: main be5e49499

Both main sandboxes were Ready. Neither output format contained a runtime block:

driver    sandbox get -o json: has("runtime")    sandbox get -o yaml: runtime: lines
Docker    false                                 0
Podman    false                                 0

For example, sandbox get -o json | jq '{phase, has_runtime: has("runtime")}' returned {"phase":"Ready","has_runtime":false} for each driver. sandbox get -o yaml showed phase: Ready and no runtime: line.

After: PR 12bbf172d1, Docker

sandbox get -o json (runtime excerpt):

"runtime": {
  "supervisor": { "version": "0.0.0", "build_commit": null },
  "workload_image": {
    "reference": "docker.io/library/ubuntu:24.04",
    "digest": "sha256:fe81560990fae86dc4743202b2cb4d3df98d0bcf949173cecef15ce1127ef852",
    "digest_kind": "image_id"
  }
}

sandbox get -o yaml (runtime excerpt):

runtime:
  supervisor:
    version: '0.0.0'
    build_commit: null
  workload_image:
    reference: docker.io/library/ubuntu:24.04
    digest: sha256:fe81560990fae86dc4743202b2cb4d3df98d0bcf949173cecef15ce1127ef852
    digest_kind: image_id

docker inspect --format '{{.Image}}' <docker-workload-container> returned sha256:fe81560990fae86dc4743202b2cb4d3df98d0bcf949173cecef15ce1127ef852, exactly matching both status values.

After: PR 12bbf172d1, rootless Podman

sandbox get -o json (runtime excerpt):

"runtime": {
  "supervisor": { "version": "0.0.0", "build_commit": null },
  "workload_image": {
    "reference": "docker.io/library/ubuntu:24.04",
    "digest": "d9b8c34d61a9b9a3c44cab883d7d2f2619d674966ab520186f799cf32e3e0377",
    "digest_kind": "image_id"
  }
}

sandbox get -o yaml (runtime excerpt):

runtime:
  supervisor:
    version: '0.0.0'
    build_commit: null
  workload_image:
    reference: docker.io/library/ubuntu:24.04
    digest: d9b8c34d61a9b9a3c44cab883d7d2f2619d674966ab520186f799cf32e3e0377
    digest_kind: image_id

podman inspect --format '{{.Image}}' <podman-workload-container> returned d9b8c34d61a9b9a3c44cab883d7d2f2619d674966ab520186f799cf32e3e0377, exactly matching both status values. podman inspect --format '{{.Config.Image}}' returned docker.io/library/ubuntu:24.04, the reference. The PR status contains the immutable ID, not that reference.

Result and cleanup

Docker: PASS. Podman: PASS. In both runtimes, the supervisor version and workload image ID appear in JSON and YAML only with the PR head; the ID equals the workload container's engine inspect .Image. Afterward the sandboxes, gateways, local proof images, disposable user and HOME, private staging, and temporary systemd delegation were removed. Readback found no proof processes, listeners, or containers; the host OpenShell 0.1.5 listener remained on port 17670 with its original PID.

Signed-off-by: Vineeth Sai Narajala <vnarajal@cisco.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@vineethsai7

Copy link
Copy Markdown
Author

I have read the DCO document and I hereby sign the DCO.

…idence

Signed-off-by: Vineeth Sai Narajala <vnarajal@cisco.com>
@vineethsai7
vineethsai7 marked this pull request as ready for review November 3, 2026 04:23
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T04:28:16.522096Z 12bbf17 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant