Skip to content

fix: keep VM sockets short and validate Unix socket paths - #4184

Open
shiju-nv wants to merge 3 commits into
NVIDIA:mainfrom
shiju-nv:fix/vm-driver-socket-path-diagnostic-4178
Open

shiju-nv wants to merge 3 commits into
NVIDIA:mainfrom
shiju-nv:fix/vm-driver-socket-path-diagnostic-4178

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The gateway can now start with a long VM state_dir: its managed driver's control socket lives in a private, randomly named /tmp directory owned by the managed process, while persistent state stays in the configured directory. VM sandbox sockets use fixed-length directory names so accepted 128-byte IDs fit. A shared filesystem pathname length check gives configured driver/extension endpoints and OpenShell listeners an error naming the path, its byte length and the platform limit before filesystem preparation or connection retries.

Related Issue

Closes #4178

Changes

  • Add openshell_extension_core::unix_socket_path::check, re-exported from core: 107 bytes on Linux and 103 on macOS, excluding the terminating NUL.
  • Use short private paths for gateway-managed VM sockets and compact VM sandbox socket directories. Preserve socket permissions, peer authentication, collision handling and descriptor-relative cleanup. Drop, startup failure and cancellation release the temporary directory.
  • Check configured paths in the shared external-driver binder, standalone VM and credential drivers, gateway remote-driver configuration, extension transport, supervisor readiness/SSH listeners, and sandbox boundary listeners.
  • Preserve native Linux abstract-socket validation. Existing connectors reuse validated listener/configuration paths. External Docker/Podman engine paths and discovery, SPIFFE endpoints and Python endpoints retain their current behavior; those services own their socket locations.
  • Update configuration docs, VM driver docs and development-script comments.

Temporary directories are cleaned up through normal ownership; SIGKILL can leave an unused directory behind. Subsequent launches allocate a fresh private directory.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable): no real VM boot performed

18 focused tests passed on macOS: shared validator/extension checks; long VM sandbox IDs; public gateway startup with an overlong state directory; cleanup on drop, early exit and cancellation; collision handling; configured driver paths; and standalone VM argument validation. Scoped Clippy passed with warnings denied across all targets of the 11 affected packages, alongside formatting, license-header and shell-syntax checks. Independent local Gator review passed. These checks did not boot a real VM or exercise Linux runtime behavior.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

…rror

The VM driver listens on <state_dir>/run/compute-driver.sock. When that path exceeds the platform's Unix socket limit the driver exits with a bare 'path must be shorter than SUN_LEN' and the gateway can only report the driver's exit status.

Check the path in prepare_compute_driver_socket_path before any directory is created, and name the path, its length, the limit and the state_dir setting in the error. The limit is 107 bytes on Linux and 103 elsewhere. Note the constraint in the VM driver configuration docs.

Closes NVIDIA#4178

Signed-off-by: Shiju <shiju@nvidia.com>
Comment thread crates/openshell-gateway/src/vm.rs Resolved
@drew

drew commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

Could we extend the approach from #3544 to the remaining gateway-to-driver compute-driver.sock? That PR moved the per-sandbox control.sock and ssh.sock into private, randomly named directories under /tmp so socket paths no longer depend on the depth of the persistent state directory.

The diagnostic here is useful, but users with long state paths still cannot start the gateway. Please allocate a short, gateway-owned socket directory under /tmp, pass the resulting path through the existing --bind-socket argument, and tie directory cleanup to the managed driver lifecycle. Preserve the existing socket permissions and peer checks, and follow #3544's private-directory and collision-handling approach. Keep persistent VM state at the configured state_dir.

Please also cover startup with an overlong state_dir to verify that the control socket no longer inherits that limit. Use /tmp deliberately, as in #3544, since macOS's $TMPDIR can itself be too long.

Allocate a private random directory under /tmp and retain it with the managed driver process. Keep persistent VM state at the configured path and cover long-path startup, collision handling, and cleanup.

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv

shiju-nv commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

The PR initially focused on giving users a clearer error message. The gateway now creates a private, randomly named directory under /tmp for compute-driver.sock.

@elezar elezar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a question: Are there other paths where we could expect this to happen? Should we expose a shared check to sanitize these paths?

@benoitf

benoitf commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

it's all the *.sock files

@shiju-nv

shiju-nv commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Audited source:

Socket family Socket pathname and creator Source Current behavior and proposed treatment
Gateway-managed VM compute-driver.sock /tmp/os-gw-<euid>-<32-hex-random>/compute-driver.sock; gateway chooses the path, VM driver binds it. crates/openshell-gateway/src/vm.rs:344, :490; crates/openshell-server/src/compute/mod.rs:362 Already short and private; consolidate allocation and validate the final endpoint. Keep the directory alive through managed subprocess ownership.
VM sandbox control.sock and ssh.sock /tmp/os-<euid>-<32-hex-random>/<sandbox-id>/control.sock and .../<sandbox-id>/ssh.sock; VM runtime creates control, supervisor creates SSH. crates/openshell-driver-vm/src/driver.rs:5977, :6016, :951, :1539 Short random root, but full sandbox ID is still a variable-length leaf. Compact the leaf and share address validation. Preserve directory-fd operations.
Standalone VM --bind-socket Exact pathname supplied through --bind-socket <path>; VM driver creates the socket there. crates/openshell-driver-vm/src/main.rs:335, :423 Validate before driver initialization and socket preparation. Preserve supplied path and expected-peer-PID authentication.
Docker, Podman and Kubernetes external-driver listeners Exact --bind-socket <path> pathname; the selected driver creates it. No fixed basename or directory. crates/openshell-core/src/external_driver_socket.rs:16; driver main.rs:70, :253, :308 respectively Shared binder prepares/removes/binds without an address preflight. Validate at entry, before any side effect.
Remote compute-driver connector Configured compute driver socket_path (including CLI overrides); remote driver creates it, gateway only connects. crates/openshell-server/src/compute/driver_config.rs:191; compute/mod.rs:5807 Nonempty-path check followed by readiness retries. Reject invalid addresses before the readiness loop.
Credential driver launch/connect and standalone Vault/Kubernetes Secrets bind Configured credential driver socket_path, passed as --bind-socket <path> for managed launches; credential driver creates it. crates/openshell-server/src/credentials.rs:1016, :1565, :1833; driver main.rs:73, :50 respectively Absolute-path check is insufficient. Validate at configuration ingestion, launch/connect entry and standalone bind entry. Launched credentials still use explicitly configured paths.
Extension/interceptor/middleware unix:// transport The pathname after unix:// in the configured endpoint; external extension service creates it. crates/openshell-extension-core/src/transport.rs:152, :178 Checks nonempty/absolute paths, then connects. Use shared validation after parsing the scheme.
Docker/Podman engine discovery and connect Docker: DOCKER_HOST Unix path, /var/run/docker.sock, $HOME/.docker/run/docker.sock, $XDG_RUNTIME_DIR/docker.sock. Podman: OPENSHELL_PODMAN_SOCKET, $XDG_RUNTIME_DIR/podman/podman.sock, /run/user/<euid>/podman/podman.sock, $HOME/.local/share/containers/podman/machine/podman.sock, or CLI-discovered path. Engine creates these sockets; explicit driver socket_path can select another. crates/openshell-driver-docker/src/lib.rs:820; crates/openshell-driver-podman/src/socket_discovery.rs:23; client.rs:323; crates/openshell-core/src/local_api_socket.rs:50 HOME, runtime directory and explicit overrides can be long. Reject explicit invalid paths; skip invalid discovery candidates while retaining a useful diagnostic. The container engine owns these sockets.
Supervisor SSH listen/relay Configured SSH pathname, commonly /run/openshell/ssh.sock; VM host path is the sandbox SSH path above. Supervisor binds it. Linux @name selects an abstract address and creates no filesystem socket. crates/openshell-supervisor-process/src/ssh.rs:100; supervisor_session.rs:858; unix_socket.rs:14 Validate filesystem paths before preparation. Linux @name addresses require abstract-name validation, with existing runtime conversion preserved.
Supervisor health/readiness --health-socket-path <path>; drivers supply /run/openshell/health.sock (Docker/Kubernetes) or /run/openshell/supervisor-health.sock (Podman). Supervisor creates it. crates/openshell-supervisor/src/lib.rs:112, :247, :297 Defaults are short; configured paths need preflight before prepare/bind/connect.
Sandbox boundary listener and backend connector Listener transport pathname; common boundary mount target /.openshell/channel/sandbox/control.sock, with sidecar control path /run/openshell-sidecar/control.sock. VM host control path is above. The runtime listener creates the socket; backend connects through its descriptor/mount mapping. crates/openshell-sandbox/src/boundary_server.rs:3283; crates/openshell-sandbox-backend/src/runtime.rs:1933 Validate supplied paths before stale removal or retries. Keep mount mappings and descriptor endpoints consistent.
SPIFFE workload socket and Python gateway unix: endpoint Configured SPIFFE Workload API pathname, projected into containers as /spiffe-workload-api/<socket-basename>; SPIRE creates the host socket. Python uses the pathname in its configured gateway unix: endpoint; the gateway service creates it. crates/openshell-core/src/spiffe.rs:52; python/openshell/sandbox.py:232, :678, :691 External services own these addresses. Apply Rust validation where the host actually dials a socket. Python consistency is separate work; preserve its endpoint identity.

@shiju-nv

shiju-nv commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Would the team prefer expanding this PR to cover the other socket paths? I'm thinking of adding shared address validation before filesystem changes or connection retries, and consolidating private short-directory allocation. The validator could live in extension-core and be re-exported from core.

VM sandbox sockets would also need compact directory names so long sandbox IDs don’t exceed the limit. Alternatively, I can keep this PR focused and address the broader changes in a follow-up.

@github-actions

Copy link
Copy Markdown

This pull request has had no activity for 14 days and is now marked stale.

@github-actions github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Oct 20, 2026
Merge current main while preserving the managed VM driver's private
socket directory and persistent state location. Share pathname length
checks across driver, extension, supervisor and sandbox endpoints, and
use compact VM sandbox socket directory names for long sandbox IDs.

Closes NVIDIA#4178

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv shiju-nv changed the title fix(gateway): reject an overlong VM driver socket path with a clear error fix: keep VM sockets short and validate Unix socket paths Oct 30, 2026
@github-actions github-actions Bot removed the state:stale Inactive item at risk of automatic closure. label Oct 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(vm-driver): driver startup fails with a bare "path must be shorter than SUN_LEN" and does not name the path

4 participants