User Story

Context: I have been using my new Agent Gateway project. IT works with Docker but when I tried Podman I received the errors below. My repo:

Sorry bug report created and written at the behest of Open Code:

Problem Statement

Summary

With the corporate-proxy keys configured on [openshell.drivers.podman], sandbox egress through the upstream proxy fails on every CONNECT. Two distinct failure levels were verified across both 0.1.4 and 0.1.5, both ending in the agent workload's model call failing with network connection error:

• 0.1.4: the supervisor receives no proxy configuration at all — its model-fetch logs read proxy=none, and no proxy secret/bind appears in the sandbox containers.
• 0.1.5: staging improves — the supervisor container's command line includes --upstream-proxy http://192.168.10.5:8088 (an IP literal; an alias hostname host.docker.internal did not work, consistent with the new alias-validation in fix(network): bind hostname policy to supervisor DNS #4602), and the openshell-proxy-auth podman secret is created and attached to the supervisor container with byte-verified correct content (token:pass, the fresh 476-char JWT). However, at every turn-time CONNECT the upstream proxy still sees a Basic username that is not the staged JWT — pre-identity 407 invalid token: token is malformed: token contains an invalid number of segments — so the tunnel never authorizes.

The same proxy (the Agent Gateway egress proxy) accepts the identical credential file form from the docker driver's supervisor on the same machine (verified end-to-end: connect | allow | tunnel allowed rows), so the proxy side and the credential format are fine.

Versions
Component Version
openshell / openshell-gateway (CLI + gateway) 0.1.4 and 0.1.5 (both verified; openshell 0.1.5-1_amd64.deb)
podman 5.0.3, rootless, network_backend=netavark, rootless_network_cmd=pasta
Gateway proxy under test Agent Gateway egress proxy (HTTP CONNECT + Basic auth, PROXY_BASIC_AUTH=1)
OS Ubuntu Linux 24.04 (amd64), kernel 6.x

(Docker driver path on the same box ran green on 0.1.4 and 0.1.5 — reproduces the same expected behavior for the podman driver to match.)

Repro

1. Gateway config (user file, ~/.config/openshell/gateway.toml), schema v2:

2. [openshell]

3. version = 2

5. [openshell.gateway]

6. compute_driver = "podman"

7. guest_tls_ca = "…/tls/ca.crt"

9. [openshell.drivers.docker]

10. grpc_endpoint = "https://127.0.0.1:17670"

11. https_proxy = "http://127.0.0.1:8088"

12. proxy_auth_file = "/…/upstream-proxy-auth"

13. proxy_auth_allow_insecure = true

14. proxy_connect_by_hostname = true

16. [openshell.drivers.podman]

17. IP literal — alias hostname does NOT work on 0.1.5

18. https_proxy = "http://192.168.10.5:8088"

19. proxy_auth_file = "/…/upstream-proxy-auth"

20. proxy_auth_allow_insecure = true

21. proxy_connect_by_hostname = true

22. proxy_auth_file holds the user:pass form: :x (non-empty user, no control chars), mode 0600, fresh (decoded exp verified in the future at staging time).

23. Start the gateway (0.1.5), create a sandbox, apply an egress allowlist (verified loaded: ReportPolicyStatus … version=5 status=loaded), run one model/API call needing TLS CONNECT to an allowlisted host.

Observed (0.1.5)

• Supervisor container (5.x podman, separate openshell-supervisor-* container) command line carries the proxy URL: [/openshell-supervisor] [--workdir /sandbox --upstream-proxy http://192.168.10.5:8088 …]
• Podman secrets created per sandbox; supervisor container Config.Secrets includes openshell-proxy-auth-* (also openshell-token-, openshell-tls-ca-); byte-content check via podman secret inspect --showsecret confirms the fresh JWT is staged intact.
• Gateway audit at each turn: 407 | invalid token: token is malformed: token contains an invalid number of segments (pre-identity — the presented Basic username is not the staged JWT; sometimes preceded by/alternating with token expired from an older credential, even after re-mints).
• Workload: LLM request failed: network connection error (the CONNECT never authorizes; no allow row is ever produced).
• Also tested: enable_bind_mounts = true, gateway restarts (fresh staging each create), reversed credential field order, a short synthetic credential (testuser:testpass) — the proxy's verdict stays malformed in every case, so the presentation failure is independent of credential format/content.

Expected

The supervisor's upstream CONNECT should carry Proxy-Authorization: Basic base64(:) — i.e. the staged secret's content — so the Corporate-proxy egress contract works as documented for Podman.

Notes

• In the tested layout the supervisor runs in its own container (openshell-supervisor-), separate from the workload container (openshell-default--); the workload container has only the openshell-resolver-* secret attached (/run/secrets there is empty). Delivering the proxy credential to whichever process actually dials upstream (workload broker vs supervisor container) may be the root of the presentation gap — on 0.1.5 something dials the proxy, but with a credential that was never any of the staged contents.
• releases/tags/PR references for context: feat(sandbox,gateway): route sandbox egress through corporate HTTP proxy #2245 (corporate proxy routing), feat(sandbox,podman): trust corporate CA for https:// proxies and intercepted TLS #2512 (podman CA trust), fix(network): bind hostname policy to supervisor DNS #4602 (alias hostname policy), fix(sandbox): distinguish descriptor lookup outcomes #4494 (broker hard-deny of unverifiable descriptors), fix(podman): trust an operator-supplied additional CA for direct egress #3996 (podman additional_ca_bundle).
• Full evidence trail (audit rows, container mounts/secrets JSON, gateway logs, supervisor logs, staging timelines) available on request.

Impact / Why This Matters

Users openshell 0.1.5
openshell-gateway 0.1.5
podman version 5.4.2
Docker version 29.9.1, build dfc520d
Docker Compose version v5.6.0

Acceptance Criteria

My demo's 8 & 9 will work with Podman

Reproduction Steps

See description above or the document here;

Suggested UX (if applicable)

No response

Environment

openshell 0.1.5
openshell-gateway 0.1.5
podman version 5.4.2
Docker version 29.9.1, build dfc520d
Docker Compose version v5.6.0

Logs