[jquesnelle]

Summary

On Windows, the combined CA bundle that OpenShell gives an MXC sandbox held only the sandbox CA. As a result, clients that honor SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE or GIT_SSL_CAINFO could not verify any server behind a tls: skip endpoint. Linux is unaffected because the supervisor there reads the distribution's bundle from one of SYSTEM_CA_PATHS, in the sandbox's container or, for the VM driver, on the host; Windows keeps its roots in the certificate store, so those paths find nothing. This change builds the system part of the bundle from the Windows certificate store, so the bundle holds the platform roots followed by the sandbox CA, as it does on Linux.

Related Issue

Closes #5060

Changes

• read_system_ca_bundle() on Windows reads the roots that rustls-native-certs loads and returns them as PEM. Those roots come from the current user's ROOT store, which also holds the machine, Group Policy, enterprise and third-party roots; only time-valid roots usable for server authentication are returned. This is the same source the existing system-ca-roots feature uses, so SSL_CERT_FILE or SSL_CERT_DIR in the supervisor's own environment replaces the store, as it does there, and the certificates loaded from those paths are not filtered.
• If the store cannot be opened, or SSL_CERT_FILE or SSL_CERT_DIR names a path that cannot be read, the error is logged with warn! and the bundle holds only the sandbox CA, as it does today. rustls-native-certs 0.8.3 unwraps its per-certificate usage and validity queries, so a failure in one of them panics; system-ca-roots builds already run that code.
• If the store roots encode to more than 896 KiB of PEM, they are left out with a warn!, and the bundle again holds only the sandbox CA. The bundle reaches the sandbox boundary in the StartAgent control frame, which encode_frame limits to 1 MiB (MAX_CONTROL_FRAME_BYTES), so without this check an oversized store or SSL_CERT_FILE would fail every sandbox start. Sizes are below.
• The Linux and macOS read_system_ca_bundle() and SYSTEM_CA_PATHS are unchanged; they are now compiled only for non-Windows targets.
• pem_encode_certificates() writes DER certificates as CERTIFICATE PEM blocks with 64-column base64 lines.
• The other callers of read_system_ca_bundle() get the same roots. In the default bundled-ca-roots build, the proxy adds them to its upstream trust and to the trust it uses for an HTTPS upstream proxy, as it does with the distribution bundle on Linux.
• Tests:
  • pem_encode_certificates_round_trips_der and encode_system_ca_bundle_leaves_out_bundles_over_the_limit run on every platform.
  • windows_sandbox_bundle_includes_certificate_store_roots runs on Windows. On windows without this fix it fails with the bundle holds 0 certificates before the sandbox CA; the store has 47 roots.
  • windows_system_ca_bundle_over_the_limit_is_left_out runs on Windows. It points SSL_CERT_FILE at one file under the limit and one over it.
• crates/openshell-driver-mxc/README.md now states what the injected trust variables contain on Windows, and the on-demand and size limits described below.

Bundle sizes, from the encoder in this change:

Roots	Count	PEM bytes
Test host, CurrentUser\ROOT after filtering	47	70,344
Microsoft's trusted root program (certutil -generateSSTFromWU), time-valid and marked for server authentication	333	621,027
Microsoft's trusted root program, all time-valid roots	457	823,019

GitHub-hosted Windows images import the whole program into LocalMachine\Root when they are built (Install-RootCA.ps1 in actions/runner-images, used by the windows-2025 and windows-11-arm64 templates), so the Windows CI jobs run the store test with a bundle between the last two rows. The 896 KiB limit is above both and leaves about 110 KB of the frame for the agent spec, the policy and the provider environment.

Limits, the same as on the system-ca-roots path:

• rustls-native-certs does not read the Disallowed store. On the test host that store was empty.
• Windows adds a Microsoft program root to the store the first time a Schannel client needs it. A root that the host has not used yet is not in the bundle; the next sandbox gets it once any Schannel client on the host has fetched it. The test host had 47 roots, against 457 time-valid roots in the program.

The bundle holds the Windows store and not also the Mozilla roots (webpki-roots) that the proxy adds to its upstream trust in default builds. Linux works the same way: the bundle is the distribution's file, and the proxy's upstream trust is that file plus webpki-roots. Using the store alone keeps the host's trust decisions in force for sandbox clients, so a root that an administrator removed or never deployed stays out. webpki-roots holds trust anchors rather than certificates, so writing it as PEM would also need a new dependency (webpki-root-certs: 119 roots, 178,464 bytes of PEM). If maintainers prefer the proxy's view, appending those roots after the store roots on Windows is a small follow-up.

Testing

• Checks appropriate to the affected code and behavior pass
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable): none added. The MXC E2E harness's https-ca scenario runs inside windows:test:mxc-network below.

All runs used Windows 11 build 26340.9767 (x64), with MXC 1.0.1 (wxc-exec --probe: tier base-container), against base windows 5d1bfbf. The Windows checks ran through tasks/scripts/windows-msvc.ps1, which is the command mise run --skip-tools windows:<task> executes.

On commit fc00946:

• cargo fmt --all -- --check: pass.

• mise run --skip-tools windows:lint:x64 (windows-msvc.ps1 lint x86_64-pc-windows-msvc): pass (workspace clippy with -D warnings, e2e clippy, governance-interceptor check).

• cargo nextest run --profile ci -p openshell-supervisor-network --target x86_64-pc-windows-msvc: 1515 tests run, 1515 passed, 7 skipped.

• mise run --skip-tools windows:test:x64 (windows-msvc.ps1 test-precommit x86_64-pc-windows-msvc): pass, 6634 tests run, 6634 passed, 41 skipped.

• cargo test -p openshell-supervisor-network --lib --target x86_64-pc-windows-msvc l7::tls:

  • This change: 22 passed.
  • Base with only windows_sandbox_bundle_includes_certificate_store_roots added: 18 passed, 1 failed (the message above).
  • This change without the size check, with windows_system_ca_bundle_over_the_limit_is_left_out added: 20 passed, 1 failed with the 1332060-byte bundle over the limit was kept.

• Linux (Ubuntu 24.04, WSL2), from a git archive of the commit:

  • cargo fmt --all -- --check: pass.
  • cargo clippy -p openshell-supervisor-network -p openshell-supervisor --all-targets -- -D warnings: pass.
  • cargo check -p openshell-supervisor --all-targets --no-default-features --features system-ca-roots: pass.
  • cargo nextest run --profile ci -p openshell-supervisor-network: 1560 passed (base: 1558; the difference is the two cross-platform tests).

• mise run --skip-tools windows:test:mxc-network (real MXC: pc_https_egress_reads_injected_ca_bundle, a terminated HTTPS request whose peer certificate is issued by the sandbox CA, plus pc_proxy_scopes_network_policy_to_socket_owner): fails on this host before any scenario runs, on windows and with this change alike, with harness error: gateway did not start within 30 s. The gateway log shows the listener bound. run-mxc-e2e.ps1 waits for the port with GetActiveTcpListeners(), which keeps only sockets in the LISTEN state, and this build reports most listening sockets in other states.

  • With two local changes to a copy of run-mxc-e2e.ps1, not part of this PR, the three scenarios behind this task (https-ca, socket-owner-allow, socket-owner-deny) pass on both windows 5d1bfbf and this change. The changes: the port wait connects to the port, and the socket-owner scenarios start their batch file by name from the working directory (with only the first change, socket-owner-allow fails with parent cmd did not launch).
  • Across six runs of each scenario per build, some attempts on both builds got a proxy 403 before any TLS, from the intermittent process-attribution issue (bug(mxc): proxy refuses some connections from an allowed program with "No Windows process owns proxy connection" #5056): https-ca passed 5 of 6 on both; socket-owner-allow 3 of 6 on windows and 5 of 6 with this change; socket-owner-deny 6 of 6 on both. No attempt failed on a certificate or policy check.

• Live MXC sandbox, using the issue's reproduction script on base and head release builds, two runs each with the same results:

  	base 5d1bfbf	head fc00946
  certificates in CURL_CA_BUNDLE	1	48
  tls: skip example.com, curl.exe --cacert "%CURL_CA_BUNDLE%"	exit 60, schannel: the certificate chain is incomplete	exit 0, issuer Cloudflare TLS Issuing ECC CA 3
  terminated example.org, same command	exit 0, issuer OpenShell Sandbox CA	exit 0, issuer OpenShell Sandbox CA

  The same sandbox runs also covered the remaining bundle clients:

  • On base, these fail on tls: skip endpoints, and on head they pass: Python requests, Python ssl with cafile=%SSL_CERT_FILE%, and git ls-remote with http.sslBackend=openssl.
  • Terminated endpoints pass on both builds.
  • Node fetch uses NODE_EXTRA_CA_CERTS, which adds to Node's own roots rather than replacing them, so it passes on both builds.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable): not applicable; crates/openshell-driver-mxc/README.md updated

[jquesnelle]

cc @shailendra-nv

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[shailendra-nv]

/ok to test

[copy-pr-bot]

/ok to test

@shailendra-nv, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

[shailendra-nv]

/windows-ci

[sam-india-007]

Windows ARM64 CI: ✅ passed

Commit 2019faa, requested by @shailendra-nv, 1h 12m, full log

Step	Result	Duration
if ($env:PR_NUMBER) { Import-Module .\scripts\pr-ci\OpenShellPrCi.psm1; Assert-PrCiVariables -PrN...	✅ passed	0s
& .\scripts\setup-windows-arm64-runner.ps1 -VerifyOnly	✅ passed	1s
$env:PATH = "$(Split-Path -Parent $env:RUSTC_WRAPPER);$env:PATH"; $found = (Get-Command sccache -...	✅ passed	0s
New-Item -ItemType Directory -Force (Split-Path -Parent $env:SCCACHE_ERROR_LOG) | Out-Null; & $en...	✅ passed	2s
& $env:RUSTC_WRAPPER --zero-stats	✅ passed	0s
& .\scripts\checkout-openshell.ps1 -Ref $env:OPENSHELL_REF	✅ passed	3s
& .\scripts\set-openshell-version.ps1 -Version $env:OPENSHELL_VERSION	✅ passed	0s
Set-Location $env:OPENSHELL_CHECKOUT_DIR	✅ passed	0s
mise run --skip-tools windows:check:arm64	✅ passed	3m 54s
mise run --skip-tools windows:build:arm64	✅ passed	22m 19s
mise run --skip-tools windows:test:arm64	✅ passed	21m 28s
mise run --skip-tools windows:test:unsupported:arm64	✅ passed	23m 21s
mise run --skip-tools windows:artifacts	✅ passed	2s
& "$env:CI_PROJECT_DIR\scripts\package-windows-release.ps1"	✅ passed	38s

[shailendra-nv]

/ok to test fc00946

[shailendra-nv]

/ok to test 2019faa

[shailendra-nv]

/windows-ci

[shailendra-nv]

Reviewed head 2019faa77d031618fefbe8e8c7366301940b9b2a. Approving after Branch Checks, Windows MSVC x64/ARM64, and the managed native ARM64 report passed.

The original head hit the same unrelated SQLite concurrency failure twice on the GitHub ARM64 runner. Updating from windows brought in the existing repair from #5077. The PR diff remains the reviewed two-file CA-bundle fix; all fresh checks passed on the updated head.

Testing:

Suite	Command	Total	Passed	Skipped	Failed
GitHub Windows x64	mise run --skip-tools windows:test:x64	6,691	6,648	43	0
GitHub Windows ARM64	mise run --skip-tools windows:test:arm64	6,691	6,648	43	0
Managed Windows ARM64 workspace	mise run --skip-tools windows:test:arm64	6,691	6,648	43	0
Managed ARM64 focused configurations	mise run --skip-tools windows:test:unsupported:arm64	10	10	0	0
All listed executions	—	20,083	19,954	129	0

Counts include intentional focused reruns. Managed workspace check, release build and artifact packaging passed; GitHub Windows lint passed on both architectures. Optional E2E and Helm jobs were skipped by their configured filters.