[krishicks]

Summary

Collectors behind a private CA or requiring mutual TLS were out of reach: gRPC export trusted only the platform roots, OTLP/HTTP and the agent relay refused https:// endpoints, and nothing could present a client certificate. Each [openshell.gateway.otlp] table now takes ca_file, cert_file, and key_file, and every exporter verifies against that CA and presents the client certificate.

Related Issue

Closes #5080

Changes

• ca_file, cert_file, and key_file on every OTLP table, inherited with the rest of the collector. They require an https:// endpoint, are read at startup, are capped at 64 KiB each, and the key must match the certificate; a mismatch names both files. The paths and the parsed material are set together, so they cannot disagree.
• ca_file replaces the platform roots. Without it, https:// collectors are verified against the platform roots, which honor SSL_CERT_FILE and SSL_CERT_DIR.
• OTLP/HTTP connections are built through openshell-crypto's fallible try_client_builder; gRPC uses tonic's TLS configuration, recorded as an exception in the openshell-crypto README. cargo deny policy checks pass with the new dependencies.
• Supervisors receive each distinct set of PEM material once in their launch credentials, with lanes referring to it by position, so nothing is mounted into sandboxes. The gateway refuses to start when that material, measured from what is actually sent, would exceed 192 KiB, which leaves room for the rest of a shared supervisor's 256 KiB launch credentials; supervisors reject a bundle whose references fall outside its material. Each supervisor export, and the agent trace relay, reads its own headers and material through one accessor per lane, and supervisor exports apply them through the same CollectorTransport as the gateway. The agent trace relay now supports https:// collectors.
• The VM driver receives the trace collector's file paths. Standalone compute drivers take --otlp-ca-file, --otlp-cert-file, and --otlp-key-file for their own spans.
• e2e/rust and tests/suites/features lockfiles pick up the TLS dependencies.
• Gateway configuration docs cover the keys, inheritance, rotation, version skew and rollback, the shared launch-credential budget, a per-collector CA example, and mounting the files in Helm with extraVolumes.

Rotation needs a restart for now; reloading the files on an interval, including for running sandboxes, is tracked in #5142.

Release Notes

OTLP collectors can use a private CA and mutual TLS: set ca_file, and optionally cert_file and key_file, on [openshell.gateway.otlp] or a signal table. The agent trace relay and OTLP/HTTP export now support https:// collectors.

Testing

• CI lint locally on this commit: tracked lockfile and alignment checks, cargo fmt --check, and every mise run rust:lint clippy invocation with --locked
• Targeted tests pass in openshell-otel (lib and integration binaries), openshell-core, openshell-server, openshell-gateway, the supervisor binary, and all four compute drivers
• Unit tests added/updated: generated test PKI with mutual-TLS collectors over gRPC and HTTP, missing or wrong CA, missing client certificate, right and wrong hostname over both protocols, platform roots including for the relay, file errors, mismatched key errors, redaction of key material, one copy of shared material per bundle, each lane reading its own material, the 192 KiB limit, and rejected out-of-range references
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5084.docs.buildwithfern.com/openshell

[rhuss]

cc-review Summary

This review is agent-assisted (cc-review with several reviewer agents, plus Codex and CodeRabbit). The TLS design looks solid. Material is validated at load, private keys stay redacted, and supervisors receive each distinct collector's files once. One thing should happen before merge: reviews/5082-5083-5084-review.md looks accidentally committed. Everything else is Minor and non-blocking, and the failure scenarios in the comments come from reading the code, not from running it.

What Went Well

• OtlpTls::from_pem builds the full Rustls client config at load, so unparsable PEM or a key that doesn't match its certificate fails at gateway startup instead of on the first export. The test matches the exact InconsistentKeys variant.
• File reads are bounded with take(MAX_FILE_BYTES + 1), so an oversized file or a device node is rejected without being loaded into memory.
• Private keys live in Zeroizing buffers with hand-written Debug impls that redact them, and tests check the redaction. SupervisorAuthBundle::validate rejects dangling *_otlp_tls indexes, and otlp_tls_material uses get(), so a bad index can't panic.
• TLS material on an http:// endpoint is refused (check_tls, TlsWithoutHttps) rather than ignored, and a configured CA replaces the platform roots instead of adding to them.
• The tests do real handshakes: TestPki / ServerTls generate a fresh PKI per test for both tonic and hyper, and the SSL_CERT_FILE tests run in their own binaries so the environment change can't leak.

Findings

Severity	File	Description	Source
Minor	reviews/5082-5083-5084-review.md:1	Review notes covering all three stacked PRs look accidentally committed; please drop the file	architecture (also gator, goal-alignment)
Minor	crates/openshell-core/src/jwt.rs:509	New otlp_tls fields on the strict auth bundle break older standalone drivers and gateway rollback; docs mention only supervisors	production
Minor	crates/openshell-otel/src/tls.rs:15	MAX_FILE_BYTES comment predates dedupe; the 192 KiB cap and "64 KiB" error text are coupled only by comments	architecture
Minor	crates/openshell-otel/src/tls.rs:73	Client certificate expiry is never checked or surfaced, so a short-lived cert silently stops all exports	production
Minor	crates/openshell-otel/src/tls.rs:155	gRPC mTLS runs on the Rustls default provider, outside openshell-crypto (documented exception)	security
Minor	crates/openshell-otel/src/tls.rs:194	client_builder() panics on provider failure; the facade doc says to prefer try_client_builder	codex (also gator)
Minor	crates/openshell-otel/src/transport.rs:349	HTTP half of the host-name test lacks the positive localhost control that the gRPC half has	test-quality
Minor	crates/openshell-server/src/config_file.rs:399	supervisor_tls_bytes duplicates the lane and dedupe logic of supervisor_otlp_tls	architecture
Minor	crates/openshell-server/src/config_file.rs:455	Public TLS path fields and the parsed tls can drift apart (already happens in a vm.rs test)	architecture
Minor	crates/openshell-server/src/config_file.rs:803	192 KiB TLS cap ignores the rest of the 256 KiB launch budget; the overflow error doesn't mention OTLP	gator (also production, correctness, codex)
Minor	crates/openshell-server/src/config_file.rs:901	Key/certificate mismatch error shows rustls text and doesn't name key_file or cert_file	goal-alignment
Minor	crates/openshell-supervisor/src/lib.rs:1302	Relay's choice of agent_otlp_tls is inline and untested, unlike the traces and OCSF lanes	test-quality
Minor	crates/openshell-supervisor/src/main.rs:789	ExportSettings and target() duplicate CollectorTransport and OtlpTarget::with_transport	architecture
Minor	crates/openshell-supervisor/src/otlp_relay/mod.rs:352	No test for an https relay without bundle material (platform roots path)	test-quality

Review Details

• Findings posted: 14 (14 inline)
• Findings reviewed and not posted: 1 (documented behavior: the gateway's collector client key is shared with supervisors)
• Gate outcome: PASS
• Participating reviewers: architecture, correctness, goal-alignment, production, security, test-quality, gator, codex, coderabbit

The TLS work looks solid. All items are non-blocking apart from removing the stray review file, and once that file is gone this should be ready for a maintainer.

[rhuss]

cc-review found no issues at 8684b8c.

[TaylorMutch]

Reviewed the TLS/mTLS changes and the updates through 157168c. No blocking findings.

Local validation of the equivalent pre-rebase patch passed all 62 openshell-otel unit/integration tests and 84 gateway configuration tests. The rebase preserved the PR patch unchanged. Sandbox end-to-end testing was not repeated for this TLS PR.

Nonblocking follow-up: the TLS launch budget counts raw PEM string lengths rather than serialized JSON size. Valid CA files padded heavily with blank lines can pass the 192 KiB guard and exceed the shared supervisor's 256 KiB registration limit. Accurate serialized-size accounting would improve robustness, but this operator-controlled, unusually padded configuration edge case does not warrant holding this change.