[krishicks]

Summary

Collectors that need an authorization token or a routing header could get one only through OTEL_EXPORTER_OTLP_*HEADERS in the gateway's environment, which never reached supervisor spans or the agent trace relay. Each [openshell.gateway.otlp] table now takes headers, or a headers_file that keeps credentials out of the configuration file, and every exporter sends them.

Related Issue

Closes #5067

Changes

• headers (an inline table) and headers_file (one name=value header per line, at most 64 KiB) on every OTLP table, mutually exclusive and inherited with the rest of the collector. Setting either in a table without its own endpoint is a configuration error.
• Headers are validated at load: valid names set once, no exporter-owned names (content-type, host, grpc-*, and others), no binary -bin names, and non-empty printable-ASCII values. Errors never include a value or a name that failed validation, and headers_file errors cite the line number instead. Debug output redacts values, and values read from headers_file are never serialized back out. Config parse errors now give the line, column (in characters), and parser message without quoting the file or any string in it, so a malformed header table cannot echo a token.
• Exporters send headers as gRPC metadata or HTTP headers, with values sent exactly as configured. Environment headers still apply to the gateway's own exporters and replace configured headers of the same name. Supervisor trace export and the agent relay send only their table's configured headers; supervisor OCSF export keeps the environment merge it already has.
• Supervisors receive each lane's headers in their launch credentials, never through the sandbox environment. The VM driver receives the same trace headers as the gateway's own exporter, in its environment. Standalone drivers send only their own OTEL_EXPORTER_OTLP_*HEADERS.
• Gateway configuration docs describe the keys, validation, relative paths, cleartext caveats, external-driver endpoints, version skew, and rollback; the traces page mentions headers.

Release Notes

OTLP collectors can require authentication or routing headers: set headers or headers_file on [openshell.gateway.otlp] or a signal table. Supervisors and the agent trace relay send them too, so MLflow can ingest relayed agent traces directly with x-mlflow-experiment-id.

Testing

• CI lint locally on this commit: tracked lockfile and alignment checks, cargo fmt --check, and every mise run rust:lint clippy invocation with --locked
• openshell-server config tests pass on this commit; its tracing and compute tests and openshell-gateway lib tests pass at the top of the stack
• Unit tests added/updated: header validation, inheritance, headers_file parsing and line-numbered errors, redaction, parse errors that do not quote the file, environment headers winning on the gateway's exporters, which headers each driver and supervisor lane receives, and headers arriving at loopback gRPC and HTTP collectors from the gateway, supervisor, and relay
• E2E tests added/updated (if applicable)

Manually verified before the latest rebase: a dev Docker gateway relayed Pi agent traces directly to MLflow over OTLP/HTTP with x-mlflow-experiment-id, and each trace carried openshell.sandbox.id.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5083.docs.buildwithfern.com/openshell

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

With trace export enabled, existing OTEL_EXPORTER_OTLP_TRACES_HEADERS or OTEL_EXPORTER_OTLP_HEADERS settings now add traces_otlp_headers to launch_authentication instead of preserving the baseline JSON payload.

• Impact: Previously compatible external compute drivers using the baseline decoder fail sandbox creation and start-from-stopped with an unknown-field error.
• Action: For v0.1.7, negotiate trace-header support and omit new launch fields for legacy peers rather than requiring existing external drivers to upgrade.
• PR: feat(observability): send configured headers to every OTLP collector #5083

Review details

• Execution: complete
• Candidate: PR #5083 (c736ddc358f21178dc1100bdd7920221bf4588e4)
• Baseline: 0d79a30690a1cc366d5477ef02d0d866298a4a12 (0d79a30690a1cc366d5477ef02d0d866298a4a12)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[rhuss]

cc-review Summary

This is an agent-assisted review: cc-review ran several reviewer agents plus Codex and CodeRabbit over the diff. The redaction design holds up well, but three items should be fixed first: two error paths that can echo a credential into the gateway startup log, and an upgrade compatibility question that needs a maintainer decision. The failure scenarios come from reading the code, not from running a gateway. The one exception is the toml error message quoted in the parse_error_detail comment, which a reviewer agent reproduced in a scratch program against toml 0.8.23. The Minor findings are non-blocking, and once the three Important items are addressed (or a maintainer settles the compatibility question) this should be ready for a maintainer.

What Went Well

• Credential values stay out of logs. The hand-written Debug impls for OtlpTarget and OtlpHeaders print header names only, SupervisorAuthBundle redacts the new header fields, and ConfigFileError::Parse reports line and column instead of quoting the source.
• append_otlp_args hands the VM driver its trace headers through OTEL_EXPORTER_OTLP_TRACES_HEADERS rather than argv, and vm_driver_receives_trace_headers_in_its_environment_only checks that no argument contains the secret.
• read_headers_file caps the read with take(MAX_HEADERS_FILE_BYTES + 1) and rejects non-UTF-8, malformed and duplicate lines before validation.
• Header parsing now lives in the shared headers.rs, used by the gateway, supervisor and relay alike, with a round-trip test for values containing ,, =, % and spaces. configure_http removes the duplicated HTTP exporter setup between traces and logs.
• RawCollector::resolve applies one rule to every lane (own endpoint, or inherit the whole shared collector), so one collector's credentials can't leak to another.

Findings

Severity	File	Description	Source
Important	crates/openshell-server/src/config_file.rs:485	Invalid header name error echoes the headers_file line, so a padded token reaches the startup log	security
Important	crates/openshell-server/src/config_file.rs:1232	parse_error_detail forwards serde messages that quote a string-typed headers value	security
Important	crates/openshell-server/src/otel_tracing.rs:325	Env-only header setups now add traces_otlp_headers, which older supervisors and drivers reject	production
Minor	crates/openshell-gateway/src/vm.rs:730	VM driver trace headers come from the supervisor helper, a second derivation of the same value	architecture
Minor	crates/openshell-otel/src/lib.rs:318	configure_http decodes header values twice on OTLP/HTTP, changing literal %XX sequences	correctness
Minor	crates/openshell-otel/src/lib.rs:681	Header tests don't clear OTEL_EXPORTER_OTLP_*HEADERS and fail when a developer has them set	test-quality
Minor	crates/openshell-server/src/config_file.rs:460	An unreadable headers_file stops the gateway from starting; the docs should say so	production
Minor	crates/openshell-server/src/config_file.rs:795	Endpoint-required check duplicates RawCollector::resolve, and shared skips raw_collector!	architecture
Minor	crates/openshell-server/src/config_file.rs:2111	No-echo assertions cover two of about fifteen header validation error cases	test-quality
Minor	crates/openshell-server/src/otel_tracing.rs:899	No test for env headers overriding configured headers on the gateway's own exporters	test-quality
Minor	crates/openshell-server/src/tracing_setup.rs:77	No test that in-process compute drivers receive the trace collector's configured headers	test-quality

Review Details

• Findings posted: 11 (11 inline)
• Findings reviewed and not posted: 2 (documented behavior: header rotation needs restarts; external drivers must use the gateway's endpoints)
• Gate outcome: FAIL (3 Important)
• Participating reviewers: architecture, correctness, goal-alignment, production, security, test-quality, gator, codex, coderabbit

[rhuss]

cc-review found no issues at a69ff8e.