Skip to content

docs(observability): show where each OTLP signal goes, including relayed agent traces - #5153

Open
krishicks wants to merge 4 commits into
mainfrom
docs/otlp-collectors-diagram/krishicks
Open

krishicks wants to merge 4 commits into
mainfrom
docs/otlp-collectors-diagram/krishicks

Conversation

@krishicks

@krishicks krishicks commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The observability overview had no picture of which component exports which signal to which collector, agent traces appeared only in the gateway configuration reference, and the Traces page did not say that the agent's own spans travel separately. This adds an OTLP Collectors section with an architecture diagram, an Agent Traces section with an MLflow example, and links from the Traces page and the configuration reference.

Related Issue

None; docs-only follow-up to the OTLP collector stack: #5082 (protocol), #5083 (headers), and #5084 (TLS). Stacked on #5084.

Changes

  • docs/images/openshell-otlp-collectors.svg: a diagram drawn in the style of the existing architecture images. It shows the gateway and compute drivers, the sandbox supervisor's own telemetry, the agent trace relay inside the network-isolated sandbox, and the three collector tables.
  • Observability overview:
    • Agent traces listed as a signal.
    • A new OTLP Collectors section with the diagram, which summarizes each collector's protocol, header, and TLS settings and links to the configuration reference.
    • A new Agent Traces section with an MLflow example.
  • Traces page: notes that the agent's own spans travel through the relay.
  • Configuration reference: the OTLP Export intro links to the new overview section.

Testing

  • mise run docs (Fern check and navigation check)
  • Diagram rendered and checked
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Every OTLP export (gateway traces, OCSF logs, supervisor traces, and the
agent trace relay) spoke gRPC only, so trace backends that accept only
OTLP/HTTP needed a translating collector in between.

Each [openshell.gateway.otlp] table accepts protocol = "grpc" or
"http/protobuf". A table without its own endpoint uses the whole shared
collector; a table with its own endpoint inherits nothing from it. Compute
drivers take the collectors through one shared set of flags, read from
the same environment variables that supervisors receive, and pass them on
through one SupervisorOtlp value that replaces the per-lane driver fields.
OTLP/HTTP goes through a single client in openshell-otel, shared by the
SDK exporters and the relay, that bounds each request by the export
timeout and fails exports whose response is incomplete or larger than
64 KiB. Exporters send no query parameters, so an endpoint with a query
string is a configuration error.

Refs #5067

Signed-off-by: Kris Hicks <khicks@nvidia.com>
Collectors that require authentication or a routing header could only get
one through OTEL_EXPORTER_OTLP_*HEADERS in the gateway's environment, which
reached the gateway's own exporters and supervisors' OCSF records but not
supervisor spans or the agent relay, so a backend such as MLflow needed a
sidecar collector to add its experiment header.

Each [openshell.gateway.otlp] table now takes a headers table, or a
headers_file of name=value lines that keeps credentials out of the config
file, inherited with the rest of the collector by tables without their own
endpoint. Exporters send them as gRPC metadata or HTTP headers, with
environment headers replacing configured ones of the same name. Supervisors
receive each lane's headers in their launch bundle and the VM driver in its
environment; the agent relay uses only its table's headers. Config parse
errors give a line and column instead of quoting the file, so a malformed
header table cannot echo a token.

Closes #5067

Signed-off-by: Kris Hicks <khicks@nvidia.com>
…client certificate

Collectors behind a private CA or requiring mutual TLS were out of reach:
gRPC export trusted only the platform roots, OTLP/HTTP and the agent relay
refused https endpoints, and nothing could present a client certificate.

Each [openshell.gateway.otlp] table now takes ca_file, cert_file, and
key_file, inherited with the rest of the collector and read at startup.
Every exporter verifies https collectors against that CA and presents the
client certificate, over gRPC or HTTP, with HTTP connections built through
openshell-crypto. Supervisors receive each distinct material once in their
launch bundle, with a 192 KiB total checked at startup to leave room in a
shared supervisor's launch material, the VM driver receives the file paths, and
standalone drivers take --otlp-ca-file, --otlp-cert-file, and
--otlp-key-file.

Closes #5080

Signed-off-by: Kris Hicks <khicks@nvidia.com>
…yed agent traces

The observability overview had no picture of which component exports
which signal to which collector, agent traces appeared only in the
gateway configuration reference, and the Traces page did not say that
the agent's own spans travel separately.

Add an OTLP Collectors section to the overview with a diagram, drawn in
the style of the architecture images, of every lane: gateway and driver
spans, supervisor spans and OCSF events, and agent traces relayed
through the supervisor. Summarize the per-collector protocol, header,
and TLS settings, add an Agent Traces section with an MLflow example,
and link to the overview from the Traces page and the configuration
reference.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

@krishicks
krishicks force-pushed the feat/5080-otlp-collector-tls/krishicks branch 2 times, most recently from 157168c to 424e23c Compare November 2, 2026 23:35
Base automatically changed from feat/5080-otlp-collector-tls/krishicks to main November 3, 2026 01:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant