AWF: let operators configure model-router OpenTelemetry export #10421

Part of github/gh-aw-firewall#10261
(step 7b of 7; step 7a, #10413, added the api-proxy's router-call spans and trace context).

Prerequisite

The Go router in containers/router/ must support opt-in OTLP/HTTP protobuf traces and
metrics, standard endpoint/header/protocol/sampling/signal settings, W3C context
extraction, low-cardinality attributes and bounded shutdown. With no endpoint it creates
no exporters, but request logs can carry a validated incoming trace_id. Its default
service name is gh-aw-router; the default sampler is parentbased_always_on.

Summary

The parent issue explicitly requires OTel configuration propagation through AWF
router-service creation and collector/network documentation. Step 7a covered
trace-context correlation. This step implements the configuration propagation; a test that
merely prevents configuration propagation would not satisfy that requirement.

Let operators opt in to router telemetry through router-specific AWF_ROUTER_OTEL_*
variables, which AWF maps to an allowlist of standard OTEL_* settings in the router's
environment. Keep the router exclusively on the
internal awf-routing network, with no new ports published, credentials automatically
discovered, proxy access or relay.

An internal network can reach a collector attached to that network. It cannot reach an
arbitrary external endpoint. Validate the supported configuration with a test-only
collector on that network and document the limitation. Automatically exporting through
the api-proxy to external collectors is a separate optional relay proposal, not a
prerequisite for this issue.

Background

• buildRouterService in src/services/router-service.ts creates the router only on
  awf-routing. src/compose-generator.ts marks the network internal: true.
  The router currently has no service-level environment or volumes.
• The api-proxy joins that network when routing is enabled and waits for router health.
• buildOtelEnv in src/services/api-proxy-env-config.ts forwards the host's existing
  telemetry endpoint/header and workload-identity settings to the api-proxy. Do not
  copy that environment wholesale to the router.
• gh-aw sets the shared OTEL_EXPORTER_OTLP_ENDPOINT and a secret-backed
  OTEL_EXPORTER_OTLP_HEADERS in AWF's process environment whenever a workflow
  configures OTLP. It excludes them only from the agent. That endpoint is normally an
  external collector, which the api-proxy reaches through Squid and the router can't
  reach. Forwarding the shared variables would make every such run attempt failed
  router exports and copy the collector credentials into another container. So router
  export needs its own opt-in variables.
• buildExclusionSet in src/services/agent-environment/excluded-vars.ts lists the
  host variables that never reach the agent container.
• Compose snapshots are redacted by environment-variable name: redactDockerComposeSecrets
  in src/compose-generator.ts writes the audit copy, and sanitizeDockerComposeYaml in
  src/compose-sanitizer.ts writes the diagnostics copy.
• The api-proxy wraps each router call in a api_proxy.router.<operation> client span and
  sends its traceparent (step 7a), so router spans exported by this step join the
  workflow trace.

Scope and specification

Router environment

Build a router-specific environment in src/services/router-service.ts or an adjacent
helper following the existing service-builder patterns.

• Opt-in names. For each supported setting OTEL_<NAME> below, read the host
  variable AWF_ROUTER_OTEL_<NAME> and, when it's set and non-empty, pass its value to
  the router as OTEL_<NAME>. For example, AWF_ROUTER_OTEL_EXPORTER_OTLP_ENDPOINT
  becomes the router's OTEL_EXPORTER_OTLP_ENDPOINT. Supported settings:
  • OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_SDK_DISABLED;
  • OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER;
  • OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG;
  • OTEL_METRIC_EXPORT_INTERVAL, OTEL_METRIC_EXPORT_TIMEOUT;
  • OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL,
    OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TIMEOUT,
    OTEL_EXPORTER_OTLP_CERTIFICATE, and their TRACES/METRICS variants.
• Shared host OTEL_* variables are never forwarded to the router. They belong to
  the api-proxy and other components (see Background).
• Preserve standard signal-specific precedence. Do not synthesize an endpoint, force
  a disabled signal on or set a metric export interval; the router's shutdown flush
  exports cumulative totals for short runs. With no AWF_ROUTER_OTEL_* variable set,
  omit environment.
  Rely on the router's own gh-aw-router service-name default when the host omits it.
• Add every AWF_ROUTER_OTEL_* name to buildExclusionSet, so none of them reaches
  the agent container, whatever the env-passthrough settings are.
• OTLP headers are an explicit operator-provided credential exception: forward them
  only as configured exporter settings, never as span/log attributes or request headers
  on the router API. Redact the router's OTEL_EXPORTER_OTLP_HEADERS,
  OTEL_EXPORTER_OTLP_TRACES_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS values,
  and the corresponding AWF_ROUTER_OTEL_* names wherever they appear, in both the
  audit and the diagnostics compose snapshots. Also redact the endpoints' user info.
  Add explicit rules and tests for these names rather than relying on keyword matching.
• Do not forward GH_AW_OTLP_ENDPOINTS, GH_AW_OTLP_WORKLOAD_IDENTITY,
  ACTIONS_ID_TOKEN_REQUEST_URL, ACTIONS_ID_TOKEN_REQUEST_TOKEN, provider credentials,
  GITHUB_AW_OTEL_* or proxy variables. Workflow correlation arrives over traceparent.
• Pass resource settings to the router's validator. Only service.namespace and
  deployment.environment.name resource attributes are accepted; the binary owns
  service.version. Do not add a separate, more permissive resource parser in AWF.
• Do not attach another network, publish ports or mount host certificate/key paths.
  Certificate paths must already exist inside the image; unsupported or unreachable
  exporter configuration produces sanitized router diagnostics without failing routing.

Documentation

Update docs/otel-sentry.md and containers/router/README.md to explain:

• the AWF_ROUTER_OTEL_* opt-in variables and their mapping, precedence, signal
  disablement and default service name, and that shared host OTEL_* variables
  configure only the api-proxy;
• OTLP/HTTP protobuf, the resource allowlist and privacy rules;
• collector reachability on the internal network, certificate paths inside containers,
  and why host localhost is not a collector address inside the router;
• that external api-proxy fan-out/workload identity is not supported for router export
  by this change; no collector or external access is required for routing;
• how router spans join the api-proxy's router-call spans from step 7a, and log
  correlation, including that current teardown does not retain router Docker logs. Do not
  claim the separate log-capture proposal has already landed.

Out of scope

• Changes to the routing algorithm, router HTTP/CLI contracts or router instrumentation.
• External telemetry relaying, new AWF collector infrastructure or network egress.
• Keeping router logs after teardown or changing teardown order.
• Changes to gh-aw audit data. Routing outcomes continue through the unified
  firewall.model_routing records in usage/aw_session.jsonl.

Acceptance criteria

• AWF_ROUTER_OTEL_* settings propagate through buildRouterService as the
  router's OTEL_* settings, with explicit allowlist, precedence and
  credential-redaction tests covering both compose snapshots.
• Shared host OTEL_* variables (for example a gh-aw-style external endpoint and
  authorization header), unrelated host credentials, workload identity and proxy
  settings never reach the router, and its network membership remains unchanged.
• No AWF_ROUTER_OTEL_* variable reaches the agent container.
• A collector on the internal network receives router traces and metrics with the
  configured identity, headers and signal settings. Router spans have the router-call
  span as parent.
• No configuration, disabled signals and an unreachable collector preserve routing
  behavior. With no supported variables set, compose output stays unchanged.
• The documentation describes actual supported reachability and log retention.

Validation

• Compose tests: base and signal-specific settings, SDK/signal disablement,
  resource/sampler propagation, no-config output, allowed OTLP headers and excluded
  credential canaries. Set shared host OTEL_EXPORTER_OTLP_ENDPOINT and
  OTEL_EXPORTER_OTLP_HEADERS and assert that the router's environment is unchanged.
  Check that the audit and diagnostics snapshots redact the router's header values.
• Agent environment tests: AWF_ROUTER_OTEL_* names are in the exclusion set.
• Integration: extend tests/integration/model-routing.test.ts with a test-only
  protobuf collector attached to awf-routing. Start it independently of the
  api-proxy's service_healthy dependency. Verify receipt, parent ids, both signals,
  disablement, and continued routing when the collector fails. Keep that collector
  alive through router shutdown; no production service is added.
• No router collector: in test-model-routing.yml's routed runs, set a known workflow
  parent and use the api-proxy file exporter; assert step 7a's router-call spans belong to that
  workflow trace. Assert that the route span's goal,
  mode, selected model and effort match the run's firewall.model_routing selection
  record. Do not require an LLM request span during bootstrap or make this test depend
  on optional log capture.
• Maintainer-run sandbox: confirm routing outcomes in usage/aw_session.jsonl are
  unchanged. Private sandbox access is not required of the coding agent.

Files likely touched

• src/services/router-service.ts, a router environment helper if needed, and
  src/compose-generator.test.ts
• src/services/agent-environment/excluded-vars.ts and its tests
• src/compose-generator.ts (redactDockerComposeSecrets),
  src/compose-sanitizer.ts and their tests
• tests/integration/model-routing.test.ts
• docs/otel-sentry.md and containers/router/README.md