Router: add opt-in OpenTelemetry traces and metrics #10380

Part of github/gh-aw-firewall#10261
(step 6 of 7).

Prerequisite

The shipped router must be the Go binary in containers/router/, with serve,
structured JSON stderr logs, CA certificates in the image and passing Go CLI and HTTP
tests. Its server currently drains for at most 0.5 s and exits within 1 s on SIGTERM.
The logging middleware emits request records with an internally generated
request_id, normalized method, route template, status, latency and error code;
successful health checks are suppressed. Extend that middleware rather than creating
a second access log.

Summary

This step adds OpenTelemetry traces and metrics to the Go router. They're turned on with
the standard OTEL_* environment variables. When none are set, the router exports nothing
and keeps its HTTP responses and CLI stdout unchanged. Request logs can carry validated
trace identifiers, and startup can include a sanitized telemetry-status record.
Telemetry takes attributes only from a fixed allowlist and never records conversation
content, caller-supplied model/choice ids or raw headers.
Shutdown flushes it within the 1 s that AWF's teardown allows between SIGTERM and SIGKILL.

AWF configuration propagation and the api-proxy's trace-context injection are a separate
required step. The router's internal awf-routing network can reach a collector attached
to that network, but not an arbitrary external collector. This issue does not change
that network or add a relay. Without export, request logs still carry a validated
incoming trace_id; preserving those logs after AWF teardown is a separate follow-up.

Background

• Rewrite the model-router Python server in Go #10261 asks for structured logs plus OpenTelemetry. The tests have to
  cover trace propagation, spans and metrics, low cardinality, redaction, disabled and
  unavailable exporters, and bounded flush.
• The api-proxy already emits one client span per LLM request. It uses a custom OTLP/HTTP
  JSON exporter in containers/api-proxy/otel.js, configured through
  GH_AW_OTLP_ENDPOINTS or OTEL_EXPORTER_OTLP_ENDPOINT, and its parent comes from
  GITHUB_AW_OTEL_TRACE_ID and GITHUB_AW_OTEL_PARENT_SPAN_ID. The subsequent AWF
  wiring step adds traceparent on router calls, making router spans children of
  api-proxy router-call spans.
• The router is stateless. Each /classify or /route call is one CPU-bound operation
  with no outbound calls, so one server span with a single child span is enough.
• The router API's enums are small and fixed:
  • goal: cost, cost-speed.
  • mode: economy, balanced, robust, auto.
  • resolved_mode: economy, balanced, robust.
  • Classifier labels: task_type, scope and task_complexity.
  • Error code: 8 values in HTTP error envelopes (the CLI and startup also use
    invalid_model_data).

Scope

• Set up the OTel SDK from the environment, for traces and metrics over OTLP/HTTP. It's off
  unless configured.
• Accept W3C trace context on incoming requests.
• Server spans and one child span per planning operation, with attributes from an
  allowlist.
• Three metric instruments.
• Add trace_id to request log records whenever the caller sends a valid traceparent,
  and span_id when tracing is on.
• A shutdown flush within a fixed time budget.
• Tests and README documentation.

Specification

Dependencies

• Add these modules:

  • go.opentelemetry.io/otel
  • go.opentelemetry.io/otel/sdk and go.opentelemetry.io/otel/sdk/metric
  • go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
  • go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp
  • the matching semconv package for attribute names

  Use one released version across all of them.

• Transitive dependencies. Official HTTP exporters can introduce gRPC-related
  modules through generated OTLP packages even though only HTTP export is enabled.
  Inspect the pinned module graph and built binary rather than assuming that every
  transitive package is linked.

  • Record the binary and image size before and after, in the PR.
  • Confirm that govulncheck, Grype (which fails on High) and the license check in
    supply-chain-scan.yml all pass. The generated OTLP collector packages link parts of
    gRPC, grpc-gateway and golang.org/x/net/http2, so their advisories apply; raise the
    indirect golang.org/x/net requirement if govulncheck flags it.
  • Add a router gomod Dependabot entry, in the style of the existing router docker
    entry, so these modules stay current. Turn on the setup-go module cache in
    test-router.yml, keyed on containers/router/go.sum.

• Instrumentation.

  • Don't use otelhttp unless it can be configured to emit only the allowlisted
    attributes below. By default it records user_agent.original, the client address and
    the raw URL path.
  • Instrument the existing server by hand, using semconv names.

Enablement and configuration

• When it's on.
  • Off by default. A signal is on only when it has an endpoint:
    OTEL_EXPORTER_OTLP_ENDPOINT, or the per-signal OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    or OTEL_EXPORTER_OTLP_METRICS_ENDPOINT.
  • With no endpoint, the router registers no SDK providers and starts no telemetry
    goroutines or network activity. HTTP responses and CLI stdout remain unchanged;
    structured stderr can gain the documented telemetry-status and correlation fields.
  • OTEL_SDK_DISABLED=true disables all SDK providers/exporters, not W3C context
    extraction for log correlation.
  • OTEL_TRACES_EXPORTER and OTEL_METRICS_EXPORTER accept otlp (the default) and
    none. Any other value logs a warning and turns that signal off.
• Protocol.
  • OTEL_EXPORTER_OTLP_PROTOCOL and the per-signal versions accept only
    http/protobuf.
  • grpc or http/json logs a warning naming the variable and turns that signal off.
• Additional settings. Support base and per-signal OTLP HEADERS, TIMEOUT and
  CERTIFICATE variables, plus
  OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG (default parentbased_always_on),
  OTEL_METRIC_EXPORT_INTERVAL and OTEL_METRIC_EXPORT_TIMEOUT.
  Explicitly validate supported values before constructing providers; do not assume
  the SDK applies every variable or reports errors with the required privacy policy.
  Use standard precedence: signal-specific settings override the shared OTLP setting.
  A shared endpoint is a base URL; signal-specific endpoints are complete signal URLs.
  • Header values are never logged.
  • The exporters honor HTTPS_PROXY and NO_PROXY the way Go does by default.
• Resource.
  • service.name comes from OTEL_SERVICE_NAME, default gh-aw-router.
  • service.version is the router version.
  • service.instance.id is a random UUID for each process. Metrics are cumulative, so
    without it two routers exporting to the same collector would produce streams that
    look like one counter resetting. It's not derived from the host or container.
  • From OTEL_RESOURCE_ATTRIBUTES, allow service.namespace and
    deployment.environment.name only. Warn once about ignored keys without printing
    keys or values; do not permit this variable to overwrite the router version.
    Document this operational-metadata allowlist rather than forwarding arbitrary data.
  • Don't add the host, process, OS or container detectors.
• Bad configuration never stops the router. Log a WARN record naming the variable,
  without its value, and carry on with that signal turned off.
• Startup log. One INFO record, telemetry configured, logged even when both signals
  are off, with:
  • traces and metrics, each enabled or disabled;
  • protocol, when either signal is on;
  • for each enabled signal, the endpoint's scheme and host only. Never the path, query,
    user info or headers.

Propagation

• Use the W3C traceparent and tracestate propagator only. Ignore baggage, because it
  can carry anything. The caller's tracestate stays on the router's span contexts and so
  appears in exported spans, as W3C propagation requires, but is never logged.
• Register the propagator even when tracing is off. Extracting the context needs only the
  OTel API, not the SDK, so this starts nothing.
• A valid incoming traceparent makes the server span its child. A missing or invalid one
  starts a new root span, and the request is still served normally.
• The router makes no outbound calls apart from exporting telemetry, so it never injects
  trace context.

Spans

• Server span. One SERVER span per request. There is no span for GET /healthz, or
  for requests rejected with a bare 503 by the 64-request limit, which only count in
  metrics.
  • Name: {method} {route} (for example POST /route), or just {method} for unmatched
    paths, using the normalized method (unknown methods become _OTHER).
  • Attributes, which are the only ones allowed:
    • http.request.method: known methods, else _OTHER;
    • http.route (matched routes only);
    • http.response.status_code;
    • url.scheme;
    • network.protocol.version;
    • error.type: the error envelope code.
  • Status: Error only for 5xx. Client errors (4xx) leave the status unset,
    as semconv says.
  • Aborted requests. When the request body fails to arrive (a broken chunk, a client
    disconnect or the server's read timeout), the HTTP step closes the connection without
    a response and without a request log record, because no status was sent. Consider
    ending the server span with status Error and a fixed error.type such as
    request_aborted, and no http.response.status_code, so these requests become
    visible without inventing an HTTP status for the log.
• Child span. Each /classify or /route request that gets past decoding has one
  INTERNAL child span, named gh_aw_router.classify or gh_aw_router.route.
  • Attributes:
    • gh_aw_router.outcome: ok, invalid_request, no_route or internal_error;
    • gh_aw_router.conversation.messages and gh_aw_router.models.count: counts;
    • gh_aw_router.prior_turns.included: classify only;
    • gh_aw_router.goal, gh_aw_router.mode, gh_aw_router.resolved_mode and
      gh_aw_router.classification.present: route only, once the objective has been
      parsed;
    • the classifier labels gh_aw_router.task_type, gh_aw_router.scope,
      gh_aw_router.task_complexity and gh_aw_router.mode_recommendation, when the
      route request includes a classification;
    • gh_aw_router.selected.model and gh_aw_router.selected.effort: route ok only.
      Set them only if the selected model is in the loaded execution catalogue
      and take the value from that catalogue entry. A model id that came from the request and
      doesn't match the catalogue is never recorded.
  • Never record:
    • choice ids or current_id;
    • conversation text, tool names, inputs or results;
    • candidate lists;
    • prompts, the classifier system prompt, or error detail strings.

Metrics

• Router-specific metric attributes use the gh_aw_router. prefix, like the span
  attributes, so they can't collide with semconv names.
• A bare 503 from the 64-request limit has no error envelope; record it on the duration
  histogram with error.type 503, as semconv does for errors without a better type.
• gh_aw_router.requests.rejected deliberately repeats what the histogram records for
  overload and oversized bodies, so load shedding has one name to find and alert on.
• Leave out http.server.active_requests. It's a Development-status semconv instrument,
  and a router serving a handful of requests per run gets nothing useful from a
  concurrency gauge.
• /healthz counts in the HTTP metrics, where http.route keeps it separate.
• Metrics never carry model ids, choice ids, classifier labels or any request-derived
  arbitrary string. The number of distinct attribute sets is a fixed bound that can be worked
  out from the enums above, and a test checks it.
• The duration histogram's count supplies request counts by status/error; no separate
  unbounded request-count label set is needed. Normalize methods before all instruments.

Logs

• When a local server span has a valid context, its request log record also gets
  trace_id and span_id, including for a valid but unsampled local span.
• When tracing is off and the request has a valid traceparent, the record gets that
  header's trace_id only. Leave out span_id, because the router created no span. This
  lets router logs be matched to the api-proxy's traces without an exporter.
• With no valid traceparent and tracing off, leave both fields out.
• An incoming trace_id is caller-controlled, not guaranteed random or trusted.
  Treat validated, nonzero 32-hex-digit identifiers as the explicit correlation
  allowlist exception. Never log the raw traceparent, tracestate or baggage.
• Requests logged without a router span, such as a failed /healthz check, never
  borrow the remote parent's span_id and label it as a router span.

Errors, queues and shutdown

• Exporter errors. Register a global OTel error handler. It logs at most one WARN
  record per signal per minute, with a short reason: the HTTP status, timeout,
  connection refused or other. Don't log the raw error text, because it can contain
  the endpoint URL. Never log headers. Ignore a bare context.Canceled, which the SDK's
  periodic metric reader reports on every shutdown.
• Queues. Use the batch span processor's default queue of 2,048 spans and drop spans
  when it's full. Use the periodic metric reader. Exporting never blocks request handling.
• Shutdown.
  • Without telemetry, keep the existing timing: drain for up to 0.5 s.
  • With telemetry on, SIGTERM drains HTTP for up to 0.25 s. Then the trace and metric
    providers are flushed and shut down in parallel, together limited to 0.5 s. That
    leaves about 0.25 s of margin for process start-up and scheduling.
  • The process exits 0 within 1 s whether or not the flush succeeds. If the flush
    times out, log one WARN record, telemetry flush incomplete. AWF's teardown sends
    SIGKILL 1 s after SIGTERM, because it runs docker compose down -t 1.
  • Telemetry should not depend only on the final flush. The batch span processor's
    default 5 s delay exports spans while the run is still going. Metrics are cumulative,
    so the final flush exports the run's totals even when the run is shorter than the
    default 60 s interval. Don't recommend a shorter OTEL_METRIC_EXPORT_INTERVAL
    default. Document it as an operator choice, useful only to see metrics during a run
    or to keep them if the final flush fails. The AWF wiring step forwards an explicit
    setting without setting a default.
• HTTPS. Endpoints are verified against the CA bundle already in the Go image.
  SSL_CERT_FILE still works.

Tests

Use the SDK's in-memory span exporter, a manual metric reader, and an httptest fake OTLP
collector.

• Propagation.
  • A valid traceparent makes a child span.
  • A malformed one makes a root span, and the response is still 200.
  • baggage is ignored.
  • With tracing off, a valid traceparent puts its trace_id, and no span_id, in the
    request log record. A malformed one adds neither field.
• Spans. Check the structure and attributes for each endpoint and outcome: ok,
  invalid_json, invalid_request, no_route, 404, 405, 413, 415 and 500. Check
  that there are no /healthz spans and no spans for bare 503s.
• Metrics. Check every instrument and attribute set for the same matrix, including
  overload and payload-too-large rejections.
• Cardinality.
  • Send a few thousand generated requests with random paths, methods, headers, query
    strings and bodies.
  • Assert that every attribute value is in its allowlisted set, and that the number of
    distinct metric series stays within the fixed bound.
• Redaction.
  • Put canary strings in the conversation text, tool inputs, request model ids, choice
    ids, current_id, headers (Authorization, User-Agent, a custom one), the query,
    unmatched paths, OTEL_EXPORTER_OTLP_HEADERS, and the endpoint URL's user info.
  • Serialize every exported span and metric, and every log line. Assert that no canary
    appears anywhere.
  • Assert that gh_aw_router.selected.model appears only when the selected model is in
    the catalogue.
• Disabled.
  • With no endpoint, with OTEL_SDK_DISABLED=true, and with both signal exporters
    set to none: no exporter is created and the fake collector receives nothing.
    Disabling just one signal must leave the other configured signal working.
    The existing HTTP test expectations and CLI stdout remain unchanged.
  • Valid/invalid resource attributes, service identity, sampler settings, OTLP headers,
    endpoint precedence and certificate/timeout configuration have explicit tests.
  • grpc and http/json each log a warning and turn the signal off.
• Unavailable. Point the exporter at a closed port, a collector that returns 503, and
  a collector that hangs. Requests still succeed with no added latency beyond a small
  allowance. Warnings are rate-limited. Shutdown keeps within its budget.
• Bounded flush (Linux and macOS only). Run the binary with a hanging collector and
  a request in flight, then send SIGTERM. The process exits 0 within 1 s.
• End to end. The fake collector decodes the protobuf export requests (a test-only
  dependency is fine) and receives both the server span and its child span, with the
  right parent, resource and service name. Traces and metrics carry the same
  service.instance.id, and two router processes get different ones.

Out of scope

• AWF configuration propagation and api-proxy trace-context injection are a separate
  required step under the parent issue. External OTLP relaying and retaining router
  logs after teardown are optional follow-ups.
• OTLP over gRPC, and OTel logs. Logs stay on stderr as JSON.
• Changes to the HTTP API, the CLI or the log format, apart from the added fields above.

Acceptance criteria

• With no OTel configuration, the router starts no telemetry goroutines or network
  activity. HTTP behavior and CLI stdout are unchanged; structured stderr gains
  only the documented status/correlation fields.
• Traces and metrics export over OTLP/HTTP when configured through the standard
  variables. Invalid or unsupported settings log a warning without the value and turn
  the signal off; they never stop the router.
• Incoming W3C trace context is honored and baggage is ignored.
• Spans and metrics carry only the allowlisted attributes. The cardinality and canary
  redaction tests pass, and model ids appear only on spans and only from the catalogue.
• Requests with a local span context log its trace/span identifiers. With tracing
  off or no local span, only a validated incoming trace_id may be logged.
• An unavailable or hanging collector affects neither request handling nor shutdown.
  SIGTERM exits 0 within 1 s.
• govulncheck, Grype and the license check pass. The PR records the binary and image
  size change.
• The router README documents the variables, the emitted spans, metrics and
  attributes, the privacy guarantees and the shutdown budget.

Files likely touched

• containers/router/go.mod and containers/router/go.sum
• the Go main package in containers/router/: a new telemetry.go (the name is a
  suggestion), plus the server, logging and CLI files
• .github/workflows/test-router.yml and .github/dependabot.yml
• containers/router/README.md