api-proxy: trace model-router calls and send W3C trace context to the router #10413

Part of github/gh-aw-firewall#10261
(step 7a of 7; step 7b covers the router's own export configuration).

Prerequisite

The Go router reads W3C traceparent and tracestate, ignores baggage, and, when its own
export is configured, makes its spans children of the incoming context (#10380).

Summary

The parent issue, #10261, requires trace-context correlation between AWF and the router. Today the api-proxy calls
the router without trace context and without spans of its own, so routing is invisible in a
workflow's trace, and router spans, when exported, would be unconnected roots.

Add an api-proxy CLIENT span around each router call made while planning, send its
traceparent (and tracestate) to the router, and record the routing outcome on the route
span. These spans use the api-proxy's existing exporter, so they reach the workflow's collector
(or the otel.jsonl file fallback) today, whether or not router export is configured. The
router's export configuration is a separate step (7b).

Background

• containers/api-proxy/otel.js creates request spans and exports them through Squid, or
  writes /var/log/api-proxy/otel.jsonl when no endpoint is configured. _buildParentContext
  supplies the workflow parent from GITHUB_AW_OTEL_TRACE_ID and
  GITHUB_AW_OTEL_PARENT_SPAN_ID.
• The api-proxy plans once at startup in routing-controller.js: health, capabilities,
  classify and route, through createRoutingRouterClient in routing-router-client.js,
  which routing-runtime.js constructs. Planning runs outside any proxied LLM request, so no
  active request span exists.
• callPlannerWithRetry may call an operation more than once; each attempt is a separate HTTP
  call.

Scope and specification

• Span helper in otel.js. Export a tracing hook whose startCall starts a CLIENT span
  named api_proxy.router.<operation> (like the LLM request spans' api_proxy.<provider>.request) for capabilities, classify and route, and returns the
  span's trace headers and an end function. Health checks and unknown operations aren't
  traced. The parent is a caller-supplied context when it holds a valid span, otherwise the
  workflow parent, otherwise a new root. There's no shared mutable per-request parent state.
• Headers. Return only traceparent and, when the span context has one, tracestate,
  formatted from the span context. Never send baggage. No new dependency is needed.
• Attributes. http.request.method (GET/POST, otherwise _OTHER), url.path (fixed
  per operation), the configured server.address and server.port,
  http.response.status_code when a response arrived, and on failure error.type: the HTTP
  status for an error response, otherwise one of a fixed set of codes (socket codes such as
  ETIMEDOUT and ECONNREFUSED, plus routing_cancelled and routing_contract_error), or
  _OTHER. A successful call ends with status OK, as the LLM request spans do; a failed call
  ends with status ERROR whose message is the error type. Error messages and exceptions are
  never recorded.
• Router client. createRoutingRouterClient takes an optional tracing hook. A call's span
  starts when the request is sent and ends in the client's existing single settle path, so it
  ends exactly once on success, error status, malformed or oversized response, transport
  failure, timeout or cancellation. A request rejected before it's sent starts no span.
  runBoundedOperation in routing-planner.js aborts a timed-out attempt with a
  TimeoutError reason, so that attempt's span records ETIMEDOUT rather than
  routing_cancelled; the routing error it returns is unchanged.
  Without the hook, or when the hook throws, requests and results are unchanged.
• Routing outcome on the route span. The controller passes, per route call, the
  attributes gh_aw_router.goal and gh_aw_router.mode (from the validated routing config) and
  gh_aw_router.classification.present, plus a describer that runs validateRouteResponse
  against the offered choices and returns gh_aw_router.selected.model and
  gh_aw_router.selected.effort from the validated top choice. The client calls the describer
  only on success and never reads response fields itself; a failed or invalid response gets no
  selection. otel.js keeps only these five attribute names with the expected types and a
  length limit. No resolved_mode (the router doesn't return it), choice ids, current_id,
  classifier labels, conversation or response text.
• Runtime wiring. routing-runtime.js supplies the hook from otel.js, loaded optionally
  as proxy-request.js does, and accepts an override for tests.

Out of scope

• Router export configuration, compose environment, agent exclusion and snapshot redaction
  (step 7b).
• Routing algorithm, router contracts and router instrumentation.
• Audit data: firewall.model_routing in usage/aw_session.jsonl stays the record of routing
  decisions; these spans are for trace views.

Acceptance criteria

• Each capabilities, classify and route call has an api_proxy.router.<operation> client
  span in the workflow trace and sends that span's traceparent; health checks aren't
  traced and baggage is never sent.
• Spans end exactly once on every outcome, and tracing failures never change routing.
• A successful route span carries the goal, mode, classification flag and validated
  selected model and effort, and no other routing data; a failed or invalid response
  carries no selection.
• Error spans carry only a fixed error.type, never error text; a planner attempt
  timeout is recorded as ETIMEDOUT, not as a cancellation.
• Documentation describes the spans, attributes and privacy rules.

Validation

• otel.test.js: span name, kind and attributes; trace headers match the span; parent
  selection (owned context, workflow parent, invalid context, root); tracestate
  forwarding; error-type mapping and canaries; end-once; attribute allowlist; and an
  end-to-end check with the real client and an HTTP server that the router receives each
  span's own traceparent.
• routing-router-client.test.js: headers, health exclusion, span completion on each outcome,
  no span before sending, a planner attempt timeout recorded as ETIMEDOUT, and unchanged
  behavior when tracing is absent or throws.
• routing-planner.test.js: a timed-out attempt aborts with a TimeoutError reason and a
  cancelled one doesn't.
• routing-controller.test.js: route trace attributes with and without classification, and the
  describer rejecting a choice that wasn't offered.
• routing-runtime.test.js: the default hook, an unbundled otel.js, and an explicit hook.

Files touched

• containers/api-proxy/otel.js, routing-router-client.js, routing-controller.js,
  routing-planner.js, routing-runtime.js and their tests
• docs/otel-sentry.md and containers/router/README.md