Skip to content

Gateway metrics can't be grouped by deployment or service #534

Open
chethanuk/modelplane
#4

Description

@dennis-upbound

The modelplane_frontend_* series have no deployment, namespace or service label. The modelplane-gateway-genai scrape job in functions/compose-serving-stack/function/collector.py relabels nothing, and the _GATEWAY mapping in stacks/metrics.py renames only the metric name. The quantile example in docs/content/platform/telemetry.md, modelplane_frontend_ttft_seconds_bucket{deployment="qwen3-8b"}, matches no series.

The identity is there, under Envoy AI Gateway's attribute names. The AIGatewayRoute matches on <namespace>/<service>, and each ModelEndpoint's modelNameOverride is <namespace>/<deployment> (served_model_name in compose-model-deployment). As a result:

  • gen_ai_original_model is the service the caller asked for.
  • gen_ai_request_model is the deployment the request was routed to. For an external endpoint, it is the provider's model name.
  • gen_ai_response_model is what the engine echoed back. On a failed request it is unknown.

Comparing two deployments of one model, or watching a canary, needs exactly this. For example, a deployment's share of a service's traffic:

sum by (gen_ai_request_model) (rate(modelplane_frontend_request_duration_seconds_count{gen_ai_original_model="ml-team/assistant"}[5m]))

The docs don't mention this.

Suggestion:

  • In the _GATEWAY mapping, use the existing labels field to carry gen_ai_original_model onto service and gen_ai_request_model onto served_model.
  • Don't split served_model into namespace/deployment with a pattern. An external model name like Qwen/Qwen3-8B has the same shape.
  • Fix the docs example and add the traffic-share query.

The gateway can't attribute a request to a single replica. The picker can (see #533).

Activity

  1. moved this from Backlog to In progress in Modelplane Roadmapon Nov 13, 2026
  2. added a commit that references this issue on Nov 16, 2026
    baba04d
  3. chethanuk commented on Nov 16, 2026

    @chethanuk

    I'd like to take this one. I'd follow the suggestion here and rename the gateway's model attributes through the modelplane-gateway mapping's labels field.

    • gen_ai_original_model becomes service and gen_ai_request_model becomes served_model on the three latency series. The rename deletes the source attribute, so the old names go away on these series.
    • served_model stays whole, since Qwen/Qwen3-8B has the same shape as <namespace>/<deployment>. gen_ai_response_model is left alone.
    • the telemetry docs' TTFT example switches to served_model, and I've added the traffic-share query. The docs also note that service here is <namespace>/<service>, while modelplane_frontend_requests_active uses the bare name.
    • per-replica attribution stays out of scope (No per-deployment request outcome by HTTP status #533).
    • I've updated the function tests and checked the rendered config with otelcol-contrib 0.161.0 against a static gateway scrape.

    I have a draft PR ready and will link it here.

  4. chethanuk commented on Nov 16, 2026

    @chethanuk

    Draft PR is up: #675

  5. added
    g/gateway-and-routingMake model access independent of where models run, with routing operators can control and extend
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    g/gateway-and-routingMake model access independent of where models run, with routing operators can control and extend

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions