You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Gateway metrics can't be grouped by deployment or service #534
The modelplane_frontend_* series have no deployment, namespace or service label. The modelplane-gateway-genai scrape job in functions/compose-serving-stack/function/collector.py relabels nothing, and the _GATEWAY mapping in stacks/metrics.py renames only the metric name. The quantile example in docs/content/platform/telemetry.md, modelplane_frontend_ttft_seconds_bucket{deployment="qwen3-8b"}, matches no series.
The identity is there, under Envoy AI Gateway's attribute names. The AIGatewayRoute matches on <namespace>/<service>, and each ModelEndpoint's modelNameOverride is <namespace>/<deployment> (served_model_name in compose-model-deployment). As a result:
gen_ai_original_model is the service the caller asked for.
gen_ai_request_model is the deployment the request was routed to. For an external endpoint, it is the provider's model name.
gen_ai_response_model is what the engine echoed back. On a failed request it is unknown.
Comparing two deployments of one model, or watching a canary, needs exactly this. For example, a deployment's share of a service's traffic:
sum by (gen_ai_request_model) (rate(modelplane_frontend_request_duration_seconds_count{gen_ai_original_model="ml-team/assistant"}[5m]))
The docs don't mention this.
Suggestion:
In the _GATEWAY mapping, use the existing labels field to carry gen_ai_original_model onto service and gen_ai_request_model onto served_model.
Don't split served_model into namespace/deployment with a pattern. An external model name like Qwen/Qwen3-8B has the same shape.
Fix the docs example and add the traffic-share query.
The gateway can't attribute a request to a single replica. The picker can (see #533).
I'd like to take this one. I'd follow the suggestion here and rename the gateway's model attributes through the modelplane-gateway mapping's labels field.
gen_ai_original_model becomes service and gen_ai_request_model becomes served_model on the three latency series. The rename deletes the source attribute, so the old names go away on these series.
served_model stays whole, since Qwen/Qwen3-8B has the same shape as <namespace>/<deployment>. gen_ai_response_model is left alone.
the telemetry docs' TTFT example switches to served_model, and I've added the traffic-share query. The docs also note that service here is <namespace>/<service>, while modelplane_frontend_requests_active uses the bare name.
Activity
Label the gateway's latency series by service and served model
chethanuk commentedon Nov 16, 2026
I'd like to take this one. I'd follow the suggestion here and rename the gateway's model attributes through the
modelplane-gatewaymapping'slabelsfield.gen_ai_original_modelbecomesserviceandgen_ai_request_modelbecomesserved_modelon the three latency series. The rename deletes the source attribute, so the old names go away on these series.served_modelstays whole, sinceQwen/Qwen3-8Bhas the same shape as<namespace>/<deployment>.gen_ai_response_modelis left alone.served_model, and I've added the traffic-share query. The docs also note thatservicehere is<namespace>/<service>, whilemodelplane_frontend_requests_activeuses the bare name.I have a draft PR ready and will link it here.
chethanuk commentedon Nov 16, 2026
Draft PR is up: #675