Repository navigation
Conversation
The modelplane_frontend_* latency series had no Modelplane label for the model a request was for, only Envoy AI Gateway's gen_ai_original_model and gen_ai_request_model. Grouping by deployment or service meant knowing the gateway's attribute names, and the telemetry docs' TTFT example used them. The modelplane-gateway mapping now renames gen_ai_original_model to service and gen_ai_request_model to served_model, through the labels field the other built-ins use. served_model stays whole: an external endpoint's model name, such as Qwen/Qwen3-8B, has the same shape as <namespace>/<deployment>, so no pattern can split it safely. The telemetry docs now use the new labels and show a deployment's share of a service's traffic. Fixes modelplaneai#534. Signed-off-by: ChethanUK <chethanuk@outlook.com>
|
Docs preview: https://modelplane-docs-pr-675.vercel.app (ready once the site's Content workflow finishes) |
|
/review |
|
/review (Sorry, bot was broken) |
There was a problem hiding this comment.
This looks close to mergeable. It takes the approach issue #534 suggested, and what's left is a label name, a label format, and one inaccurate sentence in the docs and a code comment.
The naming questions weigh against design/telemetry.md, which already names labels for these series, and renaming a label later breaks every query written against it.
Reviewed by modelplane-review 22e194c. Reply, or react 👍 or 👎, on each comment to help tune it.
| # Qwen/Qwen3-8B has the same shape. | ||
| _GATEWAY_LABELS = [ | ||
| {"name": "service", "from": "gen_ai_original_model"}, | ||
| {"name": "served_model", "from": "gen_ai_request_model"}, |
There was a problem hiding this comment.
Should this label be model, the name design/telemetry.md gives it? The design's example frontend series has service="ml-team/assistant", model="Qwen/Qwen3-8B", and its labels section says a frontend series has service alongside model "so a reader can tell the thing asked for from the thing that answered". No series has a model label yet, so nothing would clash. If served_model is the better name, since a deployment's value is <namespace>/<deployment> rather than a model name, could we update the design to match?
| no `deployment` label. They name the model instead: `service` is the model the | ||
| caller sent, `<namespace>/<service>`, and `served_model` is the deployment the | ||
| gateway sent the request to, `<namespace>/<deployment>`. For an external | ||
| endpoint, `served_model` is the provider's model name. |
There was a problem hiding this comment.
Is this true for an external endpoint that doesn't set spec.model? That field is optional ("Unset, the caller's model name passes through unchanged"), and compose-model-route only sets modelNameOverride when it's set (compose-model-route/function/fn.py:800-801). Without it, served_model is the same as service, such as ml-team/assistant. In the traffic-share query, that backend's slice then looks like the service itself. Could we say it's the endpoint's spec.model when set and the caller's name otherwise, here and in the comment on _GATEWAY_LABELS?
| # than split into namespace and deployment, because an external name such as | ||
| # Qwen/Qwen3-8B has the same shape. | ||
| _GATEWAY_LABELS = [ | ||
| {"name": "service", "from": "gen_ai_original_model"}, |
There was a problem hiding this comment.
Are we fine with service meaning ml-team/assistant here but assistant on modelplane_frontend_requests_active? A query like {__name__=~"modelplane_frontend_.*", service="ml-team/assistant"} silently misses the in-flight series. This label matches design/telemetry.md's example, so I'd keep it. Is a follow-up issue to bring requests_active (collector._in_flight_relabels) in line the plan?
Description of your changes
Fixes #534.
The gateway's latency series,
modelplane_frontend_request_duration_seconds,_ttft_secondsand_tpot_seconds, had no Modelplane label for the model a request was for. You could only group them by Envoy AI Gateway'sgen_ai_original_modelandgen_ai_request_model.The
modelplane-gatewaymapping now uses the existinglabelsfield to renamegen_ai_original_modeltoserviceandgen_ai_request_modeltoserved_model. I leftserved_modelwhole: an external model name likeQwen/Qwen3-8Bhas the same shape as<namespace>/<deployment>. The label compiler deletes the source attribute, so thegen_ai_*names go away on these series. Anyone already querying them will need to switch.One thing to look at:
serviceon these series is<namespace>/<service>, but onmodelplane_frontend_requests_activeit's the bare service name. The telemetry docs say so. They also use the new labels in the quantile example and add a deployment's share of a service's traffic.I ran otelcol-contrib 0.161.0 against a static gateway scrape with the rendered
transform/modelplane. The series came out asmodelplane_frontend_request_duration_secondswithserviceandserved_modelset,gen_ai_response_modelkept, and the other twogen_ai_*attributes removed.I have:
RunNo nix here; ran the function tests,nix flake check(or./nix.sh flake check) and made sure it passes.hack/check_function_tests.py, ruff and Vale directly.git commit -s.