Skip to content

Label the gateway's latency series by service and served model - #675

Open
chethanuk wants to merge 1 commit into
modelplaneai:mainfrom
chethanuk:oss/534-gateway-metric-labels
Open

chethanuk wants to merge 1 commit into
modelplaneai:mainfrom
chethanuk:oss/534-gateway-metric-labels

Conversation

@chethanuk

Copy link
Copy Markdown

Description of your changes

Fixes #534.

The gateway's latency series, modelplane_frontend_request_duration_seconds, _ttft_seconds and _tpot_seconds, had no Modelplane label for the model a request was for. You could only group them by Envoy AI Gateway's gen_ai_original_model and gen_ai_request_model.

The modelplane-gateway mapping now uses the existing labels field to rename gen_ai_original_model to service and gen_ai_request_model to served_model. I left served_model whole: an external model name like Qwen/Qwen3-8B has the same shape as <namespace>/<deployment>. The label compiler deletes the source attribute, so the gen_ai_* names go away on these series. Anyone already querying them will need to switch.

One thing to look at: service on these series is <namespace>/<service>, but on modelplane_frontend_requests_active it's the bare service name. The telemetry docs say so. They also use the new labels in the quantile example and add a deployment's share of a service's traffic.

I ran otelcol-contrib 0.161.0 against a static gateway scrape with the rendered transform/modelplane. The series came out as modelplane_frontend_request_duration_seconds with service and served_model set, gen_ai_response_model kept, and the other two gen_ai_* attributes removed.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes. No nix here; ran the function tests, hack/check_function_tests.py, ruff and Vale directly.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

The modelplane_frontend_* latency series had no Modelplane label for the
model a request was for, only Envoy AI Gateway's gen_ai_original_model
and gen_ai_request_model. Grouping by deployment or service meant knowing
the gateway's attribute names, and the telemetry docs' TTFT example used
them.

The modelplane-gateway mapping now renames gen_ai_original_model to
service and gen_ai_request_model to served_model, through the labels
field the other built-ins use. served_model stays whole: an external
endpoint's model name, such as Qwen/Qwen3-8B, has the same shape as
<namespace>/<deployment>, so no pattern can split it safely.

The telemetry docs now use the new labels and show a deployment's share
of a service's traffic.

Fixes modelplaneai#534.

Signed-off-by: ChethanUK <chethanuk@outlook.com>
@github-actions

Copy link
Copy Markdown

Docs preview: https://modelplane-docs-pr-675.vercel.app (ready once the site's Content workflow finishes)

@chethanuk
chethanuk marked this pull request as ready for review November 17, 2026 04:58
@haarchri

Copy link
Copy Markdown
Contributor

/review

@dennis-upbound dennis-upbound left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me once the review bot comments are addressed. Thank you!

@negz

negz commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

/review

(Sorry, bot was broken)

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks close to mergeable. It takes the approach issue #534 suggested, and what's left is a label name, a label format, and one inaccurate sentence in the docs and a code comment.

The naming questions weigh against design/telemetry.md, which already names labels for these series, and renaming a label later breaks every query written against it.

Reviewed by modelplane-review 22e194c. Reply, or react 👍 or 👎, on each comment to help tune it.

# Qwen/Qwen3-8B has the same shape.
_GATEWAY_LABELS = [
{"name": "service", "from": "gen_ai_original_model"},
{"name": "served_model", "from": "gen_ai_request_model"},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this label be model, the name design/telemetry.md gives it? The design's example frontend series has service="ml-team/assistant", model="Qwen/Qwen3-8B", and its labels section says a frontend series has service alongside model "so a reader can tell the thing asked for from the thing that answered". No series has a model label yet, so nothing would clash. If served_model is the better name, since a deployment's value is <namespace>/<deployment> rather than a model name, could we update the design to match?

no `deployment` label. They name the model instead: `service` is the model the
caller sent, `<namespace>/<service>`, and `served_model` is the deployment the
gateway sent the request to, `<namespace>/<deployment>`. For an external
endpoint, `served_model` is the provider's model name.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true for an external endpoint that doesn't set spec.model? That field is optional ("Unset, the caller's model name passes through unchanged"), and compose-model-route only sets modelNameOverride when it's set (compose-model-route/function/fn.py:800-801). Without it, served_model is the same as service, such as ml-team/assistant. In the traffic-share query, that backend's slice then looks like the service itself. Could we say it's the endpoint's spec.model when set and the caller's name otherwise, here and in the comment on _GATEWAY_LABELS?

# than split into namespace and deployment, because an external name such as
# Qwen/Qwen3-8B has the same shape.
_GATEWAY_LABELS = [
{"name": "service", "from": "gen_ai_original_model"},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we fine with service meaning ml-team/assistant here but assistant on modelplane_frontend_requests_active? A query like {__name__=~"modelplane_frontend_.*", service="ml-team/assistant"} silently misses the in-flight series. This label matches design/telemetry.md's example, so I'd keep it. Is a follow-up issue to bring requests_active (collector._in_flight_relabels) in line the plan?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

Gateway metrics can't be grouped by deployment or service

4 participants