Repository navigation
Label the gateway's latency series by service and served model #675
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -53,6 +53,17 @@ | |
| "gen_ai_server_time_per_output_token_seconds": "modelplane_frontend_tpot_seconds", | ||
| } | ||
|
|
||
| # Which model each request was for, under Modelplane's names. The route matches | ||
| # on <namespace>/<service>, the name the caller sent, and each endpoint | ||
| # overrides it with its served model name, <namespace>/<deployment>, or the | ||
| # provider's model name for an external one. served_model stays whole rather | ||
| # than split into namespace and deployment, because an external name such as | ||
| # Qwen/Qwen3-8B has the same shape. | ||
| _GATEWAY_LABELS = [ | ||
| {"name": "service", "from": "gen_ai_original_model"}, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are we fine with |
||
| {"name": "served_model", "from": "gen_ai_request_model"}, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should this label be |
||
| ] | ||
|
|
||
| # The front door's own count of requests in flight to each model, which the | ||
| # collector keeps for the fleet gateway's model routes only (see | ||
| # collector._IN_FLIGHT). The engines' running and waiting requests well above | ||
|
|
@@ -237,13 +248,15 @@ def _mapping( | |
| tokens: tuple[str, str] | None = None, | ||
| extra: tuple[dict[str, object], ...] = (), | ||
| types: dict[str, str] | None = None, | ||
| labels: list[dict[str, str]] | None = None, | ||
| ) -> v1alpha1.MetricMapping: | ||
| units = units or {} | ||
| types = types or {} | ||
| metrics: list[dict[str, object]] = [ | ||
| {"from": src, "to": dst} | ||
| | ({"fromUnit": units[src]} if src in units else {}) | ||
| | ({"fromType": types[src]} if src in types else {}) | ||
| | ({"labels": labels} if labels else {}) | ||
| for src, dst in pairs.items() | ||
| ] | ||
| if tokens: | ||
|
|
@@ -263,7 +276,7 @@ def mappings() -> list[v1alpha1.MetricMapping]: | |
| operator's do, and a built-in that breaks breaks the path everyone uses. | ||
| """ | ||
| return [ | ||
| _mapping("modelplane-gateway", _GATEWAY), | ||
| _mapping("modelplane-gateway", _GATEWAY, labels=_GATEWAY_LABELS), | ||
| _mapping("modelplane-gateway-proxy", _GATEWAY_PROXY), | ||
| _mapping("modelplane-vllm", _VLLM, tokens=_VLLM_TOKENS, extra=(_VLLM_RESPONSES,)), | ||
| _mapping("modelplane-sglang", _SGLANG, tokens=_SGLANG_TOKENS, extra=(_SGLANG_RESPONSES,)), | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this true for an external endpoint that doesn't set
spec.model? That field is optional ("Unset, the caller's model name passes through unchanged"), and compose-model-route only setsmodelNameOverridewhen it's set (compose-model-route/function/fn.py:800-801). Without it,served_modelis the same asservice, such asml-team/assistant. In the traffic-share query, that backend's slice then looks like the service itself. Could we say it's the endpoint'sspec.modelwhen set and the caller's name otherwise, here and in the comment on_GATEWAY_LABELS?