Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 20 additions & 10 deletions docs/content/platform/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,17 @@ A series about a deployment also has `deployment`, `replica`, `namespace`,
`engine`, and `role` labels. On a prefill/decode deployment, an engine's series also
has a `phase` label, `Prefill` or `Decode`.

The gateway's `modelplane_frontend_*` series have no `deployment` label. They
carry the model instead: `gen_ai_original_model` is the model the caller sent,
`<namespace>/<service>`, and `gen_ai_request_model` is the deployment the gateway
sent the request to, `<namespace>/<deployment>`.
`modelplane_frontend_requests_active` is counted per `ModelService` instead. It
has a `service` label, and the same `namespace` label as an engine's series:
the team's namespace as it's mirrored on the inference cluster, such as
`mp-ml-team-51733`. The namespace in `gen_ai_original_model` is the team's own,
such as `ml-team`, so the two don't join on it.
The gateway's latency series, `modelplane_frontend_request_duration_seconds`,
`modelplane_frontend_ttft_seconds`, and `modelplane_frontend_tpot_seconds`, have
no `deployment` label. They name the model instead: `service` is the model the
caller sent, `<namespace>/<service>`, and `served_model` is the deployment the
gateway sent the request to, `<namespace>/<deployment>`. For an external
endpoint, `served_model` is the provider's model name.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true for an external endpoint that doesn't set spec.model? That field is optional ("Unset, the caller's model name passes through unchanged"), and compose-model-route only sets modelNameOverride when it's set (compose-model-route/function/fn.py:800-801). Without it, served_model is the same as service, such as ml-team/assistant. In the traffic-share query, that backend's slice then looks like the service itself. Could we say it's the endpoint's spec.model when set and the caller's name otherwise, here and in the comment on _GATEWAY_LABELS?

`modelplane_frontend_requests_active` is counted per `ModelService` instead. Its
`service` label is the service's name alone, and it has the same `namespace`
label as an engine's series: the team's namespace as it's mirrored on the
inference cluster, such as `mp-ml-team-51733`. The namespace in the latency
series' `service` is the team's own, such as `ml-team`, so the two don't join on it.

Each replica publishes its own series, so combine them in your query. Which
combination is right follows from what the metric measures:
Expand Down Expand Up @@ -274,7 +276,15 @@ fleet-wide p99 of time to first token, which requires streaming traffic:

```promql
histogram_quantile(0.99, sum by (le) (
rate(modelplane_frontend_ttft_seconds_bucket{gen_ai_request_model="ml-team/qwen3-8b"}[5m])))
rate(modelplane_frontend_ttft_seconds_bucket{served_model="ml-team/qwen3-8b"}[5m])))
```

Each deployment's share of a service's traffic, such as during a canary:

```promql
sum by (served_model) (rate(modelplane_frontend_request_duration_seconds_count{service="ml-team/assistant"}[5m]))
/ ignoring (served_model) group_left
sum (rate(modelplane_frontend_request_duration_seconds_count{service="ml-team/assistant"}[5m]))
```

Modelplane has no dashboards of its own. What it exports is counters and histogram buckets, and
Expand Down
15 changes: 14 additions & 1 deletion functions/compose-serving-stack/function/stacks/metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,17 @@
"gen_ai_server_time_per_output_token_seconds": "modelplane_frontend_tpot_seconds",
}

# Which model each request was for, under Modelplane's names. The route matches
# on <namespace>/<service>, the name the caller sent, and each endpoint
# overrides it with its served model name, <namespace>/<deployment>, or the
# provider's model name for an external one. served_model stays whole rather
# than split into namespace and deployment, because an external name such as
# Qwen/Qwen3-8B has the same shape.
_GATEWAY_LABELS = [
{"name": "service", "from": "gen_ai_original_model"},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we fine with service meaning ml-team/assistant here but assistant on modelplane_frontend_requests_active? A query like {__name__=~"modelplane_frontend_.*", service="ml-team/assistant"} silently misses the in-flight series. This label matches design/telemetry.md's example, so I'd keep it. Is a follow-up issue to bring requests_active (collector._in_flight_relabels) in line the plan?

{"name": "served_model", "from": "gen_ai_request_model"},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this label be model, the name design/telemetry.md gives it? The design's example frontend series has service="ml-team/assistant", model="Qwen/Qwen3-8B", and its labels section says a frontend series has service alongside model "so a reader can tell the thing asked for from the thing that answered". No series has a model label yet, so nothing would clash. If served_model is the better name, since a deployment's value is <namespace>/<deployment> rather than a model name, could we update the design to match?

]

# The front door's own count of requests in flight to each model, which the
# collector keeps for the fleet gateway's model routes only (see
# collector._IN_FLIGHT). The engines' running and waiting requests well above
Expand Down Expand Up @@ -237,13 +248,15 @@ def _mapping(
tokens: tuple[str, str] | None = None,
extra: tuple[dict[str, object], ...] = (),
types: dict[str, str] | None = None,
labels: list[dict[str, str]] | None = None,
) -> v1alpha1.MetricMapping:
units = units or {}
types = types or {}
metrics: list[dict[str, object]] = [
{"from": src, "to": dst}
| ({"fromUnit": units[src]} if src in units else {})
| ({"fromType": types[src]} if src in types else {})
| ({"labels": labels} if labels else {})
for src, dst in pairs.items()
]
if tokens:
Expand All @@ -263,7 +276,7 @@ def mappings() -> list[v1alpha1.MetricMapping]:
operator's do, and a built-in that breaks breaks the path everyone uses.
"""
return [
_mapping("modelplane-gateway", _GATEWAY),
_mapping("modelplane-gateway", _GATEWAY, labels=_GATEWAY_LABELS),
_mapping("modelplane-gateway-proxy", _GATEWAY_PROXY),
_mapping("modelplane-vllm", _VLLM, tokens=_VLLM_TOKENS, extra=(_VLLM_RESPONSES,)),
_mapping("modelplane-sglang", _SGLANG, tokens=_SGLANG_TOKENS, extra=(_SGLANG_RESPONSES,)),
Expand Down
30 changes: 27 additions & 3 deletions functions/compose-serving-stack/tests/test_collector.py
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,18 @@ def _config(
{
"context": "datapoint",
"statements": [
'set(datapoint.attributes["service"], datapoint.attributes["gen_ai_original_model"]) where metric.name == "gen_ai_server_request_duration_seconds"',
'delete_key(datapoint.attributes, "gen_ai_original_model") where metric.name == "gen_ai_server_request_duration_seconds"',
'set(datapoint.attributes["served_model"], datapoint.attributes["gen_ai_request_model"]) where metric.name == "gen_ai_server_request_duration_seconds"',
'delete_key(datapoint.attributes, "gen_ai_request_model") where metric.name == "gen_ai_server_request_duration_seconds"',
'set(datapoint.attributes["service"], datapoint.attributes["gen_ai_original_model"]) where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_original_model") where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'set(datapoint.attributes["served_model"], datapoint.attributes["gen_ai_request_model"]) where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_request_model") where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'set(datapoint.attributes["service"], datapoint.attributes["gen_ai_original_model"]) where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_original_model") where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'set(datapoint.attributes["served_model"], datapoint.attributes["gen_ai_request_model"]) where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_request_model") where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'set(datapoint.attributes["direction"], "input") where metric.name == "vllm:request_prompt_tokens_sum"',
'set(datapoint.attributes["direction"], "output") where metric.name == "vllm:request_generation_tokens_sum"',
'set(datapoint.attributes["reason"], datapoint.attributes["finished_reason"]) where metric.name == "vllm:request_success_total"',
Expand Down Expand Up @@ -977,6 +989,18 @@ def test_in_flight(case: InFlightCase) -> None:
'convert_gauge_to_sum("cumulative", true) where metric.name == "gpu_energy_consumed"',
],
[
'set(datapoint.attributes["service"], datapoint.attributes["gen_ai_original_model"]) where metric.name == "gen_ai_server_request_duration_seconds"',
'delete_key(datapoint.attributes, "gen_ai_original_model") where metric.name == "gen_ai_server_request_duration_seconds"',
'set(datapoint.attributes["served_model"], datapoint.attributes["gen_ai_request_model"]) where metric.name == "gen_ai_server_request_duration_seconds"',
'delete_key(datapoint.attributes, "gen_ai_request_model") where metric.name == "gen_ai_server_request_duration_seconds"',
'set(datapoint.attributes["service"], datapoint.attributes["gen_ai_original_model"]) where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_original_model") where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'set(datapoint.attributes["served_model"], datapoint.attributes["gen_ai_request_model"]) where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_request_model") where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'set(datapoint.attributes["service"], datapoint.attributes["gen_ai_original_model"]) where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_original_model") where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'set(datapoint.attributes["served_model"], datapoint.attributes["gen_ai_request_model"]) where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_request_model") where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'set(datapoint.attributes["direction"], "input") where metric.name == "vllm:request_prompt_tokens_sum"',
'set(datapoint.attributes["direction"], "output") where metric.name == "vllm:request_generation_tokens_sum"',
'set(datapoint.attributes["reason"], datapoint.attributes["finished_reason"]) where metric.name == "vllm:request_success_total"',
Expand Down Expand Up @@ -1245,7 +1269,7 @@ def test_transform(case: TransformCase) -> None:
(
"collector",
_deployment(
config_hash="522e31c7cedc07c5",
config_hash="fdd24284aba7958f",
volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}],
volume_mounts=[{"name": "config", "mountPath": "/conf"}],
env_from=None,
Expand Down Expand Up @@ -1290,7 +1314,7 @@ def test_transform(case: TransformCase) -> None:
(
"collector",
_deployment(
config_hash="1f03e323432223be",
config_hash="ceb2f66f9ded10e2",
volumes=[
{"name": "config", "configMap": {"name": "modelplane-collector"}},
{"name": "credentials-primary", "secret": {"secretName": "telemetry-credentials"}},
Expand Down Expand Up @@ -1357,7 +1381,7 @@ def test_transform(case: TransformCase) -> None:
(
"collector",
_deployment(
config_hash="b1539f5a25d85bfc",
config_hash="653cb26ec99e9671",
volumes=[
{"name": "config", "configMap": {"name": "modelplane-collector"}},
{"name": "credentials-vendor", "secret": {"secretName": "vendor-token"}},
Expand Down
22 changes: 17 additions & 5 deletions functions/compose-serving-stack/tests/test_fn.py
Original file line number Diff line number Diff line change
Expand Up @@ -1694,6 +1694,18 @@ def _collector_config(
{
"context": "datapoint",
"statements": [
'set(datapoint.attributes["service"], datapoint.attributes["gen_ai_original_model"]) where metric.name == "gen_ai_server_request_duration_seconds"',
'delete_key(datapoint.attributes, "gen_ai_original_model") where metric.name == "gen_ai_server_request_duration_seconds"',
'set(datapoint.attributes["served_model"], datapoint.attributes["gen_ai_request_model"]) where metric.name == "gen_ai_server_request_duration_seconds"',
'delete_key(datapoint.attributes, "gen_ai_request_model") where metric.name == "gen_ai_server_request_duration_seconds"',
'set(datapoint.attributes["service"], datapoint.attributes["gen_ai_original_model"]) where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_original_model") where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'set(datapoint.attributes["served_model"], datapoint.attributes["gen_ai_request_model"]) where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_request_model") where metric.name == "gen_ai_server_time_to_first_token_seconds"',
'set(datapoint.attributes["service"], datapoint.attributes["gen_ai_original_model"]) where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_original_model") where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'set(datapoint.attributes["served_model"], datapoint.attributes["gen_ai_request_model"]) where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'delete_key(datapoint.attributes, "gen_ai_request_model") where metric.name == "gen_ai_server_time_per_output_token_seconds"',
'set(datapoint.attributes["direction"], "input") where metric.name == "vllm:request_prompt_tokens_sum"',
'set(datapoint.attributes["direction"], "output") where metric.name == "vllm:request_generation_tokens_sum"',
'set(datapoint.attributes["reason"], datapoint.attributes["finished_reason"]) where metric.name == "vllm:request_success_total"',
Expand Down Expand Up @@ -4746,7 +4758,7 @@ def _to_dict(msg: message.Message) -> dict:
service_extensions=None,
),
"collector": _collector(
config_hash="be9842417e13d6c1",
config_hash="6a3bc6ff5fb2ef74",
volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}],
volume_mounts=[{"name": "config", "mountPath": "/conf"}],
env_from=None,
Expand Down Expand Up @@ -5065,7 +5077,7 @@ def _to_dict(msg: message.Message) -> dict:
ready=fnv1.READY_TRUE,
),
"collector": _collector(
config_hash="5d3c556be06d8ca8",
config_hash="096cca2888bd9898",
volumes=[
{"name": "config", "configMap": {"name": "modelplane-collector"}},
{"name": "credentials-primary", "secret": {"secretName": "telemetry-credentials"}},
Expand Down Expand Up @@ -5352,7 +5364,7 @@ def _to_dict(msg: message.Message) -> dict:
service_extensions=None,
),
"collector": _collector(
config_hash="0a72aaee56455e72",
config_hash="c350471b6a11500d",
volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}],
volume_mounts=[{"name": "config", "mountPath": "/conf"}],
env_from=None,
Expand Down Expand Up @@ -5615,7 +5627,7 @@ def _to_dict(msg: message.Message) -> dict:
service_extensions=None,
),
"collector": _collector(
config_hash="57e31b8f680bd6d3",
config_hash="92e391f460986abd",
volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}],
volume_mounts=[{"name": "config", "mountPath": "/conf"}],
env_from=None,
Expand Down Expand Up @@ -5906,7 +5918,7 @@ def _to_dict(msg: message.Message) -> dict:
service_extensions=None,
),
"collector": _collector(
config_hash="be9842417e13d6c1",
config_hash="6a3bc6ff5fb2ef74",
volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}],
volume_mounts=[{"name": "config", "mountPath": "/conf"}],
env_from=None,
Expand Down
Loading