Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 43 additions & 2 deletions apis/metricmappings/definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,10 @@ spec:
renames for is additive: its renames run after the built-in
ones, on whatever those left behind. A metric a built-in
already renamed no longer answers to the name it was emitted
under, so a second mapping selecting on that name matches
nothing and the built-in stands. Select on the `modelplane_`
under, so a second mapping's rename of that name does nothing
and the built-in stands. Its `match` still applies, because
matching runs before any rename and drops the series it doesn't
keep wherever the metric appears. Select on the `modelplane_`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't a match on a modelplane_ from silently do nothing? This sentence points users at the modelplane_ name, but filter/match runs before transform/modelplane, while the series still has the engine's name. So metric.name == "modelplane_requests_running" never holds and the filter keeps every series.

Could a CEL rule reject match on a modelplane_ from, or could the match description say it only applies to the emitted name?

name instead to rename one of Modelplane's own.
properties:
metrics:
Expand Down Expand Up @@ -151,6 +153,45 @@ spec:
putting an engine's own vocabulary into
Modelplane's. A value with no entry here is left
as the component wrote it.
match:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is match in the right place? As a field on one from→to entry it reads as "which series this rename takes", but conditions() compiles it to a drop keyed only on from. So it also filters sibling entries, other mappings and the built-in. Both this description and the spec-level metrics one have to explain that, and the tp_rank recipe in the docs has the user write a rename that "does nothing" just to hold a match.

Could a spec-level list of series to keep, keyed by metric name, describe what it does more directly? The cost is a second place to name a metric.

type: array
maxItems: 16
description: >-
Keep only this metric's series whose labels match every
entry, and drop the rest before the rename. For a
component that publishes the same count more than once,
such as a total beside a breakdown of it, which would
otherwise be summed twice.

It applies to the metric wherever it appears, so a
second mapping of the same `from`, such as one taking
a part of the same histogram, sees only the series it
keeps.
items:
type: object
required: [label]
x-kubernetes-validations:
- rule: "has(self.value) != has(self.absent)"
message: set either value, to keep series carrying it, or absent, to keep series without the label.
properties:
label:
type: string
maxLength: 63
pattern: '^[a-zA-Z_][a-zA-Z0-9_]*$'
description: The label to match on.
value:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't value: "" drop every series of the metric? The Prometheus receiver drops an empty label, and the attribute arrives as nil. In OTTL nil != "" is true. The condition then drops series without the label as well as those with any other value. A run against otelcol-contrib 0.161.0 confirmed that my_engine_x{kind=""} and {kind="a"} both disappear. It's an easy thing to write, since PromQL selects a missing label with ="".

Could we add minLength: 1 here, or compile value: "" the same way as absent?

type: string
maxLength: 253
description: >-
Keep series where the label has this value. A
series without the label doesn't match.
absent:
type: boolean
enum: [true]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would an operator discriminator serve better than value versus absent: true? A boolean with enum: [true] can't grow additively into "not this value", "one of these" or "present". Each would need a new sibling field and a wider CEL rule. The ModelDeployment tolerations already use operator: [Exists, Equal] (apis/modeldeployments/definition.yaml:328), and users know the same structure from matchExpressions:

match:
- label: priority
  operator: DoesNotExist

The downside is one more field on the common case.

description: >-
Keep series without the label. A label with an
empty value counts as absent, as it does in
Prometheus.
fromUnit:
type: string
enum: [Millijoules, Microjoules, Mebibytes, Milliseconds, Nanoseconds, Percent]
Expand Down
32 changes: 26 additions & 6 deletions docs/content/platform/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -387,12 +387,32 @@ spec:
to: modelplane_kv_offload_load_seconds_total
```

SGLang publishes `modelplane_requests_running` and `modelplane_requests_waiting` more
than once per engine under `--enable-priority-scheduling`, once in total and once per
priority, and once per tensor-parallel rank under
`--enable-metrics-for-all-schedulers`. With either flag, a sum over them counts each
request more than once. Keeping only one of those series is tracked in
[#549](https://github.com/modelplaneai/modelplane/issues/549).
With `--enable-priority-scheduling`, SGLang publishes its running and waiting counts
once in total and once per priority. The built-in mapping keeps only the total. With
`--enable-metrics-for-all-schedulers`, it publishes them once per tensor-parallel rank,
and a sum over `modelplane_requests_running` or `modelplane_requests_waiting` counts each
request once per rank. To keep one rank's series, give a mapping of

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should #549 get a follow-up rather than close with this PR? This tp_rank: "0" recipe for --enable-metrics-for-all-schedulers undercounts with DP attention, and the caveat only comes after the example. If the right series under DP attention is "each group's first rank", can match express that at all?

`sglang:num_running_reqs` and `sglang:num_queue_reqs` a `match` on `tp_rank`:

```yaml
- from: sglang:num_running_reqs
to: modelplane_requests_running
match:
- label: tp_rank
value: "0"
- from: sglang:num_queue_reqs
to: modelplane_requests_waiting
match:
- label: tp_rank
value: "0"
```

The built-in mapping has already renamed these metrics, so this mapping's rename does
nothing, but its `match` still applies: matching runs before any rename, on the name
SGLang emitted, wherever the metric appears.

With DP attention, each group's first rank publishes its own count, so keeping
`tp_rank: "0"` drops every other group's.

Because the gateway, not the engine, measures the `modelplane_frontend_*` series, they
work without configuration for an engine Modelplane has never seen.
Expand Down
32 changes: 30 additions & 2 deletions functions/compose-serving-stack/function/collector.py
Original file line number Diff line number Diff line change
Expand Up @@ -486,6 +486,30 @@ def statements(mappings: list[mmv1alpha1.MetricMapping]) -> tuple[list[str], lis
return extract, scale, datapoint, metric


def conditions(mappings: list[mmv1alpha1.MetricMapping]) -> list[str]:
"""Compile each metric's match to the filter conditions that drop the series it doesn't keep.

One condition per entry, since the filter drops a datapoint any condition
holds for. A filter rather than a transform statement, because OTTL in the
transform processor can't drop a datapoint. It runs ahead of the
transform, so it selects on the name the component emitted, before a part
is extracted or the metric renamed.
"""
out: list[str] = []
for mapping in mappings:
for m in mapping.spec.metrics:
for entry in m.match or []:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "a mapping can only narrow" the semantics we want? Each entry is its own drop condition, so matches on one from across all mappings intersect. A user mapping of sglang:num_running_reqs with priority: value "1" meets the built-in's priority absent and doesn't keep any series, and nothing reports it. design/telemetry.md names "nothing reports that a mapping matched no series" as a failure the kind exists to avoid.

Could a user's match for a from replace the built-in's rather than add to it? Or could the XRD at least say that matches on one metric combine and can only narrow?

attr = f'datapoint.attributes["{entry.label}"]'
if entry.value is not None:
out.append(f'metric.name == "{m.from_}" and {attr} != {_quote(entry.value)}')
else:
# absent, which the schema requires without a value. The
# Prometheus receiver drops an empty label, but a source
# that keeps one means the same by it.
out.append(f'metric.name == "{m.from_}" and {attr} != nil and {attr} != ""')
return out


def _quote(value: str) -> str:
"""One OTTL string literal, with anything that would end it escaped.

Expand All @@ -494,9 +518,11 @@ def _quote(value: str) -> str:
remap - they have to be, because they carry whatever vocabulary the
component already emits. An unescaped quote in one of them would close the
literal early and leave the rest of it as OTTL, which at best stops the
collector from starting and at worst runs.
collector from starting and at worst runs. A dollar is doubled too: the
collector expands ${...} anywhere in its config before OTTL sees it, then
turns $$ back into $.
"""
escaped = value.replace("\\", "\\\\").replace('"', '\\"')
escaped = value.replace("\\", "\\\\").replace('"', '\\"').replace("$", "$$")
return f'"{escaped}"'


Expand Down Expand Up @@ -666,6 +692,7 @@ def config(
{"context": "resource", "statements": [f"keep_keys(resource.attributes, [{_IDENTITY_OTTL}])"]}
]
},
"filter/match": {"metrics": {"datapoint": conditions(mappings)}},
"transform/modelplane": _transform(mappings),
# Discovery writes the identity onto each datapoint; this lifts it onto
# the resource, which is where an exporter that flattens a series into
Expand All @@ -687,6 +714,7 @@ def config(
"memory_limiter",
"resource/cluster",
"transform/identity",
"filter/match",
"transform/modelplane",
"groupbyattrs/identity",
"filter/modelplane",
Expand Down
15 changes: 14 additions & 1 deletion functions/compose-serving-stack/function/stacks/metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,16 @@
"sglang:http_requests_active": "modelplane_engine_requests_active",
}

# Under --enable-priority-scheduling SGLang publishes its running and waiting
# counts once as a total, with an empty priority, and again per priority, so
# only the total is kept. Not tp_rank 0 for --enable-metrics-for-all-schedulers:
# under DP attention each group's first rank publishes its own count, and only
# one of them is tp_rank 0.
_SGLANG_MATCH: dict[str, list[dict[str, object]]] = {
"sglang:num_running_reqs": [{"label": "priority", "absent": True}],
"sglang:num_queue_reqs": [{"label": "priority", "absent": True}],
}

# Each engine's prompt and generation token histograms, taken as counters: the
# prompt and generated tokens of the requests it finished, and from the prompt
# histogram's count, the requests. A prompt's tokens include any the prefix
Expand Down Expand Up @@ -237,13 +247,16 @@ def _mapping(
tokens: tuple[str, str] | None = None,
extra: tuple[dict[str, object], ...] = (),
types: dict[str, str] | None = None,
match: dict[str, list[dict[str, object]]] | None = None,
) -> v1alpha1.MetricMapping:
units = units or {}
types = types or {}
match = match or {}
metrics: list[dict[str, object]] = [
{"from": src, "to": dst}
| ({"fromUnit": units[src]} if src in units else {})
| ({"fromType": types[src]} if src in types else {})
| ({"match": match[src]} if src in match else {})
for src, dst in pairs.items()
]
if tokens:
Expand All @@ -266,7 +279,7 @@ def mappings() -> list[v1alpha1.MetricMapping]:
_mapping("modelplane-gateway", _GATEWAY),
_mapping("modelplane-gateway-proxy", _GATEWAY_PROXY),
_mapping("modelplane-vllm", _VLLM, tokens=_VLLM_TOKENS, extra=(_VLLM_RESPONSES,)),
_mapping("modelplane-sglang", _SGLANG, tokens=_SGLANG_TOKENS, extra=(_SGLANG_RESPONSES,)),
_mapping("modelplane-sglang", _SGLANG, tokens=_SGLANG_TOKENS, extra=(_SGLANG_RESPONSES,), match=_SGLANG_MATCH),
_mapping("modelplane-picker", _PICKER),
_mapping("modelplane-gpu", _GPU, _GPU_UNITS),
_mapping("modelplane-gpu-amd", _AMD_GPU, _AMD_GPU_UNITS, types=_AMD_GPU_TYPES),
Expand Down
Loading
Loading