Skip to content

Let a MetricMapping keep only the series of a metric that match - #671

Open
chethanuk wants to merge 1 commit into
modelplaneai:mainfrom
chethanuk:oss/549-metricmapping-match
Open

chethanuk wants to merge 1 commit into
modelplaneai:mainfrom
chethanuk:oss/549-metricmapping-match

Conversation

@chethanuk

Copy link
Copy Markdown

Description of your changes

Fixes #549.

Under --enable-priority-scheduling SGLang publishes its running and waiting counts once as a total, with an empty priority, and again per priority. A MetricMapping renames every series of a metric, so modelplane_requests_running and modelplane_requests_waiting counted each request twice.

This adds match to a MetricMapping metric. Each entry names a label and either a value it must hold or absent: true, and only series matching every entry are kept. The entries compile to a filter/match processor that runs before transform/modelplane, because the transform processor can't drop a datapoint. Since it runs first, a match selects on the name the component emitted, before a part is extracted or the metric renamed. An empty label counts as absent, as in Prometheus. Match and label values are quoted with every $ doubled, because the collector expands ${...} anywhere in its config before OTTL sees it.

The built-in SGLang mapping now keeps only the series without a priority. The issue also suggested matching tp_rank: "0" for --enable-metrics-for-all-schedulers. I left that out: SGLang labels by global tp_rank, and under DP attention each group's first rank publishes its own count, so tp_rank 0 alone would undercount. The telemetry docs show how to match tp_rank where DP attention isn't in use.

I ran otelcol-contrib 0.161.0 against a static SGLang-style /metrics with two ranks, each publishing a total of 5 and priorities of 2 and 3. Before the change modelplane_requests_running kept the totals and the per-priority series. With the rendered config only the two totals (5 and 5) remain. otelcol validate passes on the rendered config with and without mappings.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes. No nix here; ran the function tests, ruff and the docs validator directly.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

@github-actions

Copy link
Copy Markdown

Docs preview: https://modelplane-docs-pr-671.vercel.app (ready once the site's Content workflow finishes)

A MetricMapping renames every series of a metric, and the collector
sums them. Under --enable-priority-scheduling SGLang publishes its
running and waiting counts once as a total, with an empty priority,
and again per priority, so modelplane_requests_running and
modelplane_requests_waiting counted each request twice. Nothing in a
mapping could pick one of the two.

A metric in a MetricMapping now takes a match: label entries, each
either a value the label must hold or absent, and only the series
matching every entry are kept. They compile to a filter processor
ahead of transform/modelplane, because OTTL in the transform processor
can't drop a datapoint. Running first means a match selects on the
name the component emitted, before a part is extracted or the metric
renamed, and it applies to the metric wherever it appears. So a second
mapping of a name a built-in already renamed still filters it, even
though its rename does nothing. An empty label counts as absent, as it
does in Prometheus, whose receiver drops it anyway. Dollar signs in
quoted OTTL strings are now doubled, since the collector expands ${...}
in its config before OTTL parses it.

The built-in SGLang mapping keeps only the series without a priority.
It doesn't match tp_rank 0 for --enable-metrics-for-all-schedulers:
SGLang labels by global tp_rank, and under DP attention each group's
first rank publishes its own count, so tp_rank 0 alone would undercount.
The docs show a user how to match tp_rank on both counts where DP
attention isn't in use.

Fixes modelplaneai#549.

Signed-off-by: ChethanUK <chethanuk@outlook.com>
@chethanuk
chethanuk force-pushed the oss/549-metricmapping-match branch from 55cc494 to 8b96a8c Compare November 17, 2026 04:36
@chethanuk
chethanuk marked this pull request as ready for review November 17, 2026 04:58
@negz

negz commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

/review

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the API for match needs another pass before this is ready, though compiling it to a filter that runs ahead of the transform looks right.

The open questions are about where match sits and how its entries combine. It's a field on one rename entry, but it drops series for every mapping of that from. Each entry is either a boolean or one value, with no room for other operators. Answers to those could change the schema. No maintainer has signed off on the plan in #549 yet. Settling the schema first saves polishing code that may change.

The PR says it fixes #549, but its tp_rank recipe undercounts under DP attention, so that half of the issue isn't fixed.

Reviewed by modelplane-review 22e194c. Reply, or react 👍 or 👎, on each comment to help tune it.

putting an engine's own vocabulary into
Modelplane's. A value with no entry here is left
as the component wrote it.
match:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is match in the right place? As a field on one from→to entry it reads as "which series this rename takes", but conditions() compiles it to a drop keyed only on from. So it also filters sibling entries, other mappings and the built-in. Both this description and the spec-level metrics one have to explain that, and the tp_rank recipe in the docs has the user write a rename that "does nothing" just to hold a match.

Could a spec-level list of series to keep, keyed by metric name, describe what it does more directly? The cost is a second place to name a metric.

out: list[str] = []
for mapping in mappings:
for m in mapping.spec.metrics:
for entry in m.match or []:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "a mapping can only narrow" the semantics we want? Each entry is its own drop condition, so matches on one from across all mappings intersect. A user mapping of sglang:num_running_reqs with priority: value "1" meets the built-in's priority absent and doesn't keep any series, and nothing reports it. design/telemetry.md names "nothing reports that a mapping matched no series" as a failure the kind exists to avoid.

Could a user's match for a from replace the built-in's rather than add to it? Or could the XRD at least say that matches on one metric combine and can only narrow?

series without the label doesn't match.
absent:
type: boolean
enum: [true]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would an operator discriminator serve better than value versus absent: true? A boolean with enum: [true] can't grow additively into "not this value", "one of these" or "present". Each would need a new sibling field and a wider CEL rule. The ModelDeployment tolerations already use operator: [Exists, Equal] (apis/modeldeployments/definition.yaml:328), and users know the same structure from matchExpressions:

match:
- label: priority
  operator: DoesNotExist

The downside is one more field on the common case.

maxLength: 63
pattern: '^[a-zA-Z_][a-zA-Z0-9_]*$'
description: The label to match on.
value:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't value: "" drop every series of the metric? The Prometheus receiver drops an empty label, and the attribute arrives as nil. In OTTL nil != "" is true. The condition then drops series without the label as well as those with any other value. A run against otelcol-contrib 0.161.0 confirmed that my_engine_x{kind=""} and {kind="a"} both disappear. It's an easy thing to write, since PromQL selects a missing label with ="".

Could we add minLength: 1 here, or compile value: "" the same way as absent?

under, so a second mapping's rename of that name does nothing
and the built-in stands. Its `match` still applies, because
matching runs before any rename and drops the series it doesn't
keep wherever the metric appears. Select on the `modelplane_`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't a match on a modelplane_ from silently do nothing? This sentence points users at the modelplane_ name, but filter/match runs before transform/modelplane, while the series still has the engine's name. So metric.name == "modelplane_requests_running" never holds and the filter keeps every series.

Could a CEL rule reject match on a modelplane_ from, or could the match description say it only applies to the emitted name?

once in total and once per priority. The built-in mapping keeps only the total. With
`--enable-metrics-for-all-schedulers`, it publishes them once per tensor-parallel rank,
and a sum over `modelplane_requests_running` or `modelplane_requests_waiting` counts each
request once per rank. To keep one rank's series, give a mapping of

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should #549 get a follow-up rather than close with this PR? This tp_rank: "0" recipe for --enable-metrics-for-all-schedulers undercounts with DP attention, and the caveat only comes after the example. If the right series under DP attention is "each group's first rank", can match express that at all?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A MetricMapping can't keep one of a metric's series, so SGLang's running and waiting counts double

2 participants