Repository navigation
Let a MetricMapping keep only the series of a metric that match #671
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -39,8 +39,10 @@ spec: | |
| renames for is additive: its renames run after the built-in | ||
| ones, on whatever those left behind. A metric a built-in | ||
| already renamed no longer answers to the name it was emitted | ||
| under, so a second mapping selecting on that name matches | ||
| nothing and the built-in stands. Select on the `modelplane_` | ||
| under, so a second mapping's rename of that name does nothing | ||
| and the built-in stands. Its `match` still applies, because | ||
| matching runs before any rename and drops the series it doesn't | ||
| keep wherever the metric appears. Select on the `modelplane_` | ||
| name instead to rename one of Modelplane's own. | ||
| properties: | ||
| metrics: | ||
|
|
@@ -151,6 +153,45 @@ spec: | |
| putting an engine's own vocabulary into | ||
| Modelplane's. A value with no entry here is left | ||
| as the component wrote it. | ||
| match: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is Could a spec-level list of series to keep, keyed by metric name, describe what it does more directly? The cost is a second place to name a metric. |
||
| type: array | ||
| maxItems: 16 | ||
| description: >- | ||
| Keep only this metric's series whose labels match every | ||
| entry, and drop the rest before the rename. For a | ||
| component that publishes the same count more than once, | ||
| such as a total beside a breakdown of it, which would | ||
| otherwise be summed twice. | ||
|
|
||
| It applies to the metric wherever it appears, so a | ||
| second mapping of the same `from`, such as one taking | ||
| a part of the same histogram, sees only the series it | ||
| keeps. | ||
| items: | ||
| type: object | ||
| required: [label] | ||
| x-kubernetes-validations: | ||
| - rule: "has(self.value) != has(self.absent)" | ||
| message: set either value, to keep series carrying it, or absent, to keep series without the label. | ||
| properties: | ||
| label: | ||
| type: string | ||
| maxLength: 63 | ||
| pattern: '^[a-zA-Z_][a-zA-Z0-9_]*$' | ||
| description: The label to match on. | ||
| value: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Won't Could we add |
||
| type: string | ||
| maxLength: 253 | ||
| description: >- | ||
| Keep series where the label has this value. A | ||
| series without the label doesn't match. | ||
| absent: | ||
| type: boolean | ||
| enum: [true] | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would an match:
- label: priority
operator: DoesNotExistThe downside is one more field on the common case. |
||
| description: >- | ||
| Keep series without the label. A label with an | ||
| empty value counts as absent, as it does in | ||
| Prometheus. | ||
| fromUnit: | ||
| type: string | ||
| enum: [Millijoules, Microjoules, Mebibytes, Milliseconds, Nanoseconds, Percent] | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -387,12 +387,32 @@ spec: | |
| to: modelplane_kv_offload_load_seconds_total | ||
| ``` | ||
|
|
||
| SGLang publishes `modelplane_requests_running` and `modelplane_requests_waiting` more | ||
| than once per engine under `--enable-priority-scheduling`, once in total and once per | ||
| priority, and once per tensor-parallel rank under | ||
| `--enable-metrics-for-all-schedulers`. With either flag, a sum over them counts each | ||
| request more than once. Keeping only one of those series is tracked in | ||
| [#549](https://github.com/modelplaneai/modelplane/issues/549). | ||
| With `--enable-priority-scheduling`, SGLang publishes its running and waiting counts | ||
| once in total and once per priority. The built-in mapping keeps only the total. With | ||
| `--enable-metrics-for-all-schedulers`, it publishes them once per tensor-parallel rank, | ||
| and a sum over `modelplane_requests_running` or `modelplane_requests_waiting` counts each | ||
| request once per rank. To keep one rank's series, give a mapping of | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should #549 get a follow-up rather than close with this PR? This |
||
| `sglang:num_running_reqs` and `sglang:num_queue_reqs` a `match` on `tp_rank`: | ||
|
|
||
| ```yaml | ||
| - from: sglang:num_running_reqs | ||
| to: modelplane_requests_running | ||
| match: | ||
| - label: tp_rank | ||
| value: "0" | ||
| - from: sglang:num_queue_reqs | ||
| to: modelplane_requests_waiting | ||
| match: | ||
| - label: tp_rank | ||
| value: "0" | ||
| ``` | ||
|
|
||
| The built-in mapping has already renamed these metrics, so this mapping's rename does | ||
| nothing, but its `match` still applies: matching runs before any rename, on the name | ||
| SGLang emitted, wherever the metric appears. | ||
|
|
||
| With DP attention, each group's first rank publishes its own count, so keeping | ||
| `tp_rank: "0"` drops every other group's. | ||
|
|
||
| Because the gateway, not the engine, measures the `modelplane_frontend_*` series, they | ||
| work without configuration for an engine Modelplane has never seen. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -486,6 +486,30 @@ def statements(mappings: list[mmv1alpha1.MetricMapping]) -> tuple[list[str], lis | |
| return extract, scale, datapoint, metric | ||
|
|
||
|
|
||
| def conditions(mappings: list[mmv1alpha1.MetricMapping]) -> list[str]: | ||
| """Compile each metric's match to the filter conditions that drop the series it doesn't keep. | ||
|
|
||
| One condition per entry, since the filter drops a datapoint any condition | ||
| holds for. A filter rather than a transform statement, because OTTL in the | ||
| transform processor can't drop a datapoint. It runs ahead of the | ||
| transform, so it selects on the name the component emitted, before a part | ||
| is extracted or the metric renamed. | ||
| """ | ||
| out: list[str] = [] | ||
| for mapping in mappings: | ||
| for m in mapping.spec.metrics: | ||
| for entry in m.match or []: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is "a mapping can only narrow" the semantics we want? Each entry is its own drop condition, so matches on one Could a user's match for a |
||
| attr = f'datapoint.attributes["{entry.label}"]' | ||
| if entry.value is not None: | ||
| out.append(f'metric.name == "{m.from_}" and {attr} != {_quote(entry.value)}') | ||
| else: | ||
| # absent, which the schema requires without a value. The | ||
| # Prometheus receiver drops an empty label, but a source | ||
| # that keeps one means the same by it. | ||
| out.append(f'metric.name == "{m.from_}" and {attr} != nil and {attr} != ""') | ||
| return out | ||
|
|
||
|
|
||
| def _quote(value: str) -> str: | ||
| """One OTTL string literal, with anything that would end it escaped. | ||
|
|
||
|
|
@@ -494,9 +518,11 @@ def _quote(value: str) -> str: | |
| remap - they have to be, because they carry whatever vocabulary the | ||
| component already emits. An unescaped quote in one of them would close the | ||
| literal early and leave the rest of it as OTTL, which at best stops the | ||
| collector from starting and at worst runs. | ||
| collector from starting and at worst runs. A dollar is doubled too: the | ||
| collector expands ${...} anywhere in its config before OTTL sees it, then | ||
| turns $$ back into $. | ||
| """ | ||
| escaped = value.replace("\\", "\\\\").replace('"', '\\"') | ||
| escaped = value.replace("\\", "\\\\").replace('"', '\\"').replace("$", "$$") | ||
| return f'"{escaped}"' | ||
|
|
||
|
|
||
|
|
@@ -666,6 +692,7 @@ def config( | |
| {"context": "resource", "statements": [f"keep_keys(resource.attributes, [{_IDENTITY_OTTL}])"]} | ||
| ] | ||
| }, | ||
| "filter/match": {"metrics": {"datapoint": conditions(mappings)}}, | ||
| "transform/modelplane": _transform(mappings), | ||
| # Discovery writes the identity onto each datapoint; this lifts it onto | ||
| # the resource, which is where an exporter that flattens a series into | ||
|
|
@@ -687,6 +714,7 @@ def config( | |
| "memory_limiter", | ||
| "resource/cluster", | ||
| "transform/identity", | ||
| "filter/match", | ||
| "transform/modelplane", | ||
| "groupbyattrs/identity", | ||
| "filter/modelplane", | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Won't a
matchon amodelplane_fromsilently do nothing? This sentence points users at themodelplane_name, butfilter/matchruns beforetransform/modelplane, while the series still has the engine's name. Sometric.name == "modelplane_requests_running"never holds and the filter keeps every series.Could a CEL rule reject
matchon amodelplane_from, or could thematchdescription say it only applies to the emitted name?