From 8b96a8c816e89d1e4d8dce86b6066ae3b4cb8e0c Mon Sep 17 00:00:00 2001 From: ChethanUK Date: Tue, 17 Nov 2026 12:34:04 +0800 Subject: [PATCH] Let a MetricMapping keep only the series of a metric that match A MetricMapping renames every series of a metric, and the collector sums them. Under --enable-priority-scheduling SGLang publishes its running and waiting counts once as a total, with an empty priority, and again per priority, so modelplane_requests_running and modelplane_requests_waiting counted each request twice. Nothing in a mapping could pick one of the two. A metric in a MetricMapping now takes a match: label entries, each either a value the label must hold or absent, and only the series matching every entry are kept. They compile to a filter processor ahead of transform/modelplane, because OTTL in the transform processor can't drop a datapoint. Running first means a match selects on the name the component emitted, before a part is extracted or the metric renamed, and it applies to the metric wherever it appears. So a second mapping of a name a built-in already renamed still filters it, even though its rename does nothing. An empty label counts as absent, as it does in Prometheus, whose receiver drops it anyway. Dollar signs in quoted OTTL strings are now doubled, since the collector expands ${...} in its config before OTTL parses it. The built-in SGLang mapping keeps only the series without a priority. It doesn't match tp_rank 0 for --enable-metrics-for-all-schedulers: SGLang labels by global tp_rank, and under DP attention each group's first rank publishes its own count, so tp_rank 0 alone would undercount. The docs show a user how to match tp_rank on both counts where DP attention isn't in use. Fixes #549. Signed-off-by: ChethanUK --- apis/metricmappings/definition.yaml | 45 +++- docs/content/platform/telemetry.md | 32 ++- .../function/collector.py | 32 ++- .../function/stacks/metrics.py | 15 +- .../tests/test_collector.py | 224 +++++++++++++++++- .../compose-serving-stack/tests/test_fn.py | 21 +- schemas/.lock.json | 2 +- .../ai/modelplane/metricmapping/v1alpha1.py | 22 +- 8 files changed, 372 insertions(+), 21 deletions(-) diff --git a/apis/metricmappings/definition.yaml b/apis/metricmappings/definition.yaml index 5635a6805..acaf0c502 100644 --- a/apis/metricmappings/definition.yaml +++ b/apis/metricmappings/definition.yaml @@ -39,8 +39,10 @@ spec: renames for is additive: its renames run after the built-in ones, on whatever those left behind. A metric a built-in already renamed no longer answers to the name it was emitted - under, so a second mapping selecting on that name matches - nothing and the built-in stands. Select on the `modelplane_` + under, so a second mapping's rename of that name does nothing + and the built-in stands. Its `match` still applies, because + matching runs before any rename and drops the series it doesn't + keep wherever the metric appears. Select on the `modelplane_` name instead to rename one of Modelplane's own. properties: metrics: @@ -151,6 +153,45 @@ spec: putting an engine's own vocabulary into Modelplane's. A value with no entry here is left as the component wrote it. + match: + type: array + maxItems: 16 + description: >- + Keep only this metric's series whose labels match every + entry, and drop the rest before the rename. For a + component that publishes the same count more than once, + such as a total beside a breakdown of it, which would + otherwise be summed twice. + + It applies to the metric wherever it appears, so a + second mapping of the same `from`, such as one taking + a part of the same histogram, sees only the series it + keeps. + items: + type: object + required: [label] + x-kubernetes-validations: + - rule: "has(self.value) != has(self.absent)" + message: set either value, to keep series carrying it, or absent, to keep series without the label. + properties: + label: + type: string + maxLength: 63 + pattern: '^[a-zA-Z_][a-zA-Z0-9_]*$' + description: The label to match on. + value: + type: string + maxLength: 253 + description: >- + Keep series where the label has this value. A + series without the label doesn't match. + absent: + type: boolean + enum: [true] + description: >- + Keep series without the label. A label with an + empty value counts as absent, as it does in + Prometheus. fromUnit: type: string enum: [Millijoules, Microjoules, Mebibytes, Milliseconds, Nanoseconds, Percent] diff --git a/docs/content/platform/telemetry.md b/docs/content/platform/telemetry.md index a92d2ec34..ac4c0ea14 100644 --- a/docs/content/platform/telemetry.md +++ b/docs/content/platform/telemetry.md @@ -387,12 +387,32 @@ spec: to: modelplane_kv_offload_load_seconds_total ``` -SGLang publishes `modelplane_requests_running` and `modelplane_requests_waiting` more -than once per engine under `--enable-priority-scheduling`, once in total and once per -priority, and once per tensor-parallel rank under -`--enable-metrics-for-all-schedulers`. With either flag, a sum over them counts each -request more than once. Keeping only one of those series is tracked in -[#549](https://github.com/modelplaneai/modelplane/issues/549). +With `--enable-priority-scheduling`, SGLang publishes its running and waiting counts +once in total and once per priority. The built-in mapping keeps only the total. With +`--enable-metrics-for-all-schedulers`, it publishes them once per tensor-parallel rank, +and a sum over `modelplane_requests_running` or `modelplane_requests_waiting` counts each +request once per rank. To keep one rank's series, give a mapping of +`sglang:num_running_reqs` and `sglang:num_queue_reqs` a `match` on `tp_rank`: + +```yaml + - from: sglang:num_running_reqs + to: modelplane_requests_running + match: + - label: tp_rank + value: "0" + - from: sglang:num_queue_reqs + to: modelplane_requests_waiting + match: + - label: tp_rank + value: "0" +``` + +The built-in mapping has already renamed these metrics, so this mapping's rename does +nothing, but its `match` still applies: matching runs before any rename, on the name +SGLang emitted, wherever the metric appears. + +With DP attention, each group's first rank publishes its own count, so keeping +`tp_rank: "0"` drops every other group's. Because the gateway, not the engine, measures the `modelplane_frontend_*` series, they work without configuration for an engine Modelplane has never seen. diff --git a/functions/compose-serving-stack/function/collector.py b/functions/compose-serving-stack/function/collector.py index ab55e97e1..d9eacb1c4 100644 --- a/functions/compose-serving-stack/function/collector.py +++ b/functions/compose-serving-stack/function/collector.py @@ -486,6 +486,30 @@ def statements(mappings: list[mmv1alpha1.MetricMapping]) -> tuple[list[str], lis return extract, scale, datapoint, metric +def conditions(mappings: list[mmv1alpha1.MetricMapping]) -> list[str]: + """Compile each metric's match to the filter conditions that drop the series it doesn't keep. + + One condition per entry, since the filter drops a datapoint any condition + holds for. A filter rather than a transform statement, because OTTL in the + transform processor can't drop a datapoint. It runs ahead of the + transform, so it selects on the name the component emitted, before a part + is extracted or the metric renamed. + """ + out: list[str] = [] + for mapping in mappings: + for m in mapping.spec.metrics: + for entry in m.match or []: + attr = f'datapoint.attributes["{entry.label}"]' + if entry.value is not None: + out.append(f'metric.name == "{m.from_}" and {attr} != {_quote(entry.value)}') + else: + # absent, which the schema requires without a value. The + # Prometheus receiver drops an empty label, but a source + # that keeps one means the same by it. + out.append(f'metric.name == "{m.from_}" and {attr} != nil and {attr} != ""') + return out + + def _quote(value: str) -> str: """One OTTL string literal, with anything that would end it escaped. @@ -494,9 +518,11 @@ def _quote(value: str) -> str: remap - they have to be, because they carry whatever vocabulary the component already emits. An unescaped quote in one of them would close the literal early and leave the rest of it as OTTL, which at best stops the - collector from starting and at worst runs. + collector from starting and at worst runs. A dollar is doubled too: the + collector expands ${...} anywhere in its config before OTTL sees it, then + turns $$ back into $. """ - escaped = value.replace("\\", "\\\\").replace('"', '\\"') + escaped = value.replace("\\", "\\\\").replace('"', '\\"').replace("$", "$$") return f'"{escaped}"' @@ -666,6 +692,7 @@ def config( {"context": "resource", "statements": [f"keep_keys(resource.attributes, [{_IDENTITY_OTTL}])"]} ] }, + "filter/match": {"metrics": {"datapoint": conditions(mappings)}}, "transform/modelplane": _transform(mappings), # Discovery writes the identity onto each datapoint; this lifts it onto # the resource, which is where an exporter that flattens a series into @@ -687,6 +714,7 @@ def config( "memory_limiter", "resource/cluster", "transform/identity", + "filter/match", "transform/modelplane", "groupbyattrs/identity", "filter/modelplane", diff --git a/functions/compose-serving-stack/function/stacks/metrics.py b/functions/compose-serving-stack/function/stacks/metrics.py index ce2bfccdd..4efa82028 100644 --- a/functions/compose-serving-stack/function/stacks/metrics.py +++ b/functions/compose-serving-stack/function/stacks/metrics.py @@ -95,6 +95,16 @@ "sglang:http_requests_active": "modelplane_engine_requests_active", } +# Under --enable-priority-scheduling SGLang publishes its running and waiting +# counts once as a total, with an empty priority, and again per priority, so +# only the total is kept. Not tp_rank 0 for --enable-metrics-for-all-schedulers: +# under DP attention each group's first rank publishes its own count, and only +# one of them is tp_rank 0. +_SGLANG_MATCH: dict[str, list[dict[str, object]]] = { + "sglang:num_running_reqs": [{"label": "priority", "absent": True}], + "sglang:num_queue_reqs": [{"label": "priority", "absent": True}], +} + # Each engine's prompt and generation token histograms, taken as counters: the # prompt and generated tokens of the requests it finished, and from the prompt # histogram's count, the requests. A prompt's tokens include any the prefix @@ -237,13 +247,16 @@ def _mapping( tokens: tuple[str, str] | None = None, extra: tuple[dict[str, object], ...] = (), types: dict[str, str] | None = None, + match: dict[str, list[dict[str, object]]] | None = None, ) -> v1alpha1.MetricMapping: units = units or {} types = types or {} + match = match or {} metrics: list[dict[str, object]] = [ {"from": src, "to": dst} | ({"fromUnit": units[src]} if src in units else {}) | ({"fromType": types[src]} if src in types else {}) + | ({"match": match[src]} if src in match else {}) for src, dst in pairs.items() ] if tokens: @@ -266,7 +279,7 @@ def mappings() -> list[v1alpha1.MetricMapping]: _mapping("modelplane-gateway", _GATEWAY), _mapping("modelplane-gateway-proxy", _GATEWAY_PROXY), _mapping("modelplane-vllm", _VLLM, tokens=_VLLM_TOKENS, extra=(_VLLM_RESPONSES,)), - _mapping("modelplane-sglang", _SGLANG, tokens=_SGLANG_TOKENS, extra=(_SGLANG_RESPONSES,)), + _mapping("modelplane-sglang", _SGLANG, tokens=_SGLANG_TOKENS, extra=(_SGLANG_RESPONSES,), match=_SGLANG_MATCH), _mapping("modelplane-picker", _PICKER), _mapping("modelplane-gpu", _GPU, _GPU_UNITS), _mapping("modelplane-gpu-amd", _AMD_GPU, _AMD_GPU_UNITS, types=_AMD_GPU_TYPES), diff --git a/functions/compose-serving-stack/tests/test_collector.py b/functions/compose-serving-stack/tests/test_collector.py index 3583f936d..2028e2428 100644 --- a/functions/compose-serving-stack/tests/test_collector.py +++ b/functions/compose-serving-stack/tests/test_collector.py @@ -88,6 +88,16 @@ class TransformCase: want: dict +@dataclasses.dataclass +class ConditionsCase: + """A test case for collector.conditions.""" + + name: str + reason: str + mappings: list[mmv1alpha1.MetricMapping] + want: list[str] + + @dataclasses.dataclass class ObjectsCase: """A test case for collector.objects.""" @@ -153,6 +163,7 @@ def _config( "memory_limiter", "resource/cluster", "transform/identity", + "filter/match", "transform/modelplane", "groupbyattrs/identity", "filter/modelplane", @@ -436,6 +447,16 @@ def _config( } ] }, + # SGLang publishes its running and waiting counts as a total and again per + # priority, so the built-in keeps only the total. + "filter/match": { + "metrics": { + "datapoint": [ + 'metric.name == "sglang:num_running_reqs" and datapoint.attributes["priority"] != nil and datapoint.attributes["priority"] != ""', + 'metric.name == "sglang:num_queue_reqs" and datapoint.attributes["priority"] != nil and datapoint.attributes["priority"] != ""', + ] + } + }, "transform/modelplane": { "metric_statements": [ # The engines' token histograms as counters, extracted ahead of the @@ -1111,6 +1132,33 @@ def test_in_flight(case: InFlightCase) -> None: ['set(metric.name, "modelplane_requests_total") where metric.name == "my_engine_finish"'], ), ), + # A dollar would be substituted by the collector's config expansion, for the + # reason DollarValue in CONDITIONS_CASES gives. + StatementsCase( + name="DollarLabelValue", + reason="A dollar in a label value is escaped, so the collector's config expansion leaves it literal.", + mappings=[ + mmv1alpha1.MetricMapping.model_validate( + { + "spec": { + "metrics": [ + { + "from": "my_engine_running", + "to": "modelplane_requests_running", + "labels": [{"name": "status", "value": "a$b"}], + } + ] + } + } + ) + ], + want=( + [], + [], + ['set(datapoint.attributes["status"], "a$$b") where metric.name == "my_engine_running"'], + ['set(metric.name, "modelplane_requests_running") where metric.name == "my_engine_running"'], + ), + ), # A label carried onto its own name is how a mapping remaps values in place. # The delete that stops a carried label costing twice the cardinality would # take the label the statements before it just set. @@ -1212,6 +1260,176 @@ def test_transform(case: TransformCase) -> None: assert got == case.want, case.reason +CONDITIONS_CASES = [ + ConditionsCase( + name="NoMatch", + reason="A metric with no match keeps every datapoint, so it contributes no condition.", + mappings=[ + mmv1alpha1.MetricMapping.model_validate( + {"spec": {"metrics": [{"from": "my_engine_running", "to": "modelplane_requests_running"}]}} + ) + ], + want=[], + ), + # Prometheus treats an empty label as no label, and SGLang publishes its + # total with priority="", so an empty value has to count as absent. + ConditionsCase( + name="Absent", + reason="A label that must be absent drops every datapoint carrying it with a non-empty value.", + mappings=[ + mmv1alpha1.MetricMapping.model_validate( + { + "spec": { + "metrics": [ + { + "from": "sglang:num_running_reqs", + "to": "modelplane_requests_running", + "match": [{"label": "priority", "absent": True}], + } + ] + } + } + ) + ], + want=[ + 'metric.name == "sglang:num_running_reqs" and datapoint.attributes["priority"] != nil' + ' and datapoint.attributes["priority"] != ""' + ], + ), + ConditionsCase( + name="Value", + reason="A label that must hold a value drops every datapoint where it holds another.", + mappings=[ + mmv1alpha1.MetricMapping.model_validate( + { + "spec": { + "metrics": [ + { + "from": "my_engine_running", + "to": "modelplane_requests_running", + "match": [{"label": "tp_rank", "value": "0"}], + } + ] + } + } + ) + ], + want=['metric.name == "my_engine_running" and datapoint.attributes["tp_rank"] != "0"'], + ), + ConditionsCase( + name="Both", + reason="Each match entry is its own condition, in the order the metric lists them.", + mappings=[ + mmv1alpha1.MetricMapping.model_validate( + { + "spec": { + "metrics": [ + { + "from": "my_engine_running", + "to": "modelplane_requests_running", + "match": [{"label": "priority", "absent": True}, {"label": "tp_rank", "value": "0"}], + } + ] + } + } + ) + ], + want=[ + 'metric.name == "my_engine_running" and datapoint.attributes["priority"] != nil' + ' and datapoint.attributes["priority"] != ""', + 'metric.name == "my_engine_running" and datapoint.attributes["tp_rank"] != "0"', + ], + ), + # The filter runs before extraction mints my_engine_duration_ms_count, so a + # condition on that name would match nothing, silently. + ConditionsCase( + name="PartUsesSourceName", + reason="A metric extracting a histogram part is matched on its source name, not the extracted part's.", + mappings=[ + mmv1alpha1.MetricMapping.model_validate( + { + "spec": { + "metrics": [ + { + "from": "my_engine_duration_ms", + "to": "modelplane_requests_running", + "part": "Count", + "match": [{"label": "tp_rank", "value": "0"}], + } + ] + } + } + ) + ], + want=['metric.name == "my_engine_duration_ms" and datapoint.attributes["tp_rank"] != "0"'], + ), + # A match value is free text, like a label's, for the reason + # QuotedLabelValue in STATEMENTS_CASES gives. + ConditionsCase( + name="QuotedValue", + reason="A quote in a match value is escaped, so it can't end the OTTL string it sits in.", + mappings=[ + mmv1alpha1.MetricMapping.model_validate( + { + "spec": { + "metrics": [ + { + "from": "my_engine_running", + "to": "modelplane_requests_running", + "match": [{"label": "tp_rank", "value": 'a"b'}], + } + ] + } + } + ) + ], + want=[r'metric.name == "my_engine_running" and datapoint.attributes["tp_rank"] != "a\"b"'], + ), + # The collector expands ${scheme:...} in any config string and then turns + # $$ into $, so an unescaped dollar is substituted, not compared literally. + ConditionsCase( + name="DollarValue", + reason="A dollar in a match value is escaped, so the collector's config expansion leaves it literal.", + mappings=[ + mmv1alpha1.MetricMapping.model_validate( + { + "spec": { + "metrics": [ + { + "from": "my_engine_running", + "to": "modelplane_requests_running", + "match": [{"label": "tp_rank", "value": "${env:HOME}"}], + } + ] + } + } + ) + ], + want=['metric.name == "my_engine_running" and datapoint.attributes["tp_rank"] != "$${env:HOME}"'], + ), + # Under priority scheduling SGLang publishes a series per priority beside + # the total, and summing them all counts every request twice. + ConditionsCase( + name="BuiltIn", + reason="The built-in SGLang mapping keeps only the total of its running and queued requests.", + mappings=list(stacks.BUILTIN_MAPPINGS), + want=[ + 'metric.name == "sglang:num_running_reqs" and datapoint.attributes["priority"] != nil' + ' and datapoint.attributes["priority"] != ""', + 'metric.name == "sglang:num_queue_reqs" and datapoint.attributes["priority"] != nil' + ' and datapoint.attributes["priority"] != ""', + ], + ), +] + + +@pytest.mark.parametrize("case", CONDITIONS_CASES, ids=lambda case: case.name) +def test_conditions(case: ConditionsCase) -> None: + """conditions compiles the mappings' match entries to the filter conditions that drop a datapoint.""" + got = collector.conditions(case.mappings) + assert got == case.want, case.reason + + # Every case renders the built-in mappings, read from the stacks package, for # the reason CONFIG_CASES gives. The config hashes are of a config that carries # them, so changing a built-in changes every hash here too. @@ -1245,7 +1463,7 @@ def test_transform(case: TransformCase) -> None: ( "collector", _deployment( - config_hash="522e31c7cedc07c5", + config_hash="fdf48cbae5b2827c", volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}], volume_mounts=[{"name": "config", "mountPath": "/conf"}], env_from=None, @@ -1290,7 +1508,7 @@ def test_transform(case: TransformCase) -> None: ( "collector", _deployment( - config_hash="1f03e323432223be", + config_hash="234e99dbae76e1c5", volumes=[ {"name": "config", "configMap": {"name": "modelplane-collector"}}, {"name": "credentials-primary", "secret": {"secretName": "telemetry-credentials"}}, @@ -1357,7 +1575,7 @@ def test_transform(case: TransformCase) -> None: ( "collector", _deployment( - config_hash="b1539f5a25d85bfc", + config_hash="d1034140906566fe", volumes=[ {"name": "config", "configMap": {"name": "modelplane-collector"}}, {"name": "credentials-vendor", "secret": {"secretName": "vendor-token"}}, diff --git a/functions/compose-serving-stack/tests/test_fn.py b/functions/compose-serving-stack/tests/test_fn.py index d3e90092a..9721bdc78 100644 --- a/functions/compose-serving-stack/tests/test_fn.py +++ b/functions/compose-serving-stack/tests/test_fn.py @@ -1404,6 +1404,7 @@ def _collector_config( "memory_limiter", "resource/cluster", "transform/identity", + "filter/match", "transform/modelplane", "groupbyattrs/identity", "filter/modelplane", @@ -1660,6 +1661,16 @@ def _collector_config( } ] }, + # SGLang publishes its running and waiting counts as a total and again per + # priority, so the built-in keeps only the total. + "filter/match": { + "metrics": { + "datapoint": [ + 'metric.name == "sglang:num_running_reqs" and datapoint.attributes["priority"] != nil and datapoint.attributes["priority"] != ""', + 'metric.name == "sglang:num_queue_reqs" and datapoint.attributes["priority"] != nil and datapoint.attributes["priority"] != ""', + ] + } + }, "transform/modelplane": { "metric_statements": [ # The engines' token histograms as counters, extracted ahead of the @@ -4746,7 +4757,7 @@ def _to_dict(msg: message.Message) -> dict: service_extensions=None, ), "collector": _collector( - config_hash="be9842417e13d6c1", + config_hash="fa3cbe55df2af23e", volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}], volume_mounts=[{"name": "config", "mountPath": "/conf"}], env_from=None, @@ -5065,7 +5076,7 @@ def _to_dict(msg: message.Message) -> dict: ready=fnv1.READY_TRUE, ), "collector": _collector( - config_hash="5d3c556be06d8ca8", + config_hash="2c892622cfc36e73", volumes=[ {"name": "config", "configMap": {"name": "modelplane-collector"}}, {"name": "credentials-primary", "secret": {"secretName": "telemetry-credentials"}}, @@ -5352,7 +5363,7 @@ def _to_dict(msg: message.Message) -> dict: service_extensions=None, ), "collector": _collector( - config_hash="0a72aaee56455e72", + config_hash="2246c48054b33132", volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}], volume_mounts=[{"name": "config", "mountPath": "/conf"}], env_from=None, @@ -5615,7 +5626,7 @@ def _to_dict(msg: message.Message) -> dict: service_extensions=None, ), "collector": _collector( - config_hash="57e31b8f680bd6d3", + config_hash="4e79b223351de201", volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}], volume_mounts=[{"name": "config", "mountPath": "/conf"}], env_from=None, @@ -5906,7 +5917,7 @@ def _to_dict(msg: message.Message) -> dict: service_extensions=None, ), "collector": _collector( - config_hash="be9842417e13d6c1", + config_hash="fa3cbe55df2af23e", volumes=[{"name": "config", "configMap": {"name": "modelplane-collector"}}], volume_mounts=[{"name": "config", "mountPath": "/conf"}], env_from=None, diff --git a/schemas/.lock.json b/schemas/.lock.json index f5e193428..73a289a3e 100644 --- a/schemas/.lock.json +++ b/schemas/.lock.json @@ -1,6 +1,6 @@ { "packages": { - "fs://apis": "7765fe0a737410e9a3f3504455b088e8d2c9ba91fb33046a56acb5441f1ce741", + "fs://apis": "4df3ccd47ed5ac3a74dbd9a3889dd68999e9fe7d5d79b06da7ba162a72fe502f", "git://https://github.com/crossplane/crossplane/cluster/crds": "14c53aac914d393b5bcc00962fbc90d4be5d35bc", "git://https://github.com/modelplaneai/cluster-agent/cluster/charts/cluster-agent/files/crds": "a833d3f6f7d9a6ac6083e7453375b58b65ee741a", "xpkg://xpkg.upbound.io/upbound/provider-aws-ec2:v2.9.4": "sha256:7a0febdfb8da2d59a9fdcdbc2d476811f00566fcb941880313fc2034b0ca6177", diff --git a/schemas/python/models/ai/modelplane/metricmapping/v1alpha1.py b/schemas/python/models/ai/modelplane/metricmapping/v1alpha1.py index 3e79ca1f6..6c6cc531a 100644 --- a/schemas/python/models/ai/modelplane/metricmapping/v1alpha1.py +++ b/schemas/python/models/ai/modelplane/metricmapping/v1alpha1.py @@ -64,6 +64,21 @@ class Label(BaseModel): """ +class MatchItem(BaseModel): + absent: Literal[True] | None = None + """ + Keep series without the label. A label with an empty value counts as absent, as it does in Prometheus. + """ + label: constr(pattern=r'^[a-zA-Z_][a-zA-Z0-9_]*$', max_length=63) + """ + The label to match on. + """ + value: constr(max_length=253) | None = None + """ + Keep series where the label has this value. A series without the label doesn't match. + """ + + class Metric(BaseModel): from_: constr(pattern=r'^[a-zA-Z_:][a-zA-Z0-9_:]*$', max_length=255) = Field( ..., alias='from' @@ -95,6 +110,11 @@ class Metric(BaseModel): """ Labels to set on this metric's series. To fold several metrics into one name, give each its own entry with the same `to` and a different fixed value, such as direction: input and direction: output on modelplane_tokens_total. """ + match: list[MatchItem] | None = Field(None, max_length=16) + """ + Keep only this metric's series whose labels match every entry, and drop the rest before the rename. For a component that publishes the same count more than once, such as a total beside a breakdown of it, which would otherwise be summed twice. + It applies to the metric wherever it appears, so a second mapping of the same `from`, such as one taking a part of the same histogram, sees only the series it keeps. + """ part: Literal['Count', 'Sum'] | None = None """ Take a part of a histogram as a counter of its own, rather than the histogram itself. Count is how many observations it holds, which is a request count where the histogram measures request duration. Sum is their total. @@ -154,7 +174,7 @@ class MetricMapping(BaseModel): spec: Spec """ How one component's metrics become part of the modelplane_* surface. Modelplane renders every MetricMapping into each inference cluster's collector, so a mapping is written once on the control plane and reaches the whole fleet. - A mapping naming a component Modelplane already provides renames for is additive: its renames run after the built-in ones, on whatever those left behind. A metric a built-in already renamed no longer answers to the name it was emitted under, so a second mapping selecting on that name matches nothing and the built-in stands. Select on the `modelplane_` name instead to rename one of Modelplane's own. + A mapping naming a component Modelplane already provides renames for is additive: its renames run after the built-in ones, on whatever those left behind. A metric a built-in already renamed no longer answers to the name it was emitted under, so a second mapping's rename of that name does nothing and the built-in stands. Its `match` still applies, because matching runs before any rename and drops the series it doesn't keep wherever the metric appears. Select on the `modelplane_` name instead to rename one of Modelplane's own. """ status: Status | None = None