From 69e1bc8b36504c80a4cd1ae98f9c7d30180e3695 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=8B/Thor=28sten=29=3F/=20Schwesig?= <89909507+schwesig@users.noreply.github.com> Date: Mon, 19 Oct 2026 17:45:50 +0200 Subject: [PATCH 1/8] adr: Keep the MCO allowlist for now, keep exclusion open as an option MCO collects by allowlist, 227 pinned --match arguments against the local Prometheus. We run it that way becuase the goal was to stay close to MOC 1.0, not after comparing it with the alternative. Measured on the dev hub: MCO does pass a non-pinned matcher through, but Prometheus joins several match[] selectors with OR, so a negative matcher next to the existing 227 excludes nothing. The docs cover removing single metrics by name and say nothing about patterns. So we keep the allowlist as the model, not its contents, and write down what the measurements showed for whoever revisits this. --- .../README.md | 266 ++++++++++++++++++ 1 file changed, 266 insertions(+) create mode 100644 adrs/2026-10-19-mco-metrics-allowlist/README.md diff --git a/adrs/2026-10-19-mco-metrics-allowlist/README.md b/adrs/2026-10-19-mco-metrics-allowlist/README.md new file mode 100644 index 0000000..5ab1545 --- /dev/null +++ b/adrs/2026-10-19-mco-metrics-allowlist/README.md @@ -0,0 +1,266 @@ +--- +title: Keep the MCO allowlist for now, keep exclusion open as an option +--- + +## Context + +ACM MultiClusterObservability collects by allowlist. The hub's +`observability-metrics-allowlist` names 214 metrics plus 13 label matchers, and the +`metrics-collector` turns that into **227 `--match` arguments** against the local Prometheus +`/federate` endpoint. A second collector does the same for user workload metrics with 35 matchers. + +Every argument pins `__name__` to one exact value: + +``` +--match={__name__=":node_memory_MemAvailable_bytes:sum"} +--match={__name__="ALERTS"} +--match={__name__="acm_managed_cluster_labels"} +``` + +A metric that is not named does not get filtered out. It is never asked for. The collector has no +`--drop`, no `--exclude` and no `--relabel`, so this list is the whole mechanism. + +One correction to how we first described this: the allowlist is not purely additive. ACM documents +[removing a default metric][rm] by putting its name into `metrics_list.yaml` with a hyphen in +front, `-cluster_infrastructure_provider`. So excluding single metrics is supported, by name. What +has no documented form is excluding by pattern, which is the thing this ADR is about. + +We run it this way becuase the goal was to stay **as close to MOC 1.0 as possible**, and MOC 1.0 runs +the allowlist that ACM ships. Worth writing down that this was the reason, and not a comparison with +the alternative. + +### Why look at it again + +Two things made the question visible. + +**COO showed us the other shape.** The Cluster Observability Operator works with collection rules, +and those can exclude as easy as include. Which raises the question whether "everything except" is +something we cannot have, or something we never tried. + +**On the HCP side we already collect by exclusion.** We run `METRICS_SET=SRE` with the +`sre-metric-set` ConfigMap, and that is a drop configuration. A new metric from a control plane +component is collected unless a rule removes it. So inside one environment we run both models, and +only the central hop is allow-only. + +This asymmetry costs us something real. A new metric shows up on a cluster, gets scraped locally, +sits in the local Prometheus, and quietly does not reach the hub. Nobody gets told. It arrives +centrally when somebody remembers to add the name, and normaly somebody remembers only after they +needed it and it was not there. + +### How big the gap is + +A hosted cluster runs its own full monitoring stack, `prometheus-k8s` and `thanos-querier` plus user workload monitoring. That comes +with OpenShift and has nothing to do with MCO or COO. So everything gets measured locally, whether +we collect it or not. + +Measured on `oac-dev-workload0`, 2026-10-19: + +| | metric names | +|---|---| +| visible locally in the hosted cluster | **3412** | +| arriving at the hub for that cluster | **27** | + +The 27 are the control plane: apiserver, etcd, grpc, workqueue. One thing we did not establish is +how much of that gap the allowlist actually causes. The 27 arrive labelled with the hosted cluster +but are scraped on the hub by UWM, so the two numbers may not even describe the same path, and +central recording rules could carry some of the rest under different names. The gap is measured, +its cause is not. These are not exotic metrics: + +| metric | series locally | at the hub | +|---|---|---| +| `container_cpu_usage_seconds_total` | 1322 | not collected | +| `kube_deployment_status_replicas` | 106 | not collected | +| `kube_node_info` | 5 | not collected | +| `node_memory_MemFree_bytes` | 5 | not collected | + +`container_cpu_usage_seconds_total` is the base for every utilisation question. It gets measured on +every cluster and you cannot ask for it from the hub. + +So the allowlist does not cost us a handful of edge cases. We collect about one percent of what the +fleet measures anyway, and which one percent that is was decided by whatever metrics MOC 1.0 +happened to need. + +### What we measured + +Dev hub, 2026-10-19. OpenShift 4.22.14, ACM 2.17.3, COO 1.5.3. Read-only apart from one short test +described below. + +The installed collector is the classic one, `metrics-collector-deployment` plus +`endpoint-observability-operator`. There is no `PrometheusAgent` and no `ScrapeConfig`, so the newer +add-on is not in use. MCO federates from `prometheus-k8s.openshift-monitoring.svc:9091` and writes +to Observatorium every 300 seconds. + +There is also no `MonitoringStack`, and the reason is worth stating correctly, because it is neither +a limitation of COO nor a cost we decided against. A COO `MonitoringStack` per cluster, with +`remoteWrite` to a stack on the hub, was the **fallback** we had planned for the case that no object +storage became available for MCO. S3 did become available, MCO runs, and the fallback was never +needed. Our COO chart therefore deploys the operator and the UI plugin and nothing else. + +Building one now would not block MCO, the two do not interfere, but it would also not help. MCO +federates from platform monitoring and UWM, and would not read a third Prometheus unless we +repointed the collector or set up `remoteWrite`. It would add scrape load and storage on the +cluster for data that is already collected locally. How much depends on which targets it takes and +at what interval, which we did not work out, because a second stack is not needed for a change to +the MCO selection in the first place. + +**Prometheus handles exclusion without any trouble.** Through `/federate` on the hub's own +Prometheus: + +| matcher | distinct metric names | +|---|---| +| `{__name__=~".+"}` | 4038 | +| `{__name__=~".+",__name__!="go_goroutines"}` | 4037 | +| `{namespace="openshift-monitoring"}` | 1303 | + +**MCO passes a non-pinned matcher through unchanged.** We created +`observability-metrics-custom-allowlist` with one entry, `__name__=~"oactest_.*"`, picked because it +matches nothing today and therefore moves no data: + +| | | +|---|---| +| `--match` arguments before | 227 | +| after the ConfigMap was created | **228** | +| time until the collector was redeployed | 15 seconds | + +It was not rejected and not rewriten into a pinned form. The ConfigMap was deleted 54 seconds after +it was created, the count went back to 227, and `thanos-receive` ran through without a restart. + +### What that does and does not tell us + +Shown: the ConfigMap is accepted, the regex selector lands in the collector arguments, removing it +restores the old state. Not shown: that additional series actually travel, and that an exclusion +works next to the selectors already there. + +The second one we measured afterwards. Prometheus joins several `match[]` selectors with OR, not +AND, and our test added argument 228 next to 227 that stayed: + +| selectors passed to `/federate` | `go_goroutines` lines returned | +|---|---| +| the negative one alone | **0** | +| the negative one plus another that matches it | **138** | + +So a negative matcher added to the existing list excludes nothing. It would only work if the 227 +pinned selectors went away at the same time, which is a bigger change than adding one entry. + +What stands: the allowlist is a default and not a hard limit, and a broad selector is accepted. +Whether that is a documented interface or current behaviour, we did not find out, and an upgrade is +where an accident stops working. + +## Decision + +**We keep the allowlist**, as the model. It is what MOC 1.0 runs, there it is proven in production, +and our own reason for picking it still holds. + +We are not freezing its contents. Adding names stays normal work and does not reopen this +decision, CCI-MOC/oac-apps#140 is exactly that, and removing a default metric with `-name` is +equally normal and documented. Both directions are open by name. What we cannot have is the +pattern, so the question is narrower than "allowlist or blocklist": whether metrics we never named +should arrive by themselves. + +An alternative that exists does not make the current choice better. What changed is only what we +know, and the documentation gives us no cover for it. + +In the ACM 2.17 observability guide the custom allowlist is documented with `names` and +`recording_rules`, and nothing else. The `matches` field does appear, but only inside +`collect_rules`, in the section on [dynamic metrics for single-node OpenShift][sno], where extra +metrics get collected while a cluster is under load. Collection stops when the condition no longer +applies, at the earliest 15 minutes after it started. All four documented examples there pin +`__name__` to one exact value, like +`__name__="container_memory_rss",container!=""`. Broad or negative `__name__` matchers are not +described anywhere, and there is no warning about them either, the guide simply does not mention +the case. + +So `__name__=~"oactest_.*"` is measured implementation behaviour, not a documented interface. + +Second finding from the docs: ACM 2.17 documents a migration path away from the allowlist resource, +with a command-line tool shipped as a product tool, §1.11.12 "Migrating custom allowlist": + +> Use the allowlist-migration command-line interface to convert the +> observability-metrics-allowlist custom resource to ScrapeConfig and PrometheusRule resources. + +The reason the guide gives is scalability: "Converting to the ScrapeConfig and PrometheusRule +resources allows for a scalable way to manage your custom allowlist configuration." + +What the documentation does **not** say is that the allowlist is deprecated, legacy or superseded. +We looked, it is not there. So a migration path exists, with scalability as the stated reason, and +nothing announces the end of the current mechanism. Putting an undocumented extension on a resource +that already has a documented migration path is still a bet on two things at once. + +So we stay where we are. Not because the allowlist is better, but because we would swap something +that works for something we have not checked far enough. + +**We write the option down instead of closing it.** A custom allowlist entry like this one is +accepted: + +```yaml +data: + metrics_list.yaml: | + matches: + - __name__=~".+",__name__!="" +``` + +But on its own it does not give "everything except". Because the selectors are OR-ed, this adds a +broad selection next to the 227 that stay, and the exclusion has no effect on what they already +pull. A real block list needs the existing selectors gone, and how to do that without touching the +operator-owned `observability-metrics-allowlist` is the open question. + +The ConfigMap itself is a documented extension point. That the matcher syntax we put in it is +supported does not follow from that, and we have no evidence for it. + +### What would make us revisit + +Conditions, not a date: + +1. Red Hat documents the matcher semantics, so a broad matcher becomes a supported interface and + not just an observation. Checked on 2026-10-19 against the ACM 2.17 observability guide: it does + not, the allowlist is documented with `names` and `recording_rules` only. +2. The gap starts to cost us, meaning we keep adding metrics afterwards because somebody needed one + and it was not there. Worth counting that, right now it is a feeling and not a number. +3. We move away from the classic collector. The newer add-on with `PrometheusAgent` and + `ScrapeConfig` has a different filter model, so then the question comes up fresh anyway. ACM + 2.17 already ships `allowlist-migration` for that step, with the `multicluster-observability-addon` + on the hub as a prerequisite. Whenever we take it, the filtering model is open again and should + be decided then, not inherited. +4. We want a collection path that MCO does not give us. COO is installed and could run a + `MonitoringStack`, the fallback we planned before S3 existed. That is a different answer to a + different problem, so it only comes up if the requirement changes, not the filtering. + +## Consequences + +**By keeping the allowlist** we keep what we know: a bounded set, predictable volume, and the same +configuration MOC 1.0 runs since a long time. We keep the gap as well. A metric that exists on a +cluster you cannot query centrally until somebody names it, and nothing tells us when that happens. + +**The coverage gap is measured, its cost is not.** Of the metric names on that one cluster we +collect about one percent centrally, the rest is only reachable on the cluster itself. What that +costs in practice we have not quantified: how often somebody needs a metric that is not there, and +what the detour costs them. Every new exporter or operator means deciding whether we want its +metrics centrally, and that is easy to forget. We do not keep the allowlist because the gap is +small, we keep it because the alternative stands on behaviour we have seen once. + +**For the next person**, the measurements above are the starting point. What still needs answering: +whether the volume carries, what belongs on a block list, and how we would notice a +high-cardinality exporter that appears somewhere in the fleet. + +**One thing we would lose** if we ever switch. Today the allowlist is a kind of silent contract +about what the hub holds. Querying stays predictable because +the set is counted out. With exclusion the set is whatever the fleet exports, which is bigger and +moves by itself. + +## See also + +- `METRICS_SET=SRE` and the `sre-metric-set` ConfigMap, the drop-based model we already run for + hosted control planes +- ACM 2.17 observability guide, [dynamic metrics for single-node OpenShift][sno], the only + documented use of `matches`, and [§1.11.12 Migrating custom allowlist][mig] for the migration + path to `ScrapeConfig` and `PrometheusRule` + +[rm]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#removing-default-metrics +[sno]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#dynamic-metrics-sno +[mig]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#migrate-allowlist-mcoa +- CCI-MOC/oac-apps#140, which adds a custom allowlist as a named list, the shape this ADR keeps +- CCI-MOC/oac-apps#114, where the SRE drop rules turned out to cut about 40 percent rather than the + 94 percent that had been projected +- The ADR on per-project metric separation, which covers the other half of "why can somebody not see + this metric". This one is about what we collect at all; that one is about who may read what has + been collected. Both answers are needed to work out where to look when a metric is missing. From df0412c0656d8826eb71bf550878f081e8b1252c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=8B/Thor=28sten=29=3F/=20Schwesig?= <89909507+schwesig@users.noreply.github.com> Date: Thu, 22 Oct 2026 08:53:14 +0200 Subject: [PATCH 2/8] Update adrs/2026-10-19-mco-metrics-allowlist/README.md Co-authored-by: Lars Kellogg-Stedman --- adrs/2026-10-19-mco-metrics-allowlist/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/adrs/2026-10-19-mco-metrics-allowlist/README.md b/adrs/2026-10-19-mco-metrics-allowlist/README.md index 5ab1545..26d8249 100644 --- a/adrs/2026-10-19-mco-metrics-allowlist/README.md +++ b/adrs/2026-10-19-mco-metrics-allowlist/README.md @@ -20,10 +20,8 @@ Every argument pins `__name__` to one exact value: A metric that is not named does not get filtered out. It is never asked for. The collector has no `--drop`, no `--exclude` and no `--relabel`, so this list is the whole mechanism. -One correction to how we first described this: the allowlist is not purely additive. ACM documents -[removing a default metric][rm] by putting its name into `metrics_list.yaml` with a hyphen in -front, `-cluster_infrastructure_provider`. So excluding single metrics is supported, by name. What -has no documented form is excluding by pattern, which is the thing this ADR is about. +Despite it's name, the allowlist is not purely additive. ACM documents +[how to remove a metric][rm] by prefixing its name with a hyphen in `metrics_list.yaml`, for example `-cluster_infrastructure_provider`. This permits excluding specific metrics by name, but does not permit excluding metrics with a pattern, which is the thing this ADR is about. We run it this way becuase the goal was to stay **as close to MOC 1.0 as possible**, and MOC 1.0 runs the allowlist that ACM ships. Worth writing down that this was the reason, and not a comparison with From df94a79d28757ff0251912320198d0335c5a7167 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=8B/Thor=28sten=29=3F/=20Schwesig?= <89909507+schwesig@users.noreply.github.com> Date: Thu, 22 Oct 2026 11:36:18 +0200 Subject: [PATCH 3/8] adr: Take the review, and split the detail into two companion files larsks asked which path the metrics take, and the question was right: the central figure described one path and was reported as the whole. Corrected, together with his other three comments. The decision now rests on the documented mechanism alone, a name pattern is recorded as a convenience rather than as its justification, and the three selections that were conflated are told apart: the control plane drop list, local collection in UWM, and what reaches the hub. Measurement detail and the query recipes move to measurements.md and querying.md next to the ADR. That keeps the ADR readable without losing what somebody debugging a metric needs, and it is a new convention here, so say if you would rather have it in one file. --- .../README.md | 338 ++++++++++-------- .../measurements.md | 216 +++++++++++ .../querying.md | 126 +++++++ 3 files changed, 522 insertions(+), 158 deletions(-) create mode 100644 adrs/2026-10-19-mco-metrics-allowlist/measurements.md create mode 100644 adrs/2026-10-19-mco-metrics-allowlist/querying.md diff --git a/adrs/2026-10-19-mco-metrics-allowlist/README.md b/adrs/2026-10-19-mco-metrics-allowlist/README.md index 26d8249..d7d9075 100644 --- a/adrs/2026-10-19-mco-metrics-allowlist/README.md +++ b/adrs/2026-10-19-mco-metrics-allowlist/README.md @@ -9,7 +9,10 @@ ACM MultiClusterObservability collects by allowlist. The hub's `metrics-collector` turns that into **227 `--match` arguments** against the local Prometheus `/federate` endpoint. A second collector does the same for user workload metrics with 35 matchers. -Every argument pins `__name__` to one exact value: +Counted on 2026-10-19. It is 230 since CCI-MOC/oac-apps#140 merged on 2026-10-20 and added three DCGM +names through the custom allowlist, which is what a routine addition looks like. + +Nearly every argument pins `__name__` to one exact value: ``` --match={__name__=":node_memory_MemAvailable_bytes:sum"} @@ -17,193 +20,192 @@ Every argument pins `__name__` to one exact value: --match={__name__="acm_managed_cluster_labels"} ``` -A metric that is not named does not get filtered out. It is never asked for. The collector has no -`--drop`, no `--exclude` and no `--relabel`, so this list is the whole mechanism. +A metric no selector matches does not get filtered out. It is never asked for. The collector has no +`--drop`, no `--exclude` and no `--relabel`, so selection happens by asking rather than by filtering +what was scraped. + +The visible `--match` arguments do not describe the complete effective selection, though. A pattern +we put in the custom allowlist caused additional metrics to arrive without changing those arguments +at all. The additional configuration path was not fully traced, see +[measurements.md](measurements.md). So reading the arguments is not a reliable way to see the whole +selection. -Despite it's name, the allowlist is not purely additive. ACM documents -[how to remove a metric][rm] by prefixing its name with a hyphen in `metrics_list.yaml`, for example `-cluster_infrastructure_provider`. This permits excluding specific metrics by name, but does not permit excluding metrics with a pattern, which is the thing this ADR is about. +Despite its name, the allowlist is not purely additive. ACM documents [how to remove a metric][rm] +by prefixing its name with a hyphen in `metrics_list.yaml`, for example +`-cluster_infrastructure_provider`. This permits excluding specific metrics by name, but does not +permit excluding metrics with a pattern, which is the thing this ADR is about. We run it this way becuase the goal was to stay **as close to MOC 1.0 as possible**, and MOC 1.0 runs -the allowlist that ACM ships. Worth writing down that this was the reason, and not a comparison with -the alternative. +the allowlist that ACM ships. + +### The question + +Should locally collected metrics reach the hub by default, unless explicitly excluded? + +Today they do not. Two things made that worth asking. -### Why look at it again +**Not in question: the contents of the list.** Adding names is routine, removing one with `-name` is +documented, and a pattern adds all metrics matching a prefix in one line. Any set of metrics MCO +could collect is reachable by editing the list, so "we collect too few" is not a decision, it is a +pull request. What cannot be edited is the shape: -Two things made the question visible. +| | MCO | COO | +|---|---|---| +| allowlist | yes | yes | +| drop list | **no** | yes | + +That is upstream behaviour, not configuration, and it is the one constraint this ADR works around. **COO showed us the other shape.** The Cluster Observability Operator works with collection rules, -and those can exclude as easy as include. Which raises the question whether "everything except" is -something we cannot have, or something we never tried. +and those can exclude as easily as include. Which made it worth finding out whether "everything except" +is something MCO cannot do, or something we never tried. **On the HCP side we already collect by exclusion.** We run `METRICS_SET=SRE` with the `sre-metric-set` ConfigMap, and that is a drop configuration. A new metric from a control plane component is collected unless a rule removes it. So inside one environment we run both models, and only the central hop is allow-only. -This asymmetry costs us something real. A new metric shows up on a cluster, gets scraped locally, -sits in the local Prometheus, and quietly does not reach the hub. Nobody gets told. It arrives -centrally when somebody remembers to add the name, and normaly somebody remembers only after they +That asymmetry is defensible, and worth stating rather than fixing. The two halves have different +owners: the control plane is ours to run but the vendor's to shape, and the drop list is the shape +it ships with. What each cluster sends upward is our selection. Different owner, different model. + +What the allow-only hop does cost is attention. A new metric shows up on a cluster, gets scraped +locally, sits in the local Prometheus, and quietly does not reach the hub. Nobody gets told. It +arrives centrally when somebody remembers to name it, and normaly somebody remembers only after they needed it and it was not there. ### How big the gap is -A hosted cluster runs its own full monitoring stack, `prometheus-k8s` and `thanos-querier` plus user workload monitoring. That comes -with OpenShift and has nothing to do with MCO or COO. So everything gets measured locally, whether -we collect it or not. +A hosted cluster runs its own full monitoring stack, platform `prometheus-k8s` plus user workload +monitoring. That comes with OpenShift and is unrelated to MCO, so what those stacks scrape is +measured locally whether we collect it centrally or not. -Measured on `oac-dev-workload0`, 2026-10-19: - -| | metric names | +| | names | |---|---| -| visible locally in the hosted cluster | **3412** | -| arriving at the hub for that cluster | **27** | - -The 27 are the control plane: apiserver, etcd, grpc, workqueue. One thing we did not establish is -how much of that gap the allowlist actually causes. The 27 arrive labelled with the hosted cluster -but are scraped on the hub by UWM, so the two numbers may not even describe the same path, and -central recording rules could carry some of the rest under different names. The gap is measured, -its cause is not. These are not exotic metrics: - -| metric | series locally | at the hub | -|---|---|---| -| `container_cpu_usage_seconds_total` | 1322 | not collected | -| `kube_deployment_status_replicas` | 106 | not collected | -| `kube_node_info` | 5 | not collected | -| `node_memory_MemFree_bytes` | 5 | not collected | - -`container_cpu_usage_seconds_total` is the base for every utilisation question. It gets measured on -every cluster and you cannot ask for it from the hub. - -So the allowlist does not cost us a handful of edge cases. We collect about one percent of what the -fleet measures anyway, and which one percent that is was decided by whatever metrics MOC 1.0 -happened to need. - -### What we measured - -Dev hub, 2026-10-19. OpenShift 4.22.14, ACM 2.17.3, COO 1.5.3. Read-only apart from one short test -described below. - -The installed collector is the classic one, `metrics-collector-deployment` plus -`endpoint-observability-operator`. There is no `PrometheusAgent` and no `ScrapeConfig`, so the newer -add-on is not in use. MCO federates from `prometheus-k8s.openshift-monitoring.svc:9091` and writes -to Observatorium every 300 seconds. - -There is also no `MonitoringStack`, and the reason is worth stating correctly, because it is neither -a limitation of COO nor a cost we decided against. A COO `MonitoringStack` per cluster, with -`remoteWrite` to a stack on the hub, was the **fallback** we had planned for the case that no object -storage became available for MCO. S3 did become available, MCO runs, and the fallback was never -needed. Our COO chart therefore deploys the operator and the UI plugin and nothing else. - -Building one now would not block MCO, the two do not interfere, but it would also not help. MCO -federates from platform monitoring and UWM, and would not read a third Prometheus unless we -repointed the collector or set up `remoteWrite`. It would add scrape load and storage on the -cluster for data that is already collected locally. How much depends on which targets it takes and -at what interval, which we did not work out, because a second stack is not needed for a change to -the MCO selection in the first place. +| local on `oac-dev-workload0` | 4068 | +| arriving at the hub for it | 171 | + +One cluster on one day: the local figure was 3412 three days earlier. The two queries also ran +**without a common explicit time window**, and the count alone does not establish that the central +set is a subset of the local one. So the two numbers show that the endpoints report very different +amounts of metric names, not an exact current coverage. Details and the three paths a metric can take +to the hub in [measurements.md](measurements.md). + +Which names are missing is not the interesting part, since each is one list entry away. What matters +is where the 171 come from: a base taken over from MOC 1.0 for compatibility, extended since by +individual additions such as the three DCGM names, which are part of that count. So the selection has +been added to but never reviewed against MOC 2.0's own requirements. That was the occasion for +checking whether explicit selection is still the right model, which is what this ADR records. + +### What the measurements settled + +- **A `__name__` pattern in the custom allowlist collects.** `matches: [__name__=~"ovs_.*"]` took + `ovs_*` from 0 to 86 names across all three clusters, for one line, reverted afterwards. So + widening is cheap and fleet-wide. +- **`names:` works too**, proven in production by the three DCGM metrics since 2026-10-20. +- **An added pattern cannot narrow.** Prometheus joins `match[]` selectors with OR, so a selector + added next to the 230 can only union with them. That is an argument about adding, not proof that a + replaced selector could never restrict its own selection. +- **Our use of it is not documented.** A general name pattern in the custom allowlist works on ACM + 2.17.3, and the hub's own default list + uses it, but the guide documents only `names` and `recording_rules`. +- **Metrics can arrive more than once per cluster**, by up to three transport paths and, inside one + of them, from two ServiceMonitors scraping the same pods. A naive `count by (cluster)` counts those + series, not distinct subjects, and so overcounts when read as a count of them. Query patterns in + [querying.md](querying.md). -**Prometheus handles exclusion without any trouble.** Through `/federate` on the hub's own -Prometheus: -| matcher | distinct metric names | -|---|---| -| `{__name__=~".+"}` | 4038 | -| `{__name__=~".+",__name__!="go_goroutines"}` | 4037 | -| `{namespace="openshift-monitoring"}` | 1303 | +## Decision -**MCO passes a non-pinned matcher through unchanged.** We created -`observability-metrics-custom-allowlist` with one entry, `__name__=~"oactest_.*"`, picked because it -matches nothing today and therefore moves no data: +**Keep the MCO allowlist as the model for central collection, and do not narrow local collection to +match it.** -| | | -|---|---| -| `--match` arguments before | 227 | -| after the ConfigMap was created | **228** | -| time until the collector was redeployed | 15 seconds | +The allowlist stays for the reason it was picked: it is the collection model MOC 1.0 runs, in +production. We have since added to its contents, so this is the same model rather than the same +configuration. What this ADR adds is that keeping it costs less than assumed, and that central and +local selection are deliberately different. -It was not rejected and not rewriten into a pinned form. The ConfigMap was deleted 54 seconds after -it was created, the count went back to 227, and `thanos-receive` ran through without a restart. +Three selections exist and are often conflated. This ADR decides the third, and records that no +alignment of the second follows from it: -### What that does and does not tell us +| selection | governed by | scope | +|---|---|---| +| HCP control plane metrics | `sre-metric-set`, as `metricRelabelings` on 16 ServiceMonitors in `clusters-` | which control plane metrics survive the scrape, on the hub | +| local workload metrics | UWM, and whichever ServiceMonitors or PodMonitors exist | which configured targets a cluster scrapes for itself | +| central collection | **MCO allowlist** | which locally available metrics reach the hub | -Shown: the ConfigMap is accepted, the regex selector lands in the collector arguments, removing it -restores the old state. Not shown: that additional series actually travel, and that an exclusion -works next to the selectors already there. +What the hub buys is the comparison: every cluster in one query, grouped `by (cluster)`. The hub is +in it too, imported by ACM as `local-cluster` and sending against the same allowlist, so +`cluster:capacity_cpu_cores:sum` answers for infra and every workload at once. **All clusters, the +named metrics** — per cluster the intersection of the list and what that cluster exports, which is +why a cluster without GPUs has no `DCGM_*` although the list names it. -The second one we measured afterwards. Prometheus joins several `match[]` selectors with OR, not -AND, and our test added argument 228 next to 227 that stayed: +### Why not rebuild a drop list in MCO -| selectors passed to `/federate` | `go_goroutines` lines returned | -|---|---| -| the negative one alone | **0** | -| the negative one plus another that matches it | **138** | +`-name` removes an individual default metric, documented, so single exclusions exist. What MCO has no +form for is a **global drop rule that overrides the existing selectors**: an added `match[]` selector +is OR-ed with the 230 already there, so as an additional selector it can only widen. Reproducing +exclusion would mean replacing those selectors, which is a different change from editing a list. -So a negative matcher added to the existing list excludes nothing. It would only work if the 227 -pinned selectors went away at the same time, which is a bigger change than adding one entry. +### Why the gap is maintenance, not technology -What stands: the allowlist is a default and not a hard limit, and a broad selector is accepted. -Whether that is a documented interface or current behaviour, we did not find out, and an upgrade is -where an accident stops working. +Editing the selection is routine work and does not reopen this decision. CCI-MOC/oac-apps#140 is +exactly that: three names added, documented mechanism, no change of model. That holds with `names:` +alone, which is what the argument rests on. -## Decision - -**We keep the allowlist**, as the model. It is what MOC 1.0 runs, there it is proven in production, -and our own reason for picking it still holds. +A pattern makes a bulk addition more convenient — `matches: [__name__=~"ovs_.*"]` took `ovs_*` from 0 +to 86 names across all three clusters, see [measurements.md](measurements.md) — but it is a +convenience on top, not the reason the model is affordable. -We are not freezing its contents. Adding names stays normal work and does not reopen this -decision, CCI-MOC/oac-apps#140 is exactly that, and removing a default metric with `-name` is -equally normal and documented. Both directions are open by name. What we cannot have is the -pattern, so the question is narrower than "allowlist or blocklist": whether metrics we never named -should arrive by themselves. +What stays true is that a metric **no existing selector matches** does not reach the hub, and +nothing reports that it is missing. A pattern shifts where that line sits, it does not remove it. -An alternative that exists does not make the current choice better. What changed is only what we -know, and the documentation gives us no cover for it. +### Why local collection stays broader -In the ACM 2.17 observability guide the custom allowlist is documented with `names` and -`recording_rules`, and nothing else. The `matches` field does appear, but only inside -`collect_rules`, in the section on [dynamic metrics for single-node OpenShift][sno], where extra -metrics get collected while a cluster is under load. Collection stops when the condition no longer -applies, at the earliest 15 minutes after it started. All four documented examples there pin -`__name__` to one exact value, like -`__name__="container_memory_rss",container!=""`. Broad or negative `__name__` matchers are not -described anywhere, and there is no warning about them either, the guide simply does not mention -the case. +Aligning the two would mean governing local collection by an allowlist as well, one model and one +kind of edit. Rejected, because the trade is bad: -So `__name__=~"oactest_.*"` is measured implementation behaviour, not a documented interface. +| | today | local collection allow-only | +|---|---|---| +| metric allowlists to maintain | 1 (MCO) | 2 | +| a forgotten metric is missing | at the hub | at the hub **and on the cluster** | +| a new metric on an already scraped target | there locally, for the configured retention | only after somebody names it | -Second finding from the docs: ACM 2.17 documents a migration path away from the allowlist resource, -with a command-line tool shipped as a product tool, §1.11.12 "Migrating custom allowlist": +The third row is the point, and it needs stating precisely. A new exporter is **not** automatically +collected: it needs a ServiceMonitor or PodMonitor, and local filters and limits still apply. What +local collection gives for free is narrower and still valuable — **a new metric from a target that is +already scraped needs no central approval to be available locally**, for the retention that stack is +configured with, today 15d on the workload clusters. For a project that is the difference between +debugging now and filing a request. -> Use the allowlist-migration command-line interface to convert the -> observability-metrics-allowlist custom resource to ScrapeConfig and PrometheusRule resources. +This is also where the HCP drop list does **not** apply. `sre-metric-set` governs the control plane +ServiceMonitors on the hub, not project exporters in UWM, so nothing about it would change if local +collection were narrowed. The two are separate levers and the argument for keeping local collection +broad rests on UWM alone. -The reason the guide gives is scalability: "Converting to the ScrapeConfig and PrometheusRule -resources allows for a scalable way to manage your custom allowlist configuration." +Ownership matches the split: the control plane is ours to run but the vendor's to shape, and the drop +list is the shape it ships with. What each cluster sends to the hub is our selection. -What the documentation does **not** say is that the allowlist is deprecated, legacy or superseded. -We looked, it is not there. So a migration path exists, with scalability as the stated reason, and -nothing announces the end of the current mechanism. Putting an undocumented extension on a resource -that already has a documented migration path is still a bet on two things at once. +Not in scope: what the HCP `metrics-proxy` fetches on the hub-to-hosted-cluster hop. That list is +pinned by the operator, a transport limit rather than a selection one. -So we stay where we are. Not because the allowlist is better, but because we would swap something -that works for something we have not checked far enough. +### Caveat on the pattern -**We write the option down instead of closing it.** A custom allowlist entry like this one is -accepted: +Using `matches` as a **general name pattern in the custom allowlist** is measured behaviour on ACM +2.17.3, not a documented use. The guide documents the custom allowlist with `names` and +`recording_rules`, and shows `matches` only for dynamic SNO metrics with `__name__` pinned. Whether +Red Hat would treat a regression here as a bug is a support question we have not asked. Our chart +renders only `names:`, so a pattern needs a chart change first. -```yaml -data: - metrics_list.yaml: | - matches: - - __name__=~".+",__name__!="" -``` +ACM 2.17 also documents a migration away from the allowlist resource, §1.11.12, with an +`allowlist-migration` CLI converting it to `ScrapeConfig` and `PrometheusRule`, stated reason being +scalability. It does **not** say the allowlist is deprecated or superseded; we looked. -But on its own it does not give "everything except". Because the selectors are OR-ed, this adds a -broad selection next to the 227 that stay, and the exclusion has no effect on what they already -pull. A real block list needs the existing selectors gone, and how to do that without touching the -operator-owned `observability-metrics-allowlist` is the open question. +Hence the rule: **a pattern is a convenience, never the only thing standing between us and a metric +we need.** A metric something depends on — a dashboard, an alert, billing — gets named. If the +pattern stops working, the fallback is more entries in `names:`, documented and already rendered by +the chart, and the decision above stands unchanged. -The ConfigMap itself is a documented extension point. That the matcher syntax we put in it is -supported does not follow from that, and we have no evidence for it. ### What would make us revisit @@ -225,16 +227,33 @@ Conditions, not a date: ## Consequences -**By keeping the allowlist** we keep what we know: a bounded set, predictable volume, and the same -configuration MOC 1.0 runs since a long time. We keep the gap as well. A metric that exists on a -cluster you cannot query centrally until somebody names it, and nothing tells us when that happens. - -**The coverage gap is measured, its cost is not.** Of the metric names on that one cluster we -collect about one percent centrally, the rest is only reachable on the cluster itself. What that -costs in practice we have not quantified: how often somebody needs a metric that is not there, and -what the detour costs them. Every new exporter or operator means deciding whether we want its -metrics centrally, and that is easy to forget. We do not keep the allowlist because the gap is -small, we keep it because the alternative stands on behaviour we have seen once. +**By keeping the allowlist** we keep what we know: a bounded set of names and the collection model +MOC 1.0 has run for a long time. We keep the gap as well. A metric that exists on a cluster cannot be +queried centrally until a selector matches it, and nothing tells us when that happens. + +**Bounded names are not bounded volume.** The allowlist caps which names arrive, not how many series +they carry: one permitted name can grow without limit through new label values. So the two counts +describe the breadth of the selection and neither the share of ingested data nor the share of +operational value, and they are not a ratio. Cardinality and ingestion still need watching on their +own, and the allowlist only makes that easier, not unnecessary. + +**The measurements illustrate the difference in selection breadth, not its cost.** What that +difference costs in practice we have not quantified: how often somebody needs a metric that is not +there, and what the detour costs them. Every new exporter or operator means deciding whether we want its +metrics centrally, and that is easy to forget. + +**Closing the gap is cheap to configure, noticing it is not.** Adding a name is one list entry, and +a pattern brought all metrics matching a prefix in one line, 86 `ovs_*` names from three clusters. +Cheap here means configuration effort: the storage and ingestion a wider selection costs is not +something we measured. So the work is not in the editing, it is in somebody realising there is something to edit, +and effort is better spent on noticing than on the list. + +**The allowlist does not stop a project querying its own local data.** Project metrics from targets +the cluster's UWM stack already scrapes sit in that stack for the retention configured there, today +15d on the three workload clusters' UWM, whether or not a central selector matches them. That covers +local debugging. It does not cover everything a project might need centrally — cross-cluster +comparison, or history beyond the local retention — and for those the allowlist does apply. This has +to be explained that way, or the central count gets read as what a project can see. **For the next person**, the measurements above are the starting point. What still needs answering: whether the volume carries, what belongs on a block list, and how we would notice a @@ -247,18 +266,21 @@ moves by itself. ## See also +- [measurements.md](measurements.md), the numbers behind this decision and how they were taken +- [querying.md](querying.md), why a metric can arrive more than once per cluster and how to query + around it - `METRICS_SET=SRE` and the `sre-metric-set` ConfigMap, the drop-based model we already run for hosted control planes - ACM 2.17 observability guide, [dynamic metrics for single-node OpenShift][sno], the only documented use of `matches`, and [§1.11.12 Migrating custom allowlist][mig] for the migration path to `ScrapeConfig` and `PrometheusRule` - -[rm]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#removing-default-metrics -[sno]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#dynamic-metrics-sno -[mig]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#migrate-allowlist-mcoa - CCI-MOC/oac-apps#140, which adds a custom allowlist as a named list, the shape this ADR keeps - CCI-MOC/oac-apps#114, where the SRE drop rules turned out to cut about 40 percent rather than the 94 percent that had been projected - The ADR on per-project metric separation, which covers the other half of "why can somebody not see this metric". This one is about what we collect at all; that one is about who may read what has been collected. Both answers are needed to work out where to look when a metric is missing. + +[rm]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#removing-default-metrics +[sno]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#dynamic-metrics-sno +[mig]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#migrate-allowlist-mcoa diff --git a/adrs/2026-10-19-mco-metrics-allowlist/measurements.md b/adrs/2026-10-19-mco-metrics-allowlist/measurements.md new file mode 100644 index 0000000..e86757b --- /dev/null +++ b/adrs/2026-10-19-mco-metrics-allowlist/measurements.md @@ -0,0 +1,216 @@ +# Measurements behind the MCO allowlist ADR + +Supporting detail for [the ADR](README.md). Numbers are one cluster on one day; the local figure went +from 3412 on 2026-10-19 to 4068 on 2026-10-22 on the same cluster, so re-measure before quoting. + +**On reproducing these counts.** Every **metric-name count** below came from +`GET /api/v1/label/__name__/values` with the `match[]` shown, through the route named in each +section, and **without explicit `start` and `end`** — so each is the names the endpoint reported at +that moment, under whatever default window it applies, rather than names active in a stated interval. +Series counts come from `GET /api/v1/query` with `count(...)`, and the `/federate` figures further +down from that endpoint directly; both are noted where they appear. Good enough for the orders of magnitude this +ADR argues with; not good enough to compare a local against a central figure precisely, because the +two sides retain for different periods. A repeat should pin an identical explicit window on both +sides and say whether it counts names seen in that window or series present now. + +How volatile that makes the figures is visible in the data itself: the local count went from 3412 to +4068 in three days, and the central count for one cluster read 171 in the morning and 257 in the +afternoon of 2026-10-22, the difference being test data still inside retention. + +## Environment + +Dev hub, 2026-10-19 and 2026-10-22. OpenShift 4.22.14, ACM 2.17.3, COO 1.5.3. Read-only apart from +the one allowlist test below. + +The installed collector is the classic one, `metrics-collector-deployment` plus +`endpoint-observability-operator`. No `PrometheusAgent` and no `ScrapeConfig`, so the newer add-on is +not in use. MCO federates from `prometheus-k8s.openshift-monitoring.svc:9091` and writes to +Observatorium every 300s. + +## How much the allowlist selects + +| | names | queried against | +|---|---|---| +| local on `oac-dev-workload0` | 4068 | the cluster's own `thanos-querier` route | +| arriving at the hub for it | 171 | the hub's `rbac-query-proxy` route | + +``` +# local, run against the hosted cluster +GET /api/v1/label/__name__/values + +# central, run against the hub +GET /api/v1/label/__name__/values?match[]={cluster="oac-dev-workload0"} +``` + +Use the right endpoint on the hub: in our configuration the `thanos-querier` route there serves the +hub's own monitoring data, so it answers for `local-cluster` and not for the fleet. MCO data is +behind `rbac-query-proxy`. + +## The three paths to the hub + +Three ways exist. The counts below come from **label selections used to investigate the paths**, +not from selectors proven to identify a path, measured 2026-10-22 against the hub: + +| path | how it reaches the hub | selection used | names | +|---|---|---|---| +| HCP control plane | scraped by hub UWM in `clusters-` | `managementcluster="local-cluster"` | 28 | +| the cluster itself | scraped locally, federated by its own `metrics-collector` | no `managementcluster`, `container!="haproxy"` | 141 | +| HCP forwarder | fetched by `metrics-proxy` on the hub, relabelled to look native | `container="haproxy"` | 37 | + +Measured intersections, and they reconcile with the total: + +| | names | +|---|---| +| control plane ∩ forwarder | 22 | +| forwarder ∩ local | 12 | +| control plane ∩ local | 3 | +| **all three** | **2** (`up`, `grpc_server_started_total`) | +| union, 28 + 141 + 37 − 22 − 12 − 3 + 2 | **171** | + +Both names occur in all three label selections; that alone does not establish which transport paths +their series took. They are generic names, which is the likely reason they match everywhere. The +large control plane/forwarder overlap is expected in the same loose sense: both are understood to +carry the same control plane metrics, which is the duplication [querying.md](querying.md) is about. + +``` +# all against the hub, C = cluster="oac-dev-workload0",__name__!~"ovs_.*" +?match[]={C,managementcluster="local-cluster"} -> 28 +?match[]={C,managementcluster="",container!="haproxy"} -> 141 +?match[]={C,container="haproxy"} -> 37 +?match[]={C} -> 171 +?match[]={C,managementcluster=""} -> 166 # local + forwarder +?match[]={cluster="oac-dev-workload0",managementcluster=""} -> 252 # same, with the test names +``` + +**Why these are selections and not paths.** `container="haproxy"` is not a path marker: for this +cluster it appears in 11 namespaces, among them `openshift-monitoring` and `openshift-kube-scheduler` +as well as `clusters-oac-dev-workload0`, because ordinary routers use that container name too. It +coincides with the forwarder for control plane metric names, but **the 37 counted here are all names +matching `container="haproxy"`**, not a filtered subset: no extra control plane condition was applied +to that query. The `etcd_server_has_leader` example below illustrates the forwarder, it does not +identify the whole selection. A selector that identifies the forwarder on its own was not +established. `managementcluster=""` is likewise not the local path by itself — forwarded series carry +no such label, which is why that filter returns 166 rather than 141. + +So the three selections cover the 171 exactly and the intersections are consistent, but the mapping +from selection to transport path rests on the label sets below rather than on a proven filter. + +A note on the live totals: queried today the same cluster reports **257** names rather than 171, +because the 86 `ovs_*` names from the pattern test are still in the TSDB and age out with retention. +Every count above excludes them with `__name__!~"ovs_.*"` except where shown, and 257 − 86 = 171 +reproduces the figure from earlier in the day. + +The forwarder path exists only where `spec.monitoring.metricsForwarding` is on, today +`oac-dev-workload0` alone. + +One metric on two paths, both labelled `cluster="oac-dev-workload0"`, with the labels that tell them +apart: + +``` +etcd_server_has_leader job=etcd namespace=clusters-oac-dev-workload0 container=etcd-metrics managementcluster=local-cluster +etcd_server_has_leader job=etcd namespace=openshift-etcd container=haproxy (no managementcluster) +``` + +A metric that only takes the local path looks different again: `node_cpu_seconds_total` arrives with +`job=node-exporter`, `namespace=openshift-monitoring` and a node name in `instance`. + +## Patterns in the custom allowlist + +`matches: [__name__=~"ovs_.*"]` added next to the three DCGM names, dev hub, 2026-10-22, reverted +afterwards. Before the change the central query returned no `ovs_*` names at all, so the before value +is a clean zero. Locally the metrics existed throughout, which is what made them a usable test +subject. Both figures against that cluster's own `thanos-querier` route: + +``` +GET /api/v1/label/__name__/values?match[]={__name__=~"ovs_.*"} -> 86 names +GET /api/v1/query?query=count({__name__=~"ovs_.*"}) -> 445 series +``` + +| | before | with the pattern | +|---|---|---| +| `ovs_*` names at the hub, **fleet wide** | 0 | 86 | +| clusters contributing | — | 3 of 3 | +| `--match` arguments on the collector | 230 | 230 | + +``` +# before and after, run against the hub +?match[]={__name__=~"ovs_.*"} -> 0, then 86 names +?match[]={__name__=~"ovs_.*"} on label cluster -> workload0, workload1, workload2 +``` + +The 86 is the **union across the three clusters**, not 86 per cluster; the per-cluster split was not +recorded. The series arrived on the federated path, `job=ovnkube-node`, no `managementcluster` label. +86 is whatever `ovs_*` exports today; what the test shows is that one pattern reached all three +clusters under test. + +**The matcher count did not change, and the data arrived anyway.** The pattern appeared neither in +the merged `observability-metrics-allowlist` on the managed cluster nor in the collector arguments, +yet the metrics arrived. So the visible `--match` list is **not the whole effective selection**; +there is at least one further configuration path. + +What we observed of that path, and no more: five seconds after the write the operator logged +`triggered_by="observability-metrics-custom-allowlist"` together with +`"Find updated data in secret" secret="observability-metrics-allowlist"`. We did **not** inspect that +secret, so its contents and who consumes them are unverified. Treat the secret as a pointer to the +additional path, not as a documented mechanism. `names:` by contrast is visible: the three DCGM names +sit in the merged ConfigMap and in the collector arguments. + +Practical consequence for anyone testing this: watching the ConfigMap or the collector arguments +makes a working pattern look ignored. Watch the arriving metrics, or the operator log. + +`names:` is proven separately and in production: the three DCGM metrics have been at the hub since +2026-10-20 through this mechanism, and appear in the collector arguments as +`--match={__name__="DCGM_..."}`. + +## Exclusion + +Prometheus joins several `match[]` selectors with OR, so a selector **added** next to the 230 that +stay can only union with them. Three statements worth keeping apart: + +1. A **single** selector can restrict its own selection with a negative condition. Measured below, + 4038 against 4037. +2. An **added** selector removes nothing that the other selectors already match. This follows from + the OR semantics. +3. **Rebuilding the whole MCO selection** was never tested, and it is a different change from + editing a list. + +An earlier `/federate` observation from 2026-10-19 is not used as evidence: the requests were not +recorded and the route answers 503 for our tokens since 2026-10-22, so it cannot be repeated. Point 2 +rests on the OR semantics. + +For reference, a selector restricting itself is not the hard part. Measured 2026-10-19 through +`/federate` on the hub's own Prometheus, counting distinct `__name__` values in the response: + +| matcher | distinct names | +|---|---| +| `{__name__=~".+"}` | 4038 | +| `{__name__=~".+",__name__!="go_goroutines"}` | 4037 | +| `{namespace="openshift-monitoring"}` | 1303 | + +The single removed name is the whole point of the middle row. This route no longer answers for us, +see above. + +## Why there is no COO MonitoringStack + +Neither a COO limitation nor a cost we decided against. A `MonitoringStack` per cluster with +`remoteWrite` to a hub stack was the **fallback** planned in case no object storage became available +for MCO. S3 did become available, MCO runs, the fallback was never needed. Our COO chart deploys the +operator and the UI plugin, nothing else. + +Building one now would not block MCO, and it would not answer the question this ADR is about: MCO +federates from platform monitoring and UWM and would not read a third Prometheus without repointing +the collector or setting up `remoteWrite`, so the central selection would stay exactly as it is. It +would also add scrape load and storage for data already collected locally. + +## Documentation status + +ACM 2.17 documents the custom allowlist with `names` and `recording_rules`. `matches` appears in the +guide only inside `collect_rules`, for dynamic metrics on single-node OpenShift, and all four +examples there pin `__name__` to one value. + +The hub's own default allowlist nevertheless carries a top-level `matches:` with 13 entries, +including `job=~"apiserver|etcd"` and `container!=""`, and those reach the collector arguments on +every managed cluster. So the key is real and in production use; what is undocumented is the +specific use as a **general name pattern in the custom allowlist**, not `matches` as such. That is +measured implementation behaviour on 2.17.3. How Red Hat would classify a regression there is a +support question nobody has asked. diff --git a/adrs/2026-10-19-mco-metrics-allowlist/querying.md b/adrs/2026-10-19-mco-metrics-allowlist/querying.md new file mode 100644 index 0000000..01f0583 --- /dev/null +++ b/adrs/2026-10-19-mco-metrics-allowlist/querying.md @@ -0,0 +1,126 @@ +# Querying fleet metrics without double counting + +Supporting detail for [the ADR](README.md). A metric name can reach the hub by more than one path, +and more than one scrape of the same target can sit inside a single path, so one subject can produce +several series under the same `cluster` label. All queries below were run against +`rbac-query-proxy` on the dev hub, 2026-10-22. + +## The duplication + +`etcd_server_has_leader` for `oac-dev-workload0`: + +| namespace | container | `managementcluster` | `job` | series | +|---|---|---|---|---| +| `clusters-oac-dev-workload0` | `etcd-metrics` | `local-cluster` | `etcd` | 3 | +| `clusters-oac-dev-workload0` | `etcd-metrics` | `local-cluster` | `etcd-client` | 3 | +| `clusters-oac-dev-workload0` | `haproxy` | absent | `acm-etcd` | 3 | +| `openshift-etcd` | `haproxy` | absent | `etcd` | 3 | + +Twelve series for three etcd members, and **the transport paths explain only part of it**. Two +ServiceMonitors scrape the same pods on the hub, one as `job=etcd` and one as `job=etcd-client`, so +the hub-scraped group alone holds six series for three members. Why both exist was not investigated; +it is a second multiplication sitting inside the first. + +So `count by (cluster)` reports 12 series for workload0's three members and 6 for each of the other +clusters' three. Those series counts are correct; what is wrong is reading them as a number of +members, which overstates by four on workload0 and by two elsewhere. + +The two `haproxy` rows are the HCP forwarder. The second carries `namespace="openshift-etcd"`, which +on a hosted cluster holds no etcd: checked on the cluster, the namespace exists and is empty, no +pods. Forwarding relabels to the control plane's native identity, so the namespace names where such +a metric would live on a standalone cluster, not where it was produced. + +`cluster` says which cluster the data is **about**. The differences shown in `namespace`, `container`, +`managementcluster` and `job` reflect the scrape and forwarding contexts, not different subjects — +while `pod` below is the identity of the observed member. + +## How many names are affected + +Per cluster, counted as **names appearing in multiple `managementcluster` groups**, which is a +candidate list rather than proven duplication: + +| cluster | names in multiple `managementcluster` groups | +|---|---| +| `local-cluster` | 0 | +| `oac-dev-workload0` | 23 | +| `oac-dev-workload1` | 3 | +| `oac-dev-workload2` | 3 | + +workload1 and workload2 show only the generic three: `up`, `grpc_server_started_total` and one +recording rule. For generic names like these the groups may well describe entirely different targets, +so a hit is a reason to look, not a duplicate. workload0 shows twenty more, almost all etcd, and it +is the cluster with forwarding enabled. + +Enabling forwarding elsewhere should be expected to add overlaps, though not necessarily these same +23 names. The set therefore grows with a switch we control, not only with new exporters. + +## Counting members instead of series + +```promql +count by (cluster) (group by (cluster, pod) (etcd_server_has_leader)) +# -> local-cluster 3, workload0 3, workload1 3, workload2 3 +``` + +This answers **how many distinct etcd pods appear in the available samples**, which is usually what +somebody counting etcd wanted. `group by` says that intent plainly; `max by (cluster, pod)` returns the same +numbers here and is the older form of the same query. + +Two things it does **not** do. It is not a health check: the outer `count` counts groups regardless +of their value, so it returns 3 even if all three pods report `etcd_server_has_leader=0`. And it is +not a general recipe for deduplicating metric values — it works because `(cluster, pod)` identifies a +member for this metric, which we checked, and because every path reports the same `etcd-0/1/2`. + +**Do not group by `namespace`, `container` or `job`**: those are exactly the labels that differ per +scrape context, so adding them restores the duplication. + +## Showing scrape and forwarding contexts + +```promql +count by (cluster, namespace, container) (etcd_server_has_leader) +``` + +Suitable for a table or an overview panel, where seeing the separate rows is the point. This grouping +collapses `job=etcd` and `job=etcd-client` into one row; add `job` to see the four rows from the +table at the top. + +## Do not use managementcluster as the separator + +It distinguishes hub-scraped series from the rest, but forwarded and collector-sent series both have +it absent, so it does not separate all three paths. Filtering on it also drops clusters silently, +measured **for `etcd_server_has_leader`** — which clusters a filter returns depends on the metric: + +| filter | clusters returned | +|---|---| +| `managementcluster=""` | `local-cluster`, `workload0` | +| `managementcluster="local-cluster"` | `workload0`, `workload1`, `workload2` | + +Neither returns all four. The hub does have its own control plane, but its etcd metrics do not carry +`managementcluster="local-cluster"` in this measurement — that label marks hosted control planes +scraped on the hub, not the hub's own components. + +`container="haproxy"` identifies the forwarder for control plane metric names, but `haproxy` also +names ordinary router containers — 786 series on workload0 — so use it together with a metric name, +never alone. + +## Searching for possible overlaps + +Per cluster; an unbounded `{__name__=~".+"}` across the fleet times out here. + +```promql +count by (__name__) ( + count by (__name__, managementcluster) ({__name__=~".+", cluster="$cluster"}) +) > 1 +``` + +Grouping on `managementcluster` is deliberate. Adding `namespace` or `container` turns the query into +a list of ordinary multi-pod metrics: that variant returned 35 names for the hub, where this method +finds none. As written it returned 0 for the hub and 23, 3, 3 for the workloads. + +It is a search, not a proof, in both directions. **A hit does not establish a duplicate** — the +groups may describe different targets, which is likely for generic names such as `up`. **No hit does +not rule one out** either, because duplication sitting entirely on one side of the +`managementcluster` split is invisible to it; for that, compare `count by (namespace, container, job)` +for the metric in question. + +Worth running after enabling forwarding on a cluster, or after adding an exporter that uses generic +names such as `up`, `go_*` or `process_*`. From fe7e8ef7baea0d6d380707a378d0825a0e9390d7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=8B/Thor=28sten=29=3F/=20Schwesig?= <89909507+schwesig@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:20:35 +0100 Subject: [PATCH 4/8] adr: Rework the wording after the first review round Retitles the ADR, tightens the context paragraph and folds a few asides into the sentences they belong to. No change of substance; this replaces seven consecutive web editor saves from the same sitting. --- .../README.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/adrs/2026-10-19-mco-metrics-allowlist/README.md b/adrs/2026-10-19-mco-metrics-allowlist/README.md index d7d9075..711d56a 100644 --- a/adrs/2026-10-19-mco-metrics-allowlist/README.md +++ b/adrs/2026-10-19-mco-metrics-allowlist/README.md @@ -1,16 +1,15 @@ --- -title: Keep the MCO allowlist for now, keep exclusion open as an option +title: Collect central metrics using MultiClusterObservability allowlist keeping exclusion lists as a future change --- ## Context -ACM MultiClusterObservability collects by allowlist. The hub's +ACM MultiClusterObservability (MCO) collects metrics with an allowlist. The hub's `observability-metrics-allowlist` names 214 metrics plus 13 label matchers, and the `metrics-collector` turns that into **227 `--match` arguments** against the local Prometheus -`/federate` endpoint. A second collector does the same for user workload metrics with 35 matchers. - -Counted on 2026-10-19. It is 230 since CCI-MOC/oac-apps#140 merged on 2026-10-20 and added three DCGM -names through the custom allowlist, which is what a routine addition looks like. +`/federate` endpoint. A second collector does the same for user workload metrics with 35 matchers +(counted on 2026-10-19. It is 230 since CCI-MOC/oac-apps#140 merged on 2026-10-20 and added three +DCGM names through the custom allowlist, which is what a routine addition looks like). Nearly every argument pins `__name__` to one exact value: @@ -49,10 +48,12 @@ documented, and a pattern adds all metrics matching a prefix in one line. Any se could collect is reachable by editing the list, so "we collect too few" is not a decision, it is a pull request. What cannot be edited is the shape: -| | MCO | COO | -|---|---|---| -| allowlist | yes | yes | +| | MCO | COO | +|-----------|--------|-----| +| allowlist | yes | yes | | drop list | **no** | yes | +- MCO: Multi Cluster Observability +- COO: Cluster Observability Operator That is upstream behaviour, not configuration, and it is the one constraint this ADR works around. @@ -120,7 +121,7 @@ checking whether explicit selection is still the right model, which is what this **Keep the MCO allowlist as the model for central collection, and do not narrow local collection to match it.** -The allowlist stays for the reason it was picked: it is the collection model MOC 1.0 runs, in +The allowlist stays for the reason it was picked: it was the collection model used in MOC 1.0, in production. We have since added to its contents, so this is the same model rather than the same configuration. What this ADR adds is that keeping it costs less than assumed, and that central and local selection are deliberately different. From e76a522dcc6d6d9449b83b6865ac0111a8eef6c3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=8B/Thor=28sten=29=3F/=20Schwesig?= <89909507+schwesig@users.noreply.github.com> Date: Tue, 27 Oct 2026 19:45:22 +0100 Subject: [PATCH 5/8] adr: Say which use of matches is undocumented The bullet said "our use of it" and left the reader to work out what "it" pointed at. The key itself is documented and the hub's default list uses it; what the guide does not show is a general name pattern in the custom allowlist. --- adrs/2026-10-19-mco-metrics-allowlist/README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/adrs/2026-10-19-mco-metrics-allowlist/README.md b/adrs/2026-10-19-mco-metrics-allowlist/README.md index 711d56a..ddbee9c 100644 --- a/adrs/2026-10-19-mco-metrics-allowlist/README.md +++ b/adrs/2026-10-19-mco-metrics-allowlist/README.md @@ -107,9 +107,10 @@ checking whether explicit selection is still the right model, which is what this - **An added pattern cannot narrow.** Prometheus joins `match[]` selectors with OR, so a selector added next to the 230 can only union with them. That is an argument about adding, not proof that a replaced selector could never restrict its own selection. -- **Our use of it is not documented.** A general name pattern in the custom allowlist works on ACM - 2.17.3, and the hub's own default list - uses it, but the guide documents only `names` and `recording_rules`. +- **Using `matches:` as a general name pattern is not documented.** The key itself is, and the hub's + own default list carries 13 such entries, but the guide shows `matches` only inside `collect_rules` + with `__name__` pinned to one value. For the custom allowlist it documents `names` and + `recording_rules`. Measured on ACM 2.17.3, see [measurements.md](measurements.md). - **Metrics can arrive more than once per cluster**, by up to three transport paths and, inside one of them, from two ServiceMonitors scraping the same pods. A naive `count by (cluster)` counts those series, not distinct subjects, and so overcounts when read as a count of them. Query patterns in From 0e5aac66d6a5b0d9ee2d68d3d56ba9650df704f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=8B/Thor=28sten=29=3F/=20Schwesig?= <89909507+schwesig@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:10:09 +0100 Subject: [PATCH 6/8] adr: Drop the section that repeated earlier points All three paragraphs said something the ADR had already said: oac-apps#140 as a routine addition appears four times before this, the ovs_* measurement twice, and the quiet gap when no selector matches twice. What only this section carried, that the decision rests on names: alone and a pattern is convenience, is already the closing paragraph of "Caveat on the pattern", stated better there. --- adrs/2026-10-19-mco-metrics-allowlist/README.md | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/adrs/2026-10-19-mco-metrics-allowlist/README.md b/adrs/2026-10-19-mco-metrics-allowlist/README.md index ddbee9c..8360759 100644 --- a/adrs/2026-10-19-mco-metrics-allowlist/README.md +++ b/adrs/2026-10-19-mco-metrics-allowlist/README.md @@ -149,19 +149,6 @@ form for is a **global drop rule that overrides the existing selectors**: an add is OR-ed with the 230 already there, so as an additional selector it can only widen. Reproducing exclusion would mean replacing those selectors, which is a different change from editing a list. -### Why the gap is maintenance, not technology - -Editing the selection is routine work and does not reopen this decision. CCI-MOC/oac-apps#140 is -exactly that: three names added, documented mechanism, no change of model. That holds with `names:` -alone, which is what the argument rests on. - -A pattern makes a bulk addition more convenient — `matches: [__name__=~"ovs_.*"]` took `ovs_*` from 0 -to 86 names across all three clusters, see [measurements.md](measurements.md) — but it is a -convenience on top, not the reason the model is affordable. - -What stays true is that a metric **no existing selector matches** does not reach the hub, and -nothing reports that it is missing. A pattern shifts where that line sits, it does not remove it. - ### Why local collection stays broader Aligning the two would mean governing local collection by an allowlist as well, one model and one From 3f83ff30b6f24ad247306a987265826390d6fc59 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=8B/Thor=28sten=29=3F/=20Schwesig?= <89909507+schwesig@users.noreply.github.com> Date: Thu, 19 Nov 2026 08:06:30 +0100 Subject: [PATCH 7/8] adr: Take the two nits from the MOC review The sentence about a metric no selector matches read as if it were kept. And the ownership paragraph said the same thing as the one in the context section. --- adrs/2026-10-19-mco-metrics-allowlist/README.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/adrs/2026-10-19-mco-metrics-allowlist/README.md b/adrs/2026-10-19-mco-metrics-allowlist/README.md index 8360759..bb75b1e 100644 --- a/adrs/2026-10-19-mco-metrics-allowlist/README.md +++ b/adrs/2026-10-19-mco-metrics-allowlist/README.md @@ -19,9 +19,9 @@ Nearly every argument pins `__name__` to one exact value: --match={__name__="acm_managed_cluster_labels"} ``` -A metric no selector matches does not get filtered out. It is never asked for. The collector has no -`--drop`, no `--exclude` and no `--relabel`, so selection happens by asking rather than by filtering -what was scraped. +A metric that no selector matches never reaches the hub, not because it is filtered out but because +it is never requested. The collector has no `--drop`, no `--exclude` and no `--relabel`, so selection +happens by asking rather than by filtering what was scraped. The visible `--match` arguments do not describe the complete effective selection, though. A pattern we put in the custom allowlist caused additional metrics to arrive without changing those arguments @@ -172,9 +172,6 @@ ServiceMonitors on the hub, not project exporters in UWM, so nothing about it wo collection were narrowed. The two are separate levers and the argument for keeping local collection broad rests on UWM alone. -Ownership matches the split: the control plane is ours to run but the vendor's to shape, and the drop -list is the shape it ships with. What each cluster sends to the hub is our selection. - Not in scope: what the HCP `metrics-proxy` fetches on the hub-to-hosted-cluster hop. That list is pinned by the operator, a transport limit rather than a selection one. From 6a6b12b90093e7863981772b9ebb659f3b07b4cb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=8B/Thor=28sten=29=3F/=20Schwesig?= <89909507+schwesig@users.noreply.github.com> Date: Thu, 19 Nov 2026 08:19:58 +0100 Subject: [PATCH 8/8] adr: Fix the markdown the linter flags A blank line between the table and the list below it, and two places with a doubled blank line. --- adrs/2026-10-19-mco-metrics-allowlist/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/adrs/2026-10-19-mco-metrics-allowlist/README.md b/adrs/2026-10-19-mco-metrics-allowlist/README.md index bb75b1e..a6565a3 100644 --- a/adrs/2026-10-19-mco-metrics-allowlist/README.md +++ b/adrs/2026-10-19-mco-metrics-allowlist/README.md @@ -52,6 +52,7 @@ pull request. What cannot be edited is the shape: |-----------|--------|-----| | allowlist | yes | yes | | drop list | **no** | yes | + - MCO: Multi Cluster Observability - COO: Cluster Observability Operator @@ -116,7 +117,6 @@ checking whether explicit selection is still the right model, which is what this series, not distinct subjects, and so overcounts when read as a count of them. Query patterns in [querying.md](querying.md). - ## Decision **Keep the MCO allowlist as the model for central collection, and do not narrow local collection to @@ -192,7 +192,6 @@ we need.** A metric something depends on — a dashboard, an alert, billing — pattern stops working, the fallback is more entries in `names:`, documented and already rendered by the chart, and the decision above stands unchanged. - ### What would make us revisit Conditions, not a date: