diff --git a/adrs/2026-10-19-mco-metrics-allowlist/README.md b/adrs/2026-10-19-mco-metrics-allowlist/README.md new file mode 100644 index 0000000..a6565a3 --- /dev/null +++ b/adrs/2026-10-19-mco-metrics-allowlist/README.md @@ -0,0 +1,271 @@ +--- +title: Collect central metrics using MultiClusterObservability allowlist keeping exclusion lists as a future change +--- + +## Context + +ACM MultiClusterObservability (MCO) collects metrics with an allowlist. The hub's +`observability-metrics-allowlist` names 214 metrics plus 13 label matchers, and the +`metrics-collector` turns that into **227 `--match` arguments** against the local Prometheus +`/federate` endpoint. A second collector does the same for user workload metrics with 35 matchers +(counted on 2026-10-19. It is 230 since CCI-MOC/oac-apps#140 merged on 2026-10-20 and added three +DCGM names through the custom allowlist, which is what a routine addition looks like). + +Nearly every argument pins `__name__` to one exact value: + +``` +--match={__name__=":node_memory_MemAvailable_bytes:sum"} +--match={__name__="ALERTS"} +--match={__name__="acm_managed_cluster_labels"} +``` + +A metric that no selector matches never reaches the hub, not because it is filtered out but because +it is never requested. The collector has no `--drop`, no `--exclude` and no `--relabel`, so selection +happens by asking rather than by filtering what was scraped. + +The visible `--match` arguments do not describe the complete effective selection, though. A pattern +we put in the custom allowlist caused additional metrics to arrive without changing those arguments +at all. The additional configuration path was not fully traced, see +[measurements.md](measurements.md). So reading the arguments is not a reliable way to see the whole +selection. + +Despite its name, the allowlist is not purely additive. ACM documents [how to remove a metric][rm] +by prefixing its name with a hyphen in `metrics_list.yaml`, for example +`-cluster_infrastructure_provider`. This permits excluding specific metrics by name, but does not +permit excluding metrics with a pattern, which is the thing this ADR is about. + +We run it this way becuase the goal was to stay **as close to MOC 1.0 as possible**, and MOC 1.0 runs +the allowlist that ACM ships. + +### The question + +Should locally collected metrics reach the hub by default, unless explicitly excluded? + +Today they do not. Two things made that worth asking. + +**Not in question: the contents of the list.** Adding names is routine, removing one with `-name` is +documented, and a pattern adds all metrics matching a prefix in one line. Any set of metrics MCO +could collect is reachable by editing the list, so "we collect too few" is not a decision, it is a +pull request. What cannot be edited is the shape: + +| | MCO | COO | +|-----------|--------|-----| +| allowlist | yes | yes | +| drop list | **no** | yes | + +- MCO: Multi Cluster Observability +- COO: Cluster Observability Operator + +That is upstream behaviour, not configuration, and it is the one constraint this ADR works around. + +**COO showed us the other shape.** The Cluster Observability Operator works with collection rules, +and those can exclude as easily as include. Which made it worth finding out whether "everything except" +is something MCO cannot do, or something we never tried. + +**On the HCP side we already collect by exclusion.** We run `METRICS_SET=SRE` with the +`sre-metric-set` ConfigMap, and that is a drop configuration. A new metric from a control plane +component is collected unless a rule removes it. So inside one environment we run both models, and +only the central hop is allow-only. + +That asymmetry is defensible, and worth stating rather than fixing. The two halves have different +owners: the control plane is ours to run but the vendor's to shape, and the drop list is the shape +it ships with. What each cluster sends upward is our selection. Different owner, different model. + +What the allow-only hop does cost is attention. A new metric shows up on a cluster, gets scraped +locally, sits in the local Prometheus, and quietly does not reach the hub. Nobody gets told. It +arrives centrally when somebody remembers to name it, and normaly somebody remembers only after they +needed it and it was not there. + +### How big the gap is + +A hosted cluster runs its own full monitoring stack, platform `prometheus-k8s` plus user workload +monitoring. That comes with OpenShift and is unrelated to MCO, so what those stacks scrape is +measured locally whether we collect it centrally or not. + +| | names | +|---|---| +| local on `oac-dev-workload0` | 4068 | +| arriving at the hub for it | 171 | + +One cluster on one day: the local figure was 3412 three days earlier. The two queries also ran +**without a common explicit time window**, and the count alone does not establish that the central +set is a subset of the local one. So the two numbers show that the endpoints report very different +amounts of metric names, not an exact current coverage. Details and the three paths a metric can take +to the hub in [measurements.md](measurements.md). + +Which names are missing is not the interesting part, since each is one list entry away. What matters +is where the 171 come from: a base taken over from MOC 1.0 for compatibility, extended since by +individual additions such as the three DCGM names, which are part of that count. So the selection has +been added to but never reviewed against MOC 2.0's own requirements. That was the occasion for +checking whether explicit selection is still the right model, which is what this ADR records. + +### What the measurements settled + +- **A `__name__` pattern in the custom allowlist collects.** `matches: [__name__=~"ovs_.*"]` took + `ovs_*` from 0 to 86 names across all three clusters, for one line, reverted afterwards. So + widening is cheap and fleet-wide. +- **`names:` works too**, proven in production by the three DCGM metrics since 2026-10-20. +- **An added pattern cannot narrow.** Prometheus joins `match[]` selectors with OR, so a selector + added next to the 230 can only union with them. That is an argument about adding, not proof that a + replaced selector could never restrict its own selection. +- **Using `matches:` as a general name pattern is not documented.** The key itself is, and the hub's + own default list carries 13 such entries, but the guide shows `matches` only inside `collect_rules` + with `__name__` pinned to one value. For the custom allowlist it documents `names` and + `recording_rules`. Measured on ACM 2.17.3, see [measurements.md](measurements.md). +- **Metrics can arrive more than once per cluster**, by up to three transport paths and, inside one + of them, from two ServiceMonitors scraping the same pods. A naive `count by (cluster)` counts those + series, not distinct subjects, and so overcounts when read as a count of them. Query patterns in + [querying.md](querying.md). + +## Decision + +**Keep the MCO allowlist as the model for central collection, and do not narrow local collection to +match it.** + +The allowlist stays for the reason it was picked: it was the collection model used in MOC 1.0, in +production. We have since added to its contents, so this is the same model rather than the same +configuration. What this ADR adds is that keeping it costs less than assumed, and that central and +local selection are deliberately different. + +Three selections exist and are often conflated. This ADR decides the third, and records that no +alignment of the second follows from it: + +| selection | governed by | scope | +|---|---|---| +| HCP control plane metrics | `sre-metric-set`, as `metricRelabelings` on 16 ServiceMonitors in `clusters-` | which control plane metrics survive the scrape, on the hub | +| local workload metrics | UWM, and whichever ServiceMonitors or PodMonitors exist | which configured targets a cluster scrapes for itself | +| central collection | **MCO allowlist** | which locally available metrics reach the hub | + +What the hub buys is the comparison: every cluster in one query, grouped `by (cluster)`. The hub is +in it too, imported by ACM as `local-cluster` and sending against the same allowlist, so +`cluster:capacity_cpu_cores:sum` answers for infra and every workload at once. **All clusters, the +named metrics** — per cluster the intersection of the list and what that cluster exports, which is +why a cluster without GPUs has no `DCGM_*` although the list names it. + +### Why not rebuild a drop list in MCO + +`-name` removes an individual default metric, documented, so single exclusions exist. What MCO has no +form for is a **global drop rule that overrides the existing selectors**: an added `match[]` selector +is OR-ed with the 230 already there, so as an additional selector it can only widen. Reproducing +exclusion would mean replacing those selectors, which is a different change from editing a list. + +### Why local collection stays broader + +Aligning the two would mean governing local collection by an allowlist as well, one model and one +kind of edit. Rejected, because the trade is bad: + +| | today | local collection allow-only | +|---|---|---| +| metric allowlists to maintain | 1 (MCO) | 2 | +| a forgotten metric is missing | at the hub | at the hub **and on the cluster** | +| a new metric on an already scraped target | there locally, for the configured retention | only after somebody names it | + +The third row is the point, and it needs stating precisely. A new exporter is **not** automatically +collected: it needs a ServiceMonitor or PodMonitor, and local filters and limits still apply. What +local collection gives for free is narrower and still valuable — **a new metric from a target that is +already scraped needs no central approval to be available locally**, for the retention that stack is +configured with, today 15d on the workload clusters. For a project that is the difference between +debugging now and filing a request. + +This is also where the HCP drop list does **not** apply. `sre-metric-set` governs the control plane +ServiceMonitors on the hub, not project exporters in UWM, so nothing about it would change if local +collection were narrowed. The two are separate levers and the argument for keeping local collection +broad rests on UWM alone. + +Not in scope: what the HCP `metrics-proxy` fetches on the hub-to-hosted-cluster hop. That list is +pinned by the operator, a transport limit rather than a selection one. + +### Caveat on the pattern + +Using `matches` as a **general name pattern in the custom allowlist** is measured behaviour on ACM +2.17.3, not a documented use. The guide documents the custom allowlist with `names` and +`recording_rules`, and shows `matches` only for dynamic SNO metrics with `__name__` pinned. Whether +Red Hat would treat a regression here as a bug is a support question we have not asked. Our chart +renders only `names:`, so a pattern needs a chart change first. + +ACM 2.17 also documents a migration away from the allowlist resource, §1.11.12, with an +`allowlist-migration` CLI converting it to `ScrapeConfig` and `PrometheusRule`, stated reason being +scalability. It does **not** say the allowlist is deprecated or superseded; we looked. + +Hence the rule: **a pattern is a convenience, never the only thing standing between us and a metric +we need.** A metric something depends on — a dashboard, an alert, billing — gets named. If the +pattern stops working, the fallback is more entries in `names:`, documented and already rendered by +the chart, and the decision above stands unchanged. + +### What would make us revisit + +Conditions, not a date: + +1. Red Hat documents the matcher semantics, so a broad matcher becomes a supported interface and + not just an observation. Checked on 2026-10-19 against the ACM 2.17 observability guide: it does + not, the allowlist is documented with `names` and `recording_rules` only. +2. The gap starts to cost us, meaning we keep adding metrics afterwards because somebody needed one + and it was not there. Worth counting that, right now it is a feeling and not a number. +3. We move away from the classic collector. The newer add-on with `PrometheusAgent` and + `ScrapeConfig` has a different filter model, so then the question comes up fresh anyway. ACM + 2.17 already ships `allowlist-migration` for that step, with the `multicluster-observability-addon` + on the hub as a prerequisite. Whenever we take it, the filtering model is open again and should + be decided then, not inherited. +4. We want a collection path that MCO does not give us. COO is installed and could run a + `MonitoringStack`, the fallback we planned before S3 existed. That is a different answer to a + different problem, so it only comes up if the requirement changes, not the filtering. + +## Consequences + +**By keeping the allowlist** we keep what we know: a bounded set of names and the collection model +MOC 1.0 has run for a long time. We keep the gap as well. A metric that exists on a cluster cannot be +queried centrally until a selector matches it, and nothing tells us when that happens. + +**Bounded names are not bounded volume.** The allowlist caps which names arrive, not how many series +they carry: one permitted name can grow without limit through new label values. So the two counts +describe the breadth of the selection and neither the share of ingested data nor the share of +operational value, and they are not a ratio. Cardinality and ingestion still need watching on their +own, and the allowlist only makes that easier, not unnecessary. + +**The measurements illustrate the difference in selection breadth, not its cost.** What that +difference costs in practice we have not quantified: how often somebody needs a metric that is not +there, and what the detour costs them. Every new exporter or operator means deciding whether we want its +metrics centrally, and that is easy to forget. + +**Closing the gap is cheap to configure, noticing it is not.** Adding a name is one list entry, and +a pattern brought all metrics matching a prefix in one line, 86 `ovs_*` names from three clusters. +Cheap here means configuration effort: the storage and ingestion a wider selection costs is not +something we measured. So the work is not in the editing, it is in somebody realising there is something to edit, +and effort is better spent on noticing than on the list. + +**The allowlist does not stop a project querying its own local data.** Project metrics from targets +the cluster's UWM stack already scrapes sit in that stack for the retention configured there, today +15d on the three workload clusters' UWM, whether or not a central selector matches them. That covers +local debugging. It does not cover everything a project might need centrally — cross-cluster +comparison, or history beyond the local retention — and for those the allowlist does apply. This has +to be explained that way, or the central count gets read as what a project can see. + +**For the next person**, the measurements above are the starting point. What still needs answering: +whether the volume carries, what belongs on a block list, and how we would notice a +high-cardinality exporter that appears somewhere in the fleet. + +**One thing we would lose** if we ever switch. Today the allowlist is a kind of silent contract +about what the hub holds. Querying stays predictable because +the set is counted out. With exclusion the set is whatever the fleet exports, which is bigger and +moves by itself. + +## See also + +- [measurements.md](measurements.md), the numbers behind this decision and how they were taken +- [querying.md](querying.md), why a metric can arrive more than once per cluster and how to query + around it +- `METRICS_SET=SRE` and the `sre-metric-set` ConfigMap, the drop-based model we already run for + hosted control planes +- ACM 2.17 observability guide, [dynamic metrics for single-node OpenShift][sno], the only + documented use of `matches`, and [§1.11.12 Migrating custom allowlist][mig] for the migration + path to `ScrapeConfig` and `PrometheusRule` +- CCI-MOC/oac-apps#140, which adds a custom allowlist as a named list, the shape this ADR keeps +- CCI-MOC/oac-apps#114, where the SRE drop rules turned out to cut about 40 percent rather than the + 94 percent that had been projected +- The ADR on per-project metric separation, which covers the other half of "why can somebody not see + this metric". This one is about what we collect at all; that one is about who may read what has + been collected. Both answers are needed to work out where to look when a metric is missing. + +[rm]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#removing-default-metrics +[sno]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#dynamic-metrics-sno +[mig]: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#migrate-allowlist-mcoa diff --git a/adrs/2026-10-19-mco-metrics-allowlist/measurements.md b/adrs/2026-10-19-mco-metrics-allowlist/measurements.md new file mode 100644 index 0000000..e86757b --- /dev/null +++ b/adrs/2026-10-19-mco-metrics-allowlist/measurements.md @@ -0,0 +1,216 @@ +# Measurements behind the MCO allowlist ADR + +Supporting detail for [the ADR](README.md). Numbers are one cluster on one day; the local figure went +from 3412 on 2026-10-19 to 4068 on 2026-10-22 on the same cluster, so re-measure before quoting. + +**On reproducing these counts.** Every **metric-name count** below came from +`GET /api/v1/label/__name__/values` with the `match[]` shown, through the route named in each +section, and **without explicit `start` and `end`** — so each is the names the endpoint reported at +that moment, under whatever default window it applies, rather than names active in a stated interval. +Series counts come from `GET /api/v1/query` with `count(...)`, and the `/federate` figures further +down from that endpoint directly; both are noted where they appear. Good enough for the orders of magnitude this +ADR argues with; not good enough to compare a local against a central figure precisely, because the +two sides retain for different periods. A repeat should pin an identical explicit window on both +sides and say whether it counts names seen in that window or series present now. + +How volatile that makes the figures is visible in the data itself: the local count went from 3412 to +4068 in three days, and the central count for one cluster read 171 in the morning and 257 in the +afternoon of 2026-10-22, the difference being test data still inside retention. + +## Environment + +Dev hub, 2026-10-19 and 2026-10-22. OpenShift 4.22.14, ACM 2.17.3, COO 1.5.3. Read-only apart from +the one allowlist test below. + +The installed collector is the classic one, `metrics-collector-deployment` plus +`endpoint-observability-operator`. No `PrometheusAgent` and no `ScrapeConfig`, so the newer add-on is +not in use. MCO federates from `prometheus-k8s.openshift-monitoring.svc:9091` and writes to +Observatorium every 300s. + +## How much the allowlist selects + +| | names | queried against | +|---|---|---| +| local on `oac-dev-workload0` | 4068 | the cluster's own `thanos-querier` route | +| arriving at the hub for it | 171 | the hub's `rbac-query-proxy` route | + +``` +# local, run against the hosted cluster +GET /api/v1/label/__name__/values + +# central, run against the hub +GET /api/v1/label/__name__/values?match[]={cluster="oac-dev-workload0"} +``` + +Use the right endpoint on the hub: in our configuration the `thanos-querier` route there serves the +hub's own monitoring data, so it answers for `local-cluster` and not for the fleet. MCO data is +behind `rbac-query-proxy`. + +## The three paths to the hub + +Three ways exist. The counts below come from **label selections used to investigate the paths**, +not from selectors proven to identify a path, measured 2026-10-22 against the hub: + +| path | how it reaches the hub | selection used | names | +|---|---|---|---| +| HCP control plane | scraped by hub UWM in `clusters-` | `managementcluster="local-cluster"` | 28 | +| the cluster itself | scraped locally, federated by its own `metrics-collector` | no `managementcluster`, `container!="haproxy"` | 141 | +| HCP forwarder | fetched by `metrics-proxy` on the hub, relabelled to look native | `container="haproxy"` | 37 | + +Measured intersections, and they reconcile with the total: + +| | names | +|---|---| +| control plane ∩ forwarder | 22 | +| forwarder ∩ local | 12 | +| control plane ∩ local | 3 | +| **all three** | **2** (`up`, `grpc_server_started_total`) | +| union, 28 + 141 + 37 − 22 − 12 − 3 + 2 | **171** | + +Both names occur in all three label selections; that alone does not establish which transport paths +their series took. They are generic names, which is the likely reason they match everywhere. The +large control plane/forwarder overlap is expected in the same loose sense: both are understood to +carry the same control plane metrics, which is the duplication [querying.md](querying.md) is about. + +``` +# all against the hub, C = cluster="oac-dev-workload0",__name__!~"ovs_.*" +?match[]={C,managementcluster="local-cluster"} -> 28 +?match[]={C,managementcluster="",container!="haproxy"} -> 141 +?match[]={C,container="haproxy"} -> 37 +?match[]={C} -> 171 +?match[]={C,managementcluster=""} -> 166 # local + forwarder +?match[]={cluster="oac-dev-workload0",managementcluster=""} -> 252 # same, with the test names +``` + +**Why these are selections and not paths.** `container="haproxy"` is not a path marker: for this +cluster it appears in 11 namespaces, among them `openshift-monitoring` and `openshift-kube-scheduler` +as well as `clusters-oac-dev-workload0`, because ordinary routers use that container name too. It +coincides with the forwarder for control plane metric names, but **the 37 counted here are all names +matching `container="haproxy"`**, not a filtered subset: no extra control plane condition was applied +to that query. The `etcd_server_has_leader` example below illustrates the forwarder, it does not +identify the whole selection. A selector that identifies the forwarder on its own was not +established. `managementcluster=""` is likewise not the local path by itself — forwarded series carry +no such label, which is why that filter returns 166 rather than 141. + +So the three selections cover the 171 exactly and the intersections are consistent, but the mapping +from selection to transport path rests on the label sets below rather than on a proven filter. + +A note on the live totals: queried today the same cluster reports **257** names rather than 171, +because the 86 `ovs_*` names from the pattern test are still in the TSDB and age out with retention. +Every count above excludes them with `__name__!~"ovs_.*"` except where shown, and 257 − 86 = 171 +reproduces the figure from earlier in the day. + +The forwarder path exists only where `spec.monitoring.metricsForwarding` is on, today +`oac-dev-workload0` alone. + +One metric on two paths, both labelled `cluster="oac-dev-workload0"`, with the labels that tell them +apart: + +``` +etcd_server_has_leader job=etcd namespace=clusters-oac-dev-workload0 container=etcd-metrics managementcluster=local-cluster +etcd_server_has_leader job=etcd namespace=openshift-etcd container=haproxy (no managementcluster) +``` + +A metric that only takes the local path looks different again: `node_cpu_seconds_total` arrives with +`job=node-exporter`, `namespace=openshift-monitoring` and a node name in `instance`. + +## Patterns in the custom allowlist + +`matches: [__name__=~"ovs_.*"]` added next to the three DCGM names, dev hub, 2026-10-22, reverted +afterwards. Before the change the central query returned no `ovs_*` names at all, so the before value +is a clean zero. Locally the metrics existed throughout, which is what made them a usable test +subject. Both figures against that cluster's own `thanos-querier` route: + +``` +GET /api/v1/label/__name__/values?match[]={__name__=~"ovs_.*"} -> 86 names +GET /api/v1/query?query=count({__name__=~"ovs_.*"}) -> 445 series +``` + +| | before | with the pattern | +|---|---|---| +| `ovs_*` names at the hub, **fleet wide** | 0 | 86 | +| clusters contributing | — | 3 of 3 | +| `--match` arguments on the collector | 230 | 230 | + +``` +# before and after, run against the hub +?match[]={__name__=~"ovs_.*"} -> 0, then 86 names +?match[]={__name__=~"ovs_.*"} on label cluster -> workload0, workload1, workload2 +``` + +The 86 is the **union across the three clusters**, not 86 per cluster; the per-cluster split was not +recorded. The series arrived on the federated path, `job=ovnkube-node`, no `managementcluster` label. +86 is whatever `ovs_*` exports today; what the test shows is that one pattern reached all three +clusters under test. + +**The matcher count did not change, and the data arrived anyway.** The pattern appeared neither in +the merged `observability-metrics-allowlist` on the managed cluster nor in the collector arguments, +yet the metrics arrived. So the visible `--match` list is **not the whole effective selection**; +there is at least one further configuration path. + +What we observed of that path, and no more: five seconds after the write the operator logged +`triggered_by="observability-metrics-custom-allowlist"` together with +`"Find updated data in secret" secret="observability-metrics-allowlist"`. We did **not** inspect that +secret, so its contents and who consumes them are unverified. Treat the secret as a pointer to the +additional path, not as a documented mechanism. `names:` by contrast is visible: the three DCGM names +sit in the merged ConfigMap and in the collector arguments. + +Practical consequence for anyone testing this: watching the ConfigMap or the collector arguments +makes a working pattern look ignored. Watch the arriving metrics, or the operator log. + +`names:` is proven separately and in production: the three DCGM metrics have been at the hub since +2026-10-20 through this mechanism, and appear in the collector arguments as +`--match={__name__="DCGM_..."}`. + +## Exclusion + +Prometheus joins several `match[]` selectors with OR, so a selector **added** next to the 230 that +stay can only union with them. Three statements worth keeping apart: + +1. A **single** selector can restrict its own selection with a negative condition. Measured below, + 4038 against 4037. +2. An **added** selector removes nothing that the other selectors already match. This follows from + the OR semantics. +3. **Rebuilding the whole MCO selection** was never tested, and it is a different change from + editing a list. + +An earlier `/federate` observation from 2026-10-19 is not used as evidence: the requests were not +recorded and the route answers 503 for our tokens since 2026-10-22, so it cannot be repeated. Point 2 +rests on the OR semantics. + +For reference, a selector restricting itself is not the hard part. Measured 2026-10-19 through +`/federate` on the hub's own Prometheus, counting distinct `__name__` values in the response: + +| matcher | distinct names | +|---|---| +| `{__name__=~".+"}` | 4038 | +| `{__name__=~".+",__name__!="go_goroutines"}` | 4037 | +| `{namespace="openshift-monitoring"}` | 1303 | + +The single removed name is the whole point of the middle row. This route no longer answers for us, +see above. + +## Why there is no COO MonitoringStack + +Neither a COO limitation nor a cost we decided against. A `MonitoringStack` per cluster with +`remoteWrite` to a hub stack was the **fallback** planned in case no object storage became available +for MCO. S3 did become available, MCO runs, the fallback was never needed. Our COO chart deploys the +operator and the UI plugin, nothing else. + +Building one now would not block MCO, and it would not answer the question this ADR is about: MCO +federates from platform monitoring and UWM and would not read a third Prometheus without repointing +the collector or setting up `remoteWrite`, so the central selection would stay exactly as it is. It +would also add scrape load and storage for data already collected locally. + +## Documentation status + +ACM 2.17 documents the custom allowlist with `names` and `recording_rules`. `matches` appears in the +guide only inside `collect_rules`, for dynamic metrics on single-node OpenShift, and all four +examples there pin `__name__` to one value. + +The hub's own default allowlist nevertheless carries a top-level `matches:` with 13 entries, +including `job=~"apiserver|etcd"` and `container!=""`, and those reach the collector arguments on +every managed cluster. So the key is real and in production use; what is undocumented is the +specific use as a **general name pattern in the custom allowlist**, not `matches` as such. That is +measured implementation behaviour on 2.17.3. How Red Hat would classify a regression there is a +support question nobody has asked. diff --git a/adrs/2026-10-19-mco-metrics-allowlist/querying.md b/adrs/2026-10-19-mco-metrics-allowlist/querying.md new file mode 100644 index 0000000..01f0583 --- /dev/null +++ b/adrs/2026-10-19-mco-metrics-allowlist/querying.md @@ -0,0 +1,126 @@ +# Querying fleet metrics without double counting + +Supporting detail for [the ADR](README.md). A metric name can reach the hub by more than one path, +and more than one scrape of the same target can sit inside a single path, so one subject can produce +several series under the same `cluster` label. All queries below were run against +`rbac-query-proxy` on the dev hub, 2026-10-22. + +## The duplication + +`etcd_server_has_leader` for `oac-dev-workload0`: + +| namespace | container | `managementcluster` | `job` | series | +|---|---|---|---|---| +| `clusters-oac-dev-workload0` | `etcd-metrics` | `local-cluster` | `etcd` | 3 | +| `clusters-oac-dev-workload0` | `etcd-metrics` | `local-cluster` | `etcd-client` | 3 | +| `clusters-oac-dev-workload0` | `haproxy` | absent | `acm-etcd` | 3 | +| `openshift-etcd` | `haproxy` | absent | `etcd` | 3 | + +Twelve series for three etcd members, and **the transport paths explain only part of it**. Two +ServiceMonitors scrape the same pods on the hub, one as `job=etcd` and one as `job=etcd-client`, so +the hub-scraped group alone holds six series for three members. Why both exist was not investigated; +it is a second multiplication sitting inside the first. + +So `count by (cluster)` reports 12 series for workload0's three members and 6 for each of the other +clusters' three. Those series counts are correct; what is wrong is reading them as a number of +members, which overstates by four on workload0 and by two elsewhere. + +The two `haproxy` rows are the HCP forwarder. The second carries `namespace="openshift-etcd"`, which +on a hosted cluster holds no etcd: checked on the cluster, the namespace exists and is empty, no +pods. Forwarding relabels to the control plane's native identity, so the namespace names where such +a metric would live on a standalone cluster, not where it was produced. + +`cluster` says which cluster the data is **about**. The differences shown in `namespace`, `container`, +`managementcluster` and `job` reflect the scrape and forwarding contexts, not different subjects — +while `pod` below is the identity of the observed member. + +## How many names are affected + +Per cluster, counted as **names appearing in multiple `managementcluster` groups**, which is a +candidate list rather than proven duplication: + +| cluster | names in multiple `managementcluster` groups | +|---|---| +| `local-cluster` | 0 | +| `oac-dev-workload0` | 23 | +| `oac-dev-workload1` | 3 | +| `oac-dev-workload2` | 3 | + +workload1 and workload2 show only the generic three: `up`, `grpc_server_started_total` and one +recording rule. For generic names like these the groups may well describe entirely different targets, +so a hit is a reason to look, not a duplicate. workload0 shows twenty more, almost all etcd, and it +is the cluster with forwarding enabled. + +Enabling forwarding elsewhere should be expected to add overlaps, though not necessarily these same +23 names. The set therefore grows with a switch we control, not only with new exporters. + +## Counting members instead of series + +```promql +count by (cluster) (group by (cluster, pod) (etcd_server_has_leader)) +# -> local-cluster 3, workload0 3, workload1 3, workload2 3 +``` + +This answers **how many distinct etcd pods appear in the available samples**, which is usually what +somebody counting etcd wanted. `group by` says that intent plainly; `max by (cluster, pod)` returns the same +numbers here and is the older form of the same query. + +Two things it does **not** do. It is not a health check: the outer `count` counts groups regardless +of their value, so it returns 3 even if all three pods report `etcd_server_has_leader=0`. And it is +not a general recipe for deduplicating metric values — it works because `(cluster, pod)` identifies a +member for this metric, which we checked, and because every path reports the same `etcd-0/1/2`. + +**Do not group by `namespace`, `container` or `job`**: those are exactly the labels that differ per +scrape context, so adding them restores the duplication. + +## Showing scrape and forwarding contexts + +```promql +count by (cluster, namespace, container) (etcd_server_has_leader) +``` + +Suitable for a table or an overview panel, where seeing the separate rows is the point. This grouping +collapses `job=etcd` and `job=etcd-client` into one row; add `job` to see the four rows from the +table at the top. + +## Do not use managementcluster as the separator + +It distinguishes hub-scraped series from the rest, but forwarded and collector-sent series both have +it absent, so it does not separate all three paths. Filtering on it also drops clusters silently, +measured **for `etcd_server_has_leader`** — which clusters a filter returns depends on the metric: + +| filter | clusters returned | +|---|---| +| `managementcluster=""` | `local-cluster`, `workload0` | +| `managementcluster="local-cluster"` | `workload0`, `workload1`, `workload2` | + +Neither returns all four. The hub does have its own control plane, but its etcd metrics do not carry +`managementcluster="local-cluster"` in this measurement — that label marks hosted control planes +scraped on the hub, not the hub's own components. + +`container="haproxy"` identifies the forwarder for control plane metric names, but `haproxy` also +names ordinary router containers — 786 series on workload0 — so use it together with a metric name, +never alone. + +## Searching for possible overlaps + +Per cluster; an unbounded `{__name__=~".+"}` across the fleet times out here. + +```promql +count by (__name__) ( + count by (__name__, managementcluster) ({__name__=~".+", cluster="$cluster"}) +) > 1 +``` + +Grouping on `managementcluster` is deliberate. Adding `namespace` or `container` turns the query into +a list of ordinary multi-pod metrics: that variant returned 35 names for the hub, where this method +finds none. As written it returned 0 for the hub and 23, 3, 3 for the workloads. + +It is a search, not a proof, in both directions. **A hit does not establish a duplicate** — the +groups may describe different targets, which is likely for generic names such as `up`. **No hit does +not rule one out** either, because duplication sitting entirely on one side of the +`managementcluster` split is invisible to it; for that, compare `count by (namespace, container, job)` +for the metric in question. + +Worth running after enabling forwarding on a cluster, or after adding an exporter that uses generic +names such as `up`, `go_*` or `process_*`.