Skip to content

adr: Collect central metrics using MultiClusterObservability allowlist keeping exclusion lists as a future change - #36

Open
schwesig wants to merge 8 commits into
CCI-MOC:mainfrom
schwesig:adr-mco-metrics-allowlist
Open

schwesig wants to merge 8 commits into
CCI-MOC:mainfrom
schwesig:adr-mco-metrics-allowlist

Conversation

@schwesig

Copy link
Copy Markdown

Records why we collect central metrics by allowlist, and what we measured when we asked whether a block list would work instead.

Short version:
MCO does accept a broad matcher, but Prometheus joins the match[] selectors with OR, so a negative one next to the existing 227 excludes nothing.
The docs cover removing single metrics by name and say nothing about patterns.
So we keep the model, not the contents, and write the measurements down for whoever picks this up again.

@schwesig schwesig self-assigned this Oct 19, 2026
@schwesig schwesig added the documentation Improvements or additions to documentation label Oct 19, 2026
@larsks
larsks force-pushed the adr-mco-metrics-allowlist branch from b7231f5 to 57c6fb1 Compare October 22, 2026 01:25
@github-actions github-actions Bot added the ops-adr This is an arthiectural decision record label Oct 22, 2026
@larsks

larsks commented Oct 22, 2026

Copy link
Copy Markdown
Member

@schwesig note that I have rebased this PR on main (so that the label workflow passses); you will want to update your local working copy before making any changes.

@hpdempsey

Copy link
Copy Markdown

Not all required reviewers assigned yet. Do not merge.

@schwesig
schwesig force-pushed the adr-mco-metrics-allowlist branch 2 times, most recently from b13b746 to 948343b Compare October 22, 2026 18:36
@schwesig
schwesig requested a review from larsks October 22, 2026 18:40
@hpdempsey
hpdempsey requested a review from dsmk October 23, 2026 16:36
@hpdempsey

Copy link
Copy Markdown

Adding required reviewers. Need three approvals for merge: David, Kristi and one Red Hatter who is not the author (already done).

@dsmk

dsmk commented Oct 23, 2026

Copy link
Copy Markdown

Looks decision looks reasonable on the first read, but I'll take a deeper look later.

Please define acronyms when they are first introduced in the ADR is the one suggestion based on the first pass.

@larsks

larsks commented Oct 23, 2026

Copy link
Copy Markdown
Member

You have flushed out a problem with our pre-commit configuration. Thank you! I've proposed a fix in #45.

Comment on lines +109 to +111
- **Our use of it is not documented.** A general name pattern in the custom allowlist works on ACM
2.17.3, and the hub's own default list
uses it, but the guide documents only `names` and `recording_rules`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What exactly is undocumented -- our use of matches:?

@schwesig schwesig Oct 27, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@larsks yes, the matches: field with a name regex.

  • matches: is real and in production, the hub's own default allowlist carries 13 entries
  • but the guide shows it only inside collect_rules, for dynamic SNO (Single Node OpenShift) metrics, with name pinned to one value
  • for the custom allowlist it documents names and recording_rules only
  • what I could not find anywhere is a general name pattern like name=~"ovs_.*" in it

--> so it is measured behaviour, not a described one

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see commit, I changed it, hopefully better described...

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +150 to +161
### Why the gap is maintenance, not technology

Editing the selection is routine work and does not reopen this decision. CCI-MOC/oac-apps#140 is
exactly that: three names added, documented mechanism, no change of model. That holds with `names:`
alone, which is what the argument rests on.

A pattern makes a bulk addition more convenient — `matches: [__name__=~"ovs_.*"]` took `ovs_*` from 0
to 86 names across all three clusters, see [measurements.md](measurements.md) — but it is a
convenience on top, not the reason the model is affordable.

What stays true is that a metric **no existing selector matches** does not reach the hub, and
nothing reports that it is missing. A pattern shifts where that line sits, it does not remove it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not entirely sure what this section is trying to convey. Most of the points here have been made previously.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@larsks kind of, yes. I need to rewrite that and think, what was really the reason why I repeated that..

I tried to sum up and got lost.
What I meant:

  • if wildcard/matches works... great, saves time and manual work
  • if only names works (e.g. Red Hat does not support it anymore), then we just need to add e.g 86 name lines (instead of 1 wildcard)

so nothing depends on wildcard/matches --> not a K.O. criteria here

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remove that chapter...
good point.

@dsmk dsmk left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision seems reasonable to me - use the central logging approach from MOC 1.0 in a way that we can enhance/shift later. That way we can focus on other things in the short term.

I am approving this ADR with the provision that the documentation updates Lars and I added are addressed.

@schwesig schwesig changed the title adr: Keep the MCO allowlist for now, keep exclusion open as an option adr: Collect central metrics using MultiClusterObservability allowlist keeping exclusion lists as a future change Oct 27, 2026
@schwesig
schwesig force-pushed the adr-mco-metrics-allowlist branch from fecd698 to f7cd4db Compare October 27, 2026 19:16
@schwesig
schwesig force-pushed the adr-mco-metrics-allowlist branch from f7cd4db to 528d63e Compare October 27, 2026 19:21
@schwesig
schwesig requested a review from larsks October 27, 2026 19:22
schwesig and others added 6 commits November 3, 2026 14:43
MCO collects by allowlist, 227 pinned --match arguments against the local Prometheus. We run it
that way becuase the goal was to stay close to MOC 1.0, not after comparing it with the
alternative.

Measured on the dev hub: MCO does pass a non-pinned matcher through, but Prometheus joins several
match[] selectors with OR, so a negative matcher next to the existing 227 excludes nothing. The
docs cover removing single metrics by name and say nothing about patterns.

So we keep the allowlist as the model, not its contents, and write down what the measurements
showed for whoever revisits this.
Co-authored-by: Lars Kellogg-Stedman <lars@oddbit.com>
larsks asked which path the metrics take, and the question was right: the central figure described one path and was reported as the whole.
Corrected, together with his other three comments.

The decision now rests on the documented mechanism alone, a name pattern is recorded as a convenience rather than as its justification, and the three selections that were conflated are told apart: the control plane drop list, local collection in UWM, and what reaches the hub.

Measurement detail and the query recipes move to measurements.md and querying.md next to the ADR.
That keeps the ADR readable without losing what somebody debugging a metric needs, and it is a new convention here, so say if you would rather have it in one file.
Retitles the ADR, tightens the context paragraph and folds a few asides
into the sentences they belong to. No change of substance; this replaces
seven consecutive web editor saves from the same sitting.
The bullet said "our use of it" and left the reader to work out what "it"
pointed at. The key itself is documented and the hub's default list uses
it; what the guide does not show is a general name pattern in the custom
allowlist.
All three paragraphs said something the ADR had already said: oac-apps#140
as a routine addition appears four times before this, the ovs_* measurement
twice, and the quiet gap when no selector matches twice.

What only this section carried, that the decision rests on names: alone and
a pattern is convenience, is already the closing paragraph of "Caveat on
the pattern", stated better there.
@schwesig
schwesig force-pushed the adr-mco-metrics-allowlist branch from 528d63e to 0e5aac6 Compare November 3, 2026 13:43
@hpdempsey

Copy link
Copy Markdown

This requires a review from a Mass Open Cloud person. @naved001 please complete this, as you are the MOC person most familiar with metrics collection in general.

@naved001 naved001 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me, just left some nit picky comments that you could take it or leave it.

@knikolla

Copy link
Copy Markdown

I delegated the approval on behalf of the MOC to @naved001 who approved it in this review.

The sentence about a metric no selector matches read as if it were
kept. And the ownership paragraph said the same thing as the one in
the context section.
A blank line between the table and the list below it, and two places
with a doubled blank line.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ops-adr This is an arthiectural decision record

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants