Skip to content

adr: Separate project metrics with one Grafana per project on the tenancy port - #37

Open
schwesig wants to merge 15 commits into
CCI-MOC:mainfrom
schwesig:adr-per-project-metric-separation
Open

schwesig wants to merge 15 commits into
CCI-MOC:mainfrom
schwesig:adr-per-project-metric-separation

Conversation

@schwesig

Copy link
Copy Markdown

Records why each project gets its own Grafana instead of using the console dashboards, and where the separation actually sits.

Short version: projects share a workload cluster, so the line is the namespace.
Perses queries thanos-querier on 9091, which returns every namespace, and the port is not configurable.
Grafana on 9092 does not separate anything either, it only ever sees its own project, because the service account's view is bound to one namespace and the datasource carries a fixed namespace parameter.

One thing is still open and it is in the ADR:
the negative test, that project B is reachable neither through a built query nor through 9091 is not run thoroughly enough as I would like.

@schwesig schwesig self-assigned this Oct 19, 2026
@schwesig schwesig added the documentation Improvements or additions to documentation label Oct 19, 2026
…ancy port

Projects share a workload cluster, so the dividing line is the namespace. The console dashboards
cannot do that: Perses queries thanos-querier on 9091, which returns every namespace, and the port
is not configurable in the operator.

So each project gets its own Grafana against port 9092. Grafana itself separates nothing, it only
ever sees its own project: the service account's view is bound to the project namespace, the
datasource carries a fixed namespace parameter and is not editable, and error-on-replace rejects a
query that asks for another one.

What is missing is the negative test, project B reachable neither through a built query nor
through 9091. Impersonating the service account is forbidden for my account on the workload
cluster, so it is in the open points.
The ADR said under "No volume" that a project needing to keep its own dashboards is the point to
revisit. That revision exists now as a branch, so this records it rather than opening a second ADR.

Editor role and volume come together, alerting is switched off in the config as well so an Editor
cannot add a webhook contact point, and everybody passing the proxy is the same anonymous Editor,
so changes cannot be attributed. SQLite on the NFS storage class is untested and goes in as an open
point.
cadvisor lives in the platform instance rather than in user workload monitoring, so the container panels come from there.
Where that instance mounts emptyDir, a project gets a working dashboard that shows the age of the pod, and nothing says so.
Open point six said untested. It was tested on dev-workload0 on 2026-10-20: the database was created, the migrations ran, and after deleting the pod the new one reopened the same file without a stale lock.
What is still untested is a real project with editable and somebody saving dashboards.
Also points at the new storage ADR, which covers the filesystem these volumes sit on.
@larsks
larsks force-pushed the adr-per-project-metric-separation branch from 69f718f to fe12bdb Compare October 22, 2026 01:26
@larsks

larsks commented Oct 22, 2026

Copy link
Copy Markdown
Member

@schwesig note that I have rebased this PR on main (so that the label workflow passses); you will want to update your local working copy before making any changes.


**One Grafana instance per project namespace**, with a datasource pointing at
`https://thanos-querier.openshift-monitoring.svc:9092`, shipped by the `user-projects` chart and
opt-in per project. Implemented in CCI-MOC/oac-apps#14, merged 2026-09-28.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am unclear why we are using one Grafana per project, rather than one Perses per project. I'm not arguing the decision, I just don't see that decision addressed explicitly.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. perses can get rights from kubernetes-rbac
  2. perses can be inside the openshift console and "outside"
  3. grafana can NOT be inside the openshift console (except workarounds like iframe etc), but can be "ouside"
  4. perses is NOT in the open-shift catalogue, but grafana is (see below)
  5. perses: in console port 9092 (or any other) can not be set/changed, only outside/standalone one
  6. grafana: no console at all, outside port 9092 can be set in grafana outside (what our user-projects do today)

A standalone perses (alowin other ports than 9091) can be build, but it is not "out of the catalogue".
Therefore the recommended decisison (for now) to go with the "one click/ out of the catalogue" idea.


info checked:
The perses-operator is in none of the catalogs our clusters see.
Measured on the prod hub:
650 PackageManifests across four catalogs,

  • community-operators 314,
  • certified-operators 173,
  • redhat-operators 151 and
  • redhat-marketplace 12,

and not a Perses entry.

grafana-operator is there, in community-operators, channel v5, currently grafana-operator.v5.25.0.

@hpdempsey

Copy link
Copy Markdown

We need to assign the appropriate RIT reviewer to this (and all ADRs) in addition to these preliminary reviews. Do not merge. I am discussing the review policy with MSD and RIT now.

@schwesig

schwesig commented Oct 22, 2026 via email •

Copy link
Copy Markdown
Author

Co-authored-by: Lars Kellogg-Stedman <lars@oddbit.com>
@hpdempsey

hpdempsey commented Oct 23, 2026 •

Copy link
Copy Markdown

Per discussions yesterday, I will submit a PR for the simple review process today. In the meantime, I'm assigning the additional needed reviewers here so they can get started in parallel. In brief for now, we need 1 Red Hat review (already have it), 1 review from @knikolla and one from David King. (See #42 for ADR review process.)

@hpdempsey
hpdempsey requested a review from dsmk October 23, 2026 16:27

## Context

Projects on OAC/MOC 2.0 need to see their own metrics and only their own. The obvious way, the console

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it correct to say that projects on OAC/MOC 2.0 need to see their metrics so that they can debug their application performance on their own?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true now for the OAC. I would expect it to also be true for MOC2.0 if we want to allow as much project self-service for debugging cluster issues as possible. Do we expect this to be a requirement for all the projects on the NIST/HIPAA clusters @dsmk? I can imagine that some projects such as Harbor would want it, but maybe they are an exception?

@dsmk

dsmk commented Oct 23, 2026

Copy link
Copy Markdown

This ADR looks good to me. I agree that we should address Lars' comments before merging but the decision makes sense to me for now.

- **The CRs live in `user-projects`, not in a chart of their own.** A separate chart would hold a
second copy of the project list and the two would drift. `user-projects` already owns the
per-project loop.
- **A route with an `oauth-proxy` sidecar**, the service account as the OAuth client. It reuses the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please elaborate more on this? It is unclear to me what "requsing the same project--edit group" means.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@knikolla Thanks, it was a little unclear and partly wrong.

  • Access hangs on no group.
    The sidecar runs a SubjectAccessReview for services get in the project’s namespace:
    --openshift-sar for cookies & --openshift-delegate-urls for bearer tokens.
    Any group with edit satisfies it (edit contains services get).
  • project--edit was right until changes in oac-apps 376a542 on 2026-09-10, which made group names explicit.
  • Status:
    right now they are for example intern-aibom, eldritch-things, project-buit in dev.
    project-whiteroom-edit for example: it has both bounds so nobody should loses access.

changes in the ADR text for clarification commited


**With `editable`, everybody who passes the proxy is the same anonymous Editor.** That includes
`open-accelerator-admins` through `cluster-reader`, and project members who only have view. Changes
cannot be attributed to a person.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to get the username of the user from the proxy as a header. https://grafana.com/docs/grafana/latest/setup-grafana/configure-access/configure-authentication/auth-proxy/

@schwesig schwesig Nov 3, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@knikolla
awesome, thanks, I didn't find that.. my bad

... and better than I thought:
the header is already there!

  • --pass-basic-auth=false only drops Authorization: Basic, the one 11.3 mistook for a login.
    -pass-user-headers is separate, still true, and carries X-Forwarded-User.
  • So it is Grafana config only, nothing on the proxy.
    I mixed that up...
  • Tested it in podman against Grafana 11.3:
    name in the corner, and the dashboard history says who saved.
    That was the part I cared about, it is a "who did what". Security PLUS logs/audit.
  • One catch:
    it also needs [users] auto_assign_org_role,
    --> or the proxy user comes out as Viewer and editable gives you an Editor that cannot edit.
    Applies when the user is created.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is in the ADR now, point 9.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as talked about in the Meeting yesterday (MOC 2.0 Coordination Meeting v2, Wednesday, November 18), and also in some issues beforehand, the edit function in the grafana GUI will not be enabled, using gitops for that is enough and better trackable/version controlled by default.

schwesig and others added 4 commits October 27, 2026 19:26
Co-authored-by: Lars Kellogg-Stedman <lars@oddbit.com>
Co-authored-by: Lars Kellogg-Stedman <lars@oddbit.com>
Co-authored-by: Lars Kellogg-Stedman <lars@oddbit.com>
The document decides how a project reaches its own metrics and nothing
else, which is access control, and it does so through one service
account per project that every query carries, which is accountability.
@schwesig
schwesig force-pushed the adr-per-project-metric-separation branch from 61be542 to f101390 Compare November 3, 2026 14:33
The text said cluster wide access, which is not a role. It is
cluster-monitoring-view, and a project member holding only view opens
the Dashboards tab without getting any metrics, because view grants the
Prometheus custom resources and not the Prometheus API.
The text named a project-<name>-edit group, which is neither what the
proxy checks nor what the groups are called. Access is a
SubjectAccessReview for services get in the project namespace, and the
group name comes from the values, not from the chart.
Grafana auth.proxy takes the user from X-Forwarded-User, which the
sidecar already sends, and the dashboard history then names who saved.
It needs auto_assign_org_role as well, or a proxy user lands on Viewer.
The volume is 50Gi, not 32Gi, and 15 days is the configured ceiling
rather than what a project sees. dev-workload0 holds 6.4 days because
the cluster was rebuilt, prod-workload0 holds 15.7.
@hpdempsey

Copy link
Copy Markdown

@larsks or @knikolla have your comments been adequately addressed? If so, please approve. @dsmk approval is pending feedback on @larsks comments.

Editing dashboards in the UI was taken back in oac-apps#247, so the
open point about attributing a change no longer applies. The decision
replaces it in Consequences.
CCI-MOC/oac-apps#230 dropped the grafana-operator from user-projects, so
the chart deploys Grafana itself and sets it up from provisioning files
at startup. The text still described the operator as the decision and a
plain Deployment as the fallback.

The argument against shared Grafana organisations stood on the operator
having no CRD for them. Provisioning files do carry an orgId, but the
organisations themselves still are not provisionable, so the decision
holds on a different reason.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ops-adr This is an arthiectural decision record

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants