Repository navigation
Conversation
…ancy port Projects share a workload cluster, so the dividing line is the namespace. The console dashboards cannot do that: Perses queries thanos-querier on 9091, which returns every namespace, and the port is not configurable in the operator. So each project gets its own Grafana against port 9092. Grafana itself separates nothing, it only ever sees its own project: the service account's view is bound to the project namespace, the datasource carries a fixed namespace parameter and is not editable, and error-on-replace rejects a query that asks for another one. What is missing is the negative test, project B reachable neither through a built query nor through 9091. Impersonating the service account is forbidden for my account on the workload cluster, so it is in the open points.
The ADR said under "No volume" that a project needing to keep its own dashboards is the point to revisit. That revision exists now as a branch, so this records it rather than opening a second ADR. Editor role and volume come together, alerting is switched off in the config as well so an Editor cannot add a webhook contact point, and everybody passing the proxy is the same anonymous Editor, so changes cannot be attributed. SQLite on the NFS storage class is untested and goes in as an open point.
cadvisor lives in the platform instance rather than in user workload monitoring, so the container panels come from there. Where that instance mounts emptyDir, a project gets a working dashboard that shows the age of the pod, and nothing says so.
Open point six said untested. It was tested on dev-workload0 on 2026-10-20: the database was created, the migrations ran, and after deleting the pod the new one reopened the same file without a stale lock. What is still untested is a real project with editable and somebody saving dashboards. Also points at the new storage ADR, which covers the filesystem these volumes sit on.
69f718f to
fe12bdb
Compare
|
@schwesig note that I have rebased this PR on main (so that the label workflow passses); you will want to update your local working copy before making any changes. |
|
|
||
| **One Grafana instance per project namespace**, with a datasource pointing at | ||
| `https://thanos-querier.openshift-monitoring.svc:9092`, shipped by the `user-projects` chart and | ||
| opt-in per project. Implemented in CCI-MOC/oac-apps#14, merged 2026-09-28. |
There was a problem hiding this comment.
I am unclear why we are using one Grafana per project, rather than one Perses per project. I'm not arguing the decision, I just don't see that decision addressed explicitly.
There was a problem hiding this comment.
There was a problem hiding this comment.
- perses can get rights from kubernetes-rbac
- perses can be inside the openshift console and "outside"
- grafana can NOT be inside the openshift console (except workarounds like iframe etc), but can be "ouside"
- perses is NOT in the open-shift catalogue, but grafana is (see below)
- perses: in console port 9092 (or any other) can not be set/changed, only outside/standalone one
- grafana: no console at all, outside port 9092 can be set in grafana outside (what our user-projects do today)
A standalone perses (alowin other ports than 9091) can be build, but it is not "out of the catalogue".
Therefore the recommended decisison (for now) to go with the "one click/ out of the catalogue" idea.
info checked:
The perses-operator is in none of the catalogs our clusters see.
Measured on the prod hub:
650 PackageManifests across four catalogs,
community-operators314,certified-operators173,redhat-operators151 andredhat-marketplace12,
and not a Perses entry.
grafana-operator is there, in community-operators, channel v5, currently grafana-operator.v5.25.0.
|
We need to assign the appropriate RIT reviewer to this (and all ADRs) in addition to these preliminary reviews. Do not merge. I am discussing the review policy with MSD and RIT now. |
|
Understood, I will hold off on merging.
Just to clarify: my understanding was that introducing ADRs would give us a
simpler process for individual technical decisions, with fewer people
involved than in the RFC process.
If ADRs require the same review process and take just as long, I am
concerned we will not have gained much in that respect.
Since ADR approval is still marked as TBD in the README, could we clarify
the intended scope and review requirements as part of the discussion with
MSD and RIT?
Having the appropriate RIT reviewer makes sense; my question is how we keep
the process proportionate to the decision.
…On Thu, Oct 22, 2026 at 1:50 PM Heidi < ***@***.***> wrote:
*hpdempsey* left a comment (CCI-MOC/ops-docs#37)
<#37 (comment)>
We need to assign the appropriate RIT reviewer to this (and all ADRs) in
addition to these preliminary reviews. Do not merge. I am discussing the
review policy with MSD and RIT now.
—
Reply to this email directly, view it on GitHub
<#37?email_source=notifications&email_token=AVN6SAYA6KYCBHJR535JQ235VHYATA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMZUGAYDEMRVGU42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-6340022559>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AVN6SA5W3VZYRHO2ZDN56DT5VHYATAVCNFSNUABFKJSXA33TNF2G64TZHMZDSNRTGUZDANZXHNEXG43VMU5TKOJSG4YTCMRVGYZKC5QC>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Co-authored-by: Lars Kellogg-Stedman <lars@oddbit.com>
|
Per discussions yesterday, I will submit a PR for the simple review process today. In the meantime, I'm assigning the additional needed reviewers here so they can get started in parallel. In brief for now, we need 1 Red Hat review (already have it), 1 review from @knikolla and one from David King. (See #42 for ADR review process.) |
|
|
||
| ## Context | ||
|
|
||
| Projects on OAC/MOC 2.0 need to see their own metrics and only their own. The obvious way, the console |
There was a problem hiding this comment.
Is it correct to say that projects on OAC/MOC 2.0 need to see their metrics so that they can debug their application performance on their own?
There was a problem hiding this comment.
This is true now for the OAC. I would expect it to also be true for MOC2.0 if we want to allow as much project self-service for debugging cluster issues as possible. Do we expect this to be a requirement for all the projects on the NIST/HIPAA clusters @dsmk? I can imagine that some projects such as Harbor would want it, but maybe they are an exception?
|
This ADR looks good to me. I agree that we should address Lars' comments before merging but the decision makes sense to me for now. |
| - **The CRs live in `user-projects`, not in a chart of their own.** A separate chart would hold a | ||
| second copy of the project list and the two would drift. `user-projects` already owns the | ||
| per-project loop. | ||
| - **A route with an `oauth-proxy` sidecar**, the service account as the OAuth client. It reuses the |
There was a problem hiding this comment.
Can you please elaborate more on this? It is unclear to me what "requsing the same project--edit group" means.
There was a problem hiding this comment.
@knikolla Thanks, it was a little unclear and partly wrong.
- Access hangs on no group.
The sidecar runs a SubjectAccessReview for services get in the project’s namespace:
--openshift-sar for cookies & --openshift-delegate-urls for bearer tokens.
Any group with edit satisfies it (edit contains services get). - project--edit was right until changes in oac-apps 376a542 on 2026-09-10, which made group names explicit.
- Status:
right now they are for example intern-aibom, eldritch-things, project-buit in dev.
project-whiteroom-edit for example: it has both bounds so nobody should loses access.
changes in the ADR text for clarification commited
|
|
||
| **With `editable`, everybody who passes the proxy is the same anonymous Editor.** That includes | ||
| `open-accelerator-admins` through `cluster-reader`, and project members who only have view. Changes | ||
| cannot be attributed to a person. |
There was a problem hiding this comment.
You should be able to get the username of the user from the proxy as a header. https://grafana.com/docs/grafana/latest/setup-grafana/configure-access/configure-authentication/auth-proxy/
There was a problem hiding this comment.
@knikolla
awesome, thanks, I didn't find that.. my bad
... and better than I thought:
the header is already there!
--pass-basic-auth=falseonly drops Authorization: Basic, the one 11.3 mistook for a login.
-pass-user-headers is separate, still true, and carries X-Forwarded-User.- So it is Grafana config only, nothing on the proxy.
I mixed that up... - Tested it in podman against Grafana 11.3:
name in the corner, and the dashboard history says who saved.
That was the part I cared about, it is a "who did what". Security PLUS logs/audit. - One catch:
it also needs[users] auto_assign_org_role,
--> or the proxy user comes out asViewerandeditablegives you an Editor that cannot edit.
Applies when the user is created.
There was a problem hiding this comment.
as talked about in the Meeting yesterday (MOC 2.0 Coordination Meeting v2, Wednesday, November 18), and also in some issues beforehand, the edit function in the grafana GUI will not be enabled, using gitops for that is enough and better trackable/version controlled by default.
Co-authored-by: Lars Kellogg-Stedman <lars@oddbit.com>
Co-authored-by: Lars Kellogg-Stedman <lars@oddbit.com>
Co-authored-by: Lars Kellogg-Stedman <lars@oddbit.com>
The document decides how a project reaches its own metrics and nothing else, which is access control, and it does so through one service account per project that every query carries, which is accountability.
61be542 to
f101390
Compare
The text said cluster wide access, which is not a role. It is cluster-monitoring-view, and a project member holding only view opens the Dashboards tab without getting any metrics, because view grants the Prometheus custom resources and not the Prometheus API.
The text named a project-<name>-edit group, which is neither what the proxy checks nor what the groups are called. Access is a SubjectAccessReview for services get in the project namespace, and the group name comes from the values, not from the chart.
Grafana auth.proxy takes the user from X-Forwarded-User, which the sidecar already sends, and the dashboard history then names who saved. It needs auto_assign_org_role as well, or a proxy user lands on Viewer.
The volume is 50Gi, not 32Gi, and 15 days is the configured ceiling rather than what a project sees. dev-workload0 holds 6.4 days because the cluster was rebuilt, prod-workload0 holds 15.7.
Editing dashboards in the UI was taken back in oac-apps#247, so the open point about attributing a change no longer applies. The decision replaces it in Consequences.
CCI-MOC/oac-apps#230 dropped the grafana-operator from user-projects, so the chart deploys Grafana itself and sets it up from provisioning files at startup. The text still described the operator as the decision and a plain Deployment as the fallback. The argument against shared Grafana organisations stood on the operator having no CRD for them. Provisioning files do carry an orgId, but the organisations themselves still are not provisionable, so the decision holds on a different reason.
Records why each project gets its own Grafana instead of using the console dashboards, and where the separation actually sits.
Short version: projects share a workload cluster, so the line is the namespace.
Perses queries thanos-querier on 9091, which returns every namespace, and the port is not configurable.
Grafana on 9092 does not separate anything either, it only ever sees its own project, because the service account's view is bound to one namespace and the datasource carries a fixed namespace parameter.
One thing is still open and it is in the ADR:
the negative test, that project B is reachable neither through a built query nor through 9091 is not run thoroughly enough as I would like.