Repository navigation
adr: Separate project metrics with one Grafana per project on the tenancy port #37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
226c1e5
645af19
3a6fbf3
fe12bdb
6b844bc
230c2ef
9658bef
16f41b5
f101390
3b38466
a61cdcd
07a8a99
69bf5d0
370b5e6
9deb73a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,312 @@ | ||
| --- | ||
| title: Separate project metrics with one Grafana per project | ||
| standards: | ||
| - nist-ac | ||
| - nist-au | ||
| --- | ||
|
|
||
| ## Context | ||
|
|
||
| Projects on OAC/MOC 2.0 need to see their own metrics and only their own. The obvious way, the console | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it correct to say that projects on OAC/MOC 2.0 need to see their metrics so that they can debug their application performance on their own? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is true now for the OAC. I would expect it to also be true for MOC2.0 if we want to allow as much project self-service for debugging cluster issues as possible. Do we expect this to be a requirement for all the projects on the NIST/HIPAA clusters @dsmk? I can imagine that some projects such as Harbor would want it, but maybe they are an exception? |
||
| dashboards that come built in, does not give us that. The reason is specific enough to write down. | ||
|
|
||
| `thanos-querier` listens on two ports, and they behave differently: | ||
|
|
||
| | Port | Name | Behaviour | | ||
| |---|---|---| | ||
| | 9091 | `web` | **Does not filter.** Measured: 19 namespaces returned, with and without a `namespace` parameter, identical result | | ||
| | 9092 | `tenancy` | **Filters**, and a query cannot override the namespace | | ||
|
|
||
| Port 9092 is a chain of two proxies in front of the querier: | ||
|
|
||
| ``` | ||
| 9092 (TLS) -> kube-rbac-proxy -> 127.0.0.1:9095 -> prom-label-proxy -> 127.0.0.1:9090 (thanos-query) | ||
| | | | ||
| checks the token's k8s RBAC enforces --label=namespace | ||
| ``` | ||
|
|
||
| Read off the running pod on `oac-prod-workload0`: `kube-rbac-proxy` runs with | ||
| `--secure-listen-address=0.0.0.0:9092` and an allow-list of query paths, `prom-label-proxy` with | ||
| `--label=namespace` and `--error-on-replace`. | ||
|
|
||
| `--error-on-replace` matters, but on its own it is not the authorisation. It makes the proxy reject | ||
| a conflicting selector instead of rewriting it. Which namespace a caller may ask for is decided | ||
| before that, by the identity in the request, and prom-label-proxy says itself that it does no | ||
| authentication or authorisation. | ||
|
|
||
| So the full chain in our setup, read off the chart and the cluster: | ||
|
|
||
| | | | | ||
| |---|---| | ||
| | identity | `grafana-sa`, one service account per project namespace | | ||
| | its rights | `RoleBinding grafana-view` → ClusterRole `view`, **bound to the project namespace only** | | ||
| | cluster-wide | one ClusterRoleBinding, `system:auth-delegator`, for the oauth-proxy's SAR path, not for queries | | ||
| | the namespace asked for | `customQueryParameters: namespace=project-<name>`, set in the datasource | | ||
| | why it cannot be changed | the datasource is `editable: false`, and `--error-on-replace` rejects a query that carries a different namespace | | ||
|
|
||
| The guarantee rests on all five, not on the port alone. | ||
|
|
||
| ### Why not Perses | ||
|
|
||
| The Cluster Observability Operator (COO)'s UI plugin gives the console a dashboard page built on Perses, and that was the first plan. | ||
| It cannot separate projects, and not as a configuration gap we could close. | ||
|
|
||
| In COO 1.5.3, the version installed here, the Perses datasource is created pointing at port | ||
| **9091**, in `pkg/controllers/uiplugin/prometheus-global-datasource.go:37`, and the CRD field | ||
| `spec.monitoring.perses` has one key, `enabled`. So there is nothing to point elsewhere. Perses | ||
| does have namespace-scoped resources and RBAC for dashboards; what it does not have here is a | ||
| datasource that filters the metrics it queries. | ||
|
|
||
| We measured 9091 instead of trusting what we read: with our own token it returned all 19 | ||
| namespaces, with and without a `namespace` parameter. That is one token and one query, so it shows | ||
| the port does not filter, not what every Perses user sees. For an operator identity that is fine, | ||
| for a project it would be wrong. | ||
|
|
||
| This was decided the other way round once, on 2026-09-02, with the reasoning that "Perses gets | ||
| tenancy from the namespace mapping for free". **That reasoning was wrong**, Perses has no namespace | ||
| mapping, and the decision was reopened and re-decided on 2026-09-08. | ||
|
|
||
| ### Perses stays | ||
|
|
||
| The paragraph above reads like a verdict on Perses, and it is not. Perses is enabled in the console | ||
| on our clusters and stays there. Three reasons, only the first is about tenancy: | ||
|
|
||
| 1. **For the fleet view it is the right tool.** Seeing everything is the job of an operator | ||
|
knikolla marked this conversation as resolved.
Resolved
|
||
| dashboard, and that 9091 returns all namespaces is correct for this reader. The same property | ||
| that makes it unusable for a project makes it usefull here. | ||
| 2. **Two ways in, for two kinds of user.** Somebody holding `cluster-monitoring-view` gets | ||
| dashboards in the console that is open anyway, with nothing to deploy. A project gets its own | ||
| Grafana. To give people the choice is worth something by itself, and neither of the two has to | ||
| serve both audiences. | ||
| 3. **It is already integrated.** `openshift/monitoring-plugin` depends on `@perses-dev/*` and | ||
| renders the Dashboards tab with it, while Metrics and Alerting use Victory. So the console ships | ||
| a dashboard surface either way, and enabling the UI plugin only exposes the tab. | ||
|
|
||
| So this ADR is not "Grafana instead of Perses". It is both, for different views. | ||
|
|
||
| The role is `cluster-monitoring-view`, and a project's own `view` is not enough for it. `view` | ||
| grants `prometheuses` in the API group `monitoring.rhobs`, which are the custom resources, not | ||
| `prometheuses/api` in `monitoring.coreos.com`, which is the data. So a project member can open the | ||
| Dashboards tab and gets no metrics out of it. Measured with SubjectAccessReviews on the prod hub: | ||
| true for a member of `open-accelerator-admins`, which holds the role through | ||
| `open-accelerator-admins-cluster-monitoring-view`, false for an authenticated user outside it. The | ||
| role is scoped with `resourceNames: ["k8s"]`, so a review that leaves the name out comes back false | ||
| as well. | ||
|
|
||
| ## Decision | ||
|
|
||
| **One Grafana instance per project namespace**, with a datasource pointing at | ||
| `https://thanos-querier.openshift-monitoring.svc:9092`, shipped by the `user-projects` chart and | ||
| opt-in per project. Implemented in CCI-MOC/oac-apps#14, merged 2026-09-28. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am unclear why we are using one Grafana per project, rather than one Perses per project. I'm not arguing the decision, I just don't see that decision addressed explicitly.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
A standalone perses (alowin other ports than 9091) can be build, but it is not "out of the catalogue". info checked:
and not a Perses entry.
|
||
|
|
||
| Worth being explicit where the separation happens, because it is easy to read this the wrong | ||
| way: **Grafana does not separate anything here.** It never sees more than its own project. The | ||
| instance is unrestricted inside, one organisation, one datasource, and the limit sits in front of | ||
| it in the proxy chain on 9092. We did not configure Grafana to filter by namespace, we gave it a | ||
| datasource that cannot return anything else. | ||
|
|
||
| The supporting choices, each of which could be made differently: | ||
|
|
||
| - **Per project, not one shared Grafana with orgs.** Separation is then the cluster's, enforced by | ||
| RBAC and `--error-on-replace`, rather than Grafana's own org model. Everything else in this repo | ||
| separates by Kubernetes RBAC, so this keeps one mental model instead of two. | ||
| - **The CRs live in `user-projects`, not in a chart of their own.** A separate chart would hold a | ||
| second copy of the project list and the two would drift. `user-projects` already owns the | ||
| per-project loop. | ||
| - **A route with an `oauth-proxy` sidecar**, the service account as the OAuth client. Access is a | ||
| SubjectAccessReview for `services get` in the project's namespace, through `--openshift-sar` for | ||
| the cookie path and `--openshift-delegate-urls` for the bearer-token one. So whoever already has | ||
| `edit` in the project reaches the Grafana, no Keycloak client per project, and route access and | ||
| metric access test the same RBAC. Which group grants that `edit` is up to the project's values, | ||
| the chart does not fix a name. | ||
| - **Opt-in, off by default.** A project without a `grafana:` block renders exactly as before, no | ||
| pod and no quota used. Adding the template to the chart must not hand six existing projects a | ||
| Grafana they never asked for. | ||
| - **Dashboards are chart resources, not hand-built per project.** `project-overview` is one | ||
| provisioning file in `user-projects`, rendered into each project namespace. Because the | ||
| datasource behind it is scoped by the proxy, the same dashboard definition shows a different | ||
| project its own data without being edited. Write a panel once, every project that opts in gets it, | ||
| and nobody has to maintain six copies that drift. | ||
| - **No volume.** "No PVC" sounds like the metrics are not kept. They are, just not here: the | ||
| platform Prometheus on each workload cluster is configured for 15 days on a 50Gi volume, UWM | ||
| another 20Gi, and Grafana queries that through the tenancy port. | ||
|
|
||
| Grafana does keep state of its own, in a database: users, datasources, dashboards, preferences. | ||
| We run it without a volume anyway, which means everything we need is read from the provisioning | ||
| files at startup, and anything else is allowed to be lost. **That has a consequence for | ||
| projects: a dashboard built in the UI does not survive.** Dashboards belong in git. If a project | ||
| needs to keep its own, it adds them to the chart. An opt-in for editing in the UI existed for a | ||
| while and was dropped again in CCI-MOC/oac-apps#247. | ||
| - **No alerting in Grafana**, a separate decision from the volume. Alerting belongs to the | ||
| platform: Prometheus evaluates `PrometheusRule`, Alertmanager routes what fires, both already | ||
| running and both tenant-aware. Grafana alerting would be a second path next to the one OpenShift | ||
| ships, so it is switched off in the config (`unified_alerting.enabled: "false"`). | ||
| - **No operator.** The chart deploys Grafana itself and sets it up from provisioning files at | ||
| startup. `grafana-operator` v5 ran it until CCI-MOC/oac-apps#230, but it only ships in | ||
| `community-operators`, it is not flagged FIPS-compliant, and it pushed each project's datasource | ||
| token into Grafana's admin API over plain HTTP. Grafana listens on `127.0.0.1` only, so the | ||
| oauth-proxy in the same pod is the only way in. | ||
|
|
||
| ### What we did not pick | ||
|
|
||
| #### One shared Grafana with Grafana organisations | ||
|
|
||
| Cheaper, one pod and one URL for everybody. Organisations are core Grafana and | ||
| work in the community edition, no Enterprise licence needed: dashboards, | ||
| folders, datasources, alerts and service accounts separate per organisation, | ||
| and `org_mapping` in Generic OAuth puts a Keycloak group into the right one. | ||
| Team Sync and per-datasource permissions are Enterprise, neither is needed | ||
| here. | ||
|
|
||
| Two reasons against: | ||
|
|
||
| 1. Grafana would hold the credentials of several projects at once, so their | ||
| correct assignment inside Grafana becomes security relevant. A misconfigured | ||
| organisation or a shared datasource lets one project see another. | ||
|
|
||
| 2. Provisioning files carry an `orgId`, but the organisations themselves are | ||
| not provisionable. They would have to be created through the Grafana API or | ||
| a job of our own, outside the files the chart writes, and the datasource of | ||
| every project would move into one pod. | ||
|
|
||
| If project count makes the per-project cost hurt, this is still the option to | ||
| come back to, with that gap as part of the price. | ||
|
|
||
| #### Per-cluster separation instead of per-namespace | ||
|
|
||
| Simpler, but it does not match OAC/MOC 2.0: the projects share | ||
| `oac-prod-workload0`, so the dividing line is the namespace. Not really a | ||
| choice, it is what the environment is. | ||
|
|
||
| #### AI Telemetry with the Prometheus Keycloak Proxy | ||
|
|
||
| This is how MOC 1.0 solved this using solutions developed by Chris Tate. It | ||
| separates more than we do and one level higher: hubs, tenants, clusters and | ||
| projects, Keycloak groups created per hub and per tenant, one proxy per hub | ||
| sharing a realm with the application. For a provider with several hubs and real | ||
| tenants that is the stronger model, and it ran in production. | ||
|
|
||
| The reason against is ownership, not technique. As far as I know it is no | ||
| official Red Hat feature or product, so support and maintenance would sit with | ||
| our team. For a component between projects and their data, carrying an | ||
| authorisation decision, that is a commitment to make deliberately. The | ||
| per-project Grafana leans on supported pieces instead: the OpenShift proxy | ||
| chain, the oauth-proxy sidecar, the cluster's RBAC. | ||
|
|
||
| If OAC/MOC 2.0 grows several hubs or needs tenants above the project level, | ||
| this is the design to look at again, with the support question answered first. | ||
|
|
||
| ## Consequences | ||
|
|
||
| **The separation is enforced outside Grafana**, by the proxy and by RBAC, not by the application. | ||
| A dashboard with the wrong query does not leak across projects, because the proxy rejects it rather | ||
| than Grafana deciding to be careful. | ||
|
|
||
| Verified: the datasource points at 9092, it is `editable: false`, the service account's `view` is | ||
| bound to its own namespace, and only `auth-delegator` is cluster-wide. **Not** verified: that | ||
| `grafana-sa` gets nothing useful out of 9091, and that a hand-made query for another namespace is | ||
| refused. Both need impersonating the service account, which this account may not do on the workload | ||
| cluster. So this is a well-founded expectation, not a proven one. | ||
|
|
||
| **The cost is one Grafana pod plus one `oauth-proxy` sidecar per participating project**, against | ||
| that project's own quota. In numbers, from the chart: | ||
|
|
||
| | | requests | limits | | ||
| | --------------------- | ------------------- | ------------------- | | ||
| | Grafana | 100m CPU, 256Mi | 500m CPU, 512Mi | | ||
| | `oauth-proxy` sidecar | 50m CPU, 64Mi | 200m CPU, 256Mi | | ||
| | **per project** | **150m CPU, 320Mi** | **700m CPU, 768Mi** | | ||
|
|
||
| Deliberately small, and the sidecar needs its own entry because a container without resources picks | ||
| up the project `LimitRange` default of 1 CPU and 2Gi. No storage, see above. | ||
|
|
||
| If project count grows faster than expected, the shared-Grafana model comes back onto the table, | ||
| and with it the separation moving into the application. | ||
|
|
||
| **Two open questions nobody decided**, and they are not the same one. The pods sit in the project | ||
| namespace and consume from its quota, and what exactly they consume depends on whether the quota | ||
| caps requests, limits or both. | ||
|
|
||
| 1. **Is the quota raised when Grafana is enabled?** If not, a project trades 150m CPU and 320Mi of | ||
| its own workload for the dashboard. The quota does not shrink, its remaining capacity does. | ||
| 2. **Are those resources billed to the project?** Separate question: the pod consumes quota either | ||
| way, billing only decides who pays for it. | ||
|
|
||
| For raising the quota: observing your own workload is part of running on the platform, and making | ||
| people pay for it discourages the thing we want. Against billing it: 150m CPU is below the noise of | ||
| any billing unit. Both are policy, not technical, and better decided before the first project asks. | ||
|
|
||
| **It means also one URL per project**, every Grafana gets its own route in its own namespace. | ||
| There is no single address today where we can send everybody, so somebody has to tell a project | ||
| where its dashboard lifes. A landing page or a redirect could give one entry point without merging | ||
| the instances, we just have not built it. | ||
|
|
||
| **The upside sits on the same mechanism.** A dashboard written once in the chart works for every | ||
| project, because the scoping happens in the datasource and not in the dashboard. Adding a panel is | ||
| one change in one file, and a new project that opts in gets the current set with nothing to copy. | ||
|
|
||
| **We run two dashboard tools for two jobs.** Perses in the console on 9091 for the cluster-wide | ||
| view, Grafana per project on 9092. Intended, and still real operational work: two things to upgrade, | ||
| two things to explane. One correction to how I put it earlier: 9091 on a workload cluster shows that | ||
| cluster, not the fleet. The fleet view is the hub's. | ||
|
|
||
| **Dashboards are read only in the GUI.** They come from git, and an anonymous Editor could not be | ||
| held to a change, so editing was dropped in CCI-MOC/oac-apps#247. | ||
|
|
||
| **We maintain the Deployment ourselves** instead of letting an operator do it. In exchange nothing a | ||
| project uses depends on a community build. | ||
|
|
||
| **A project dashboard is only as good as the platform Prometheus under it**, and that is not part of | ||
| this chart. `cadvisor` lives in the platform instance, not in user workload monitoring, so the | ||
| container metrics the `project-overview` panels draw come from there. On a cluster where the platform | ||
| Prometheus mounts `emptyDir`, a project gets a dashboard that works and shows the age of the pod. | ||
| Nothing in Grafana reports this, and Argo is green either way. | ||
|
|
||
| That is a per-cluster property, so it has to be checked per cluster rather than assumed from this | ||
| decision. The three workload clusters are configured alike, `retention: 15d`, `retentionSize: 40GiB` | ||
| on a 50Gi volume, but what a project actually sees is the age of the volume. Measured 2026-11-03: | ||
| prod-workload0 holds 15.7 days, dev-workload0 holds 6.4, because that cluster was rebuilt a week | ||
| ago. The configuration is the ceiling, not the promise. | ||
|
|
||
| ## Open points | ||
|
|
||
| 1. **Nothing is built for a tenancy demo yet.** Showing two projects that cannot see each other is | ||
| the thing a reviewer will ask for, and it does not exist. | ||
| 2. **The upstream gap in Perses is unreported.** An issue making the datasource port configurable | ||
| was drafted and never opened. If that lands, the per-project Grafana becomes optional rather than | ||
| necessary, which would be a simplification worth having. | ||
| 3. **Per-project resource limits are small on purpose** and have not been tested under real | ||
| dashboard use. | ||
| 4. **The quota question above needs an owner.** It is not blocking anything today, because few | ||
| projects have Grafana enabled, and it becomes awkward the first time somebody notices. | ||
| 5. **Whether AI Telemetry is supportable** was never actually asked, I am going on what I know. If | ||
| somebody checks and it turns out there is a support path, the alternative above deserves a real | ||
| comparison rather than the short treatment it gets here. | ||
| 6. **SQLite on `pure-fb-nfsv4` works, but not under a real project yet.** The concern was file | ||
| locking, SQLite being known to be fragile on NFS. Tested on dev-workload0 on 2026-10-20 with | ||
| `grafana/grafana:11.3.0`: the export mounts `nfsvers=4.1` with no `nolock`, the pod reports | ||
| `local_lock=none`, the database was created and the migrations ran, and after deleting the pod the | ||
| new one reopened the same file and logged `migrations completed, performed=602, skipped=6`. No | ||
| stale lock. Nothing writes to that database any more since editing was dropped, so this stays a | ||
| measurement rather than something we depend on. | ||
| 7. **Nothing warns when the platform Prometheus below a project has no volume.** The dashboard looks | ||
| healthy and the panels are simply short. A check on the cluster, or a panel that shows the oldest | ||
| sample it can see, would make it visible. Neither exists. | ||
| 8. **The negative access test is missing, and it is the one that matters.** Project A reads its own | ||
| data, project B reaches neither through a hand-made namespace query nor through 9091 with the | ||
| project's service account. We could not run it: impersonating a service account is Forbidden for | ||
| this account on the workload cluster. Somebody with those rights should, and the result belongs | ||
| in here. | ||
|
|
||
| ## See also | ||
|
|
||
| - CCI-MOC/oac-apps#14, which implements this | ||
| - The ADR on central metric collection, which covers the other half of "why can somebody not see | ||
| this metric". That one is about what reaches the hub at all; this one is about who may read it. | ||
| Both answers are needed to work out where to look when a metric is missing. | ||
| - The ADR on running the Prometheus TSDB on FlashBlade NFS, for the volumes these dashboards read | ||
| through and why upstream does not stand behind that filesystem | ||
| - `pkg/controllers/uiplugin/prometheus-global-datasource.go` in the Cluster Observability Operator, | ||
| for the hardcoded Perses datasource port | ||
| - The MOC 1.0 metering and billing design, section "How AI Telemetry multi-tenancy and | ||
| authorization works", for the hub and tenant model built with the Prometheus Keycloak Proxy | ||
Uh oh!
There was an error while loading. Please reload this page.