Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
312 changes: 312 additions & 0 deletions adrs/2026-10-19-per-project-metric-separation/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,312 @@
---
title: Separate project metrics with one Grafana per project
Comment thread
schwesig marked this conversation as resolved.
Resolved
standards:
- nist-ac
- nist-au
---

## Context

Projects on OAC/MOC 2.0 need to see their own metrics and only their own. The obvious way, the console

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it correct to say that projects on OAC/MOC 2.0 need to see their metrics so that they can debug their application performance on their own?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true now for the OAC. I would expect it to also be true for MOC2.0 if we want to allow as much project self-service for debugging cluster issues as possible. Do we expect this to be a requirement for all the projects on the NIST/HIPAA clusters @dsmk? I can imagine that some projects such as Harbor would want it, but maybe they are an exception?

dashboards that come built in, does not give us that. The reason is specific enough to write down.

`thanos-querier` listens on two ports, and they behave differently:

| Port | Name | Behaviour |
|---|---|---|
| 9091 | `web` | **Does not filter.** Measured: 19 namespaces returned, with and without a `namespace` parameter, identical result |
| 9092 | `tenancy` | **Filters**, and a query cannot override the namespace |

Port 9092 is a chain of two proxies in front of the querier:

```
9092 (TLS) -> kube-rbac-proxy -> 127.0.0.1:9095 -> prom-label-proxy -> 127.0.0.1:9090 (thanos-query)
| |
checks the token's k8s RBAC enforces --label=namespace
```

Read off the running pod on `oac-prod-workload0`: `kube-rbac-proxy` runs with
`--secure-listen-address=0.0.0.0:9092` and an allow-list of query paths, `prom-label-proxy` with
`--label=namespace` and `--error-on-replace`.

`--error-on-replace` matters, but on its own it is not the authorisation. It makes the proxy reject
a conflicting selector instead of rewriting it. Which namespace a caller may ask for is decided
before that, by the identity in the request, and prom-label-proxy says itself that it does no
authentication or authorisation.

So the full chain in our setup, read off the chart and the cluster:

| | |
|---|---|
| identity | `grafana-sa`, one service account per project namespace |
| its rights | `RoleBinding grafana-view` → ClusterRole `view`, **bound to the project namespace only** |
| cluster-wide | one ClusterRoleBinding, `system:auth-delegator`, for the oauth-proxy's SAR path, not for queries |
| the namespace asked for | `customQueryParameters: namespace=project-<name>`, set in the datasource |
| why it cannot be changed | the datasource is `editable: false`, and `--error-on-replace` rejects a query that carries a different namespace |

The guarantee rests on all five, not on the port alone.

### Why not Perses

The Cluster Observability Operator (COO)'s UI plugin gives the console a dashboard page built on Perses, and that was the first plan.
It cannot separate projects, and not as a configuration gap we could close.

In COO 1.5.3, the version installed here, the Perses datasource is created pointing at port
**9091**, in `pkg/controllers/uiplugin/prometheus-global-datasource.go:37`, and the CRD field
`spec.monitoring.perses` has one key, `enabled`. So there is nothing to point elsewhere. Perses
does have namespace-scoped resources and RBAC for dashboards; what it does not have here is a
datasource that filters the metrics it queries.

We measured 9091 instead of trusting what we read: with our own token it returned all 19
namespaces, with and without a `namespace` parameter. That is one token and one query, so it shows
the port does not filter, not what every Perses user sees. For an operator identity that is fine,
for a project it would be wrong.

This was decided the other way round once, on 2026-09-02, with the reasoning that "Perses gets
tenancy from the namespace mapping for free". **That reasoning was wrong**, Perses has no namespace
mapping, and the decision was reopened and re-decided on 2026-09-08.

### Perses stays

The paragraph above reads like a verdict on Perses, and it is not. Perses is enabled in the console
on our clusters and stays there. Three reasons, only the first is about tenancy:

1. **For the fleet view it is the right tool.** Seeing everything is the job of an operator
Comment thread
knikolla marked this conversation as resolved.
Resolved
dashboard, and that 9091 returns all namespaces is correct for this reader. The same property
that makes it unusable for a project makes it usefull here.
2. **Two ways in, for two kinds of user.** Somebody holding `cluster-monitoring-view` gets
dashboards in the console that is open anyway, with nothing to deploy. A project gets its own
Grafana. To give people the choice is worth something by itself, and neither of the two has to
serve both audiences.
3. **It is already integrated.** `openshift/monitoring-plugin` depends on `@perses-dev/*` and
renders the Dashboards tab with it, while Metrics and Alerting use Victory. So the console ships
a dashboard surface either way, and enabling the UI plugin only exposes the tab.

So this ADR is not "Grafana instead of Perses". It is both, for different views.

The role is `cluster-monitoring-view`, and a project's own `view` is not enough for it. `view`
grants `prometheuses` in the API group `monitoring.rhobs`, which are the custom resources, not
`prometheuses/api` in `monitoring.coreos.com`, which is the data. So a project member can open the
Dashboards tab and gets no metrics out of it. Measured with SubjectAccessReviews on the prod hub:
true for a member of `open-accelerator-admins`, which holds the role through
`open-accelerator-admins-cluster-monitoring-view`, false for an authenticated user outside it. The
role is scoped with `resourceNames: ["k8s"]`, so a review that leaves the name out comes back false
as well.

## Decision

**One Grafana instance per project namespace**, with a datasource pointing at
`https://thanos-querier.openshift-monitoring.svc:9092`, shipped by the `user-projects` chart and
opt-in per project. Implemented in CCI-MOC/oac-apps#14, merged 2026-09-28.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am unclear why we are using one Grafana per project, rather than one Perses per project. I'm not arguing the decision, I just don't see that decision addressed explicitly.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. perses can get rights from kubernetes-rbac
  2. perses can be inside the openshift console and "outside"
  3. grafana can NOT be inside the openshift console (except workarounds like iframe etc), but can be "ouside"
  4. perses is NOT in the open-shift catalogue, but grafana is (see below)
  5. perses: in console port 9092 (or any other) can not be set/changed, only outside/standalone one
  6. grafana: no console at all, outside port 9092 can be set in grafana outside (what our user-projects do today)

A standalone perses (alowin other ports than 9091) can be build, but it is not "out of the catalogue".
Therefore the recommended decisison (for now) to go with the "one click/ out of the catalogue" idea.


info checked:
The perses-operator is in none of the catalogs our clusters see.
Measured on the prod hub:
650 PackageManifests across four catalogs,

  • community-operators 314,
  • certified-operators 173,
  • redhat-operators 151 and
  • redhat-marketplace 12,

and not a Perses entry.

grafana-operator is there, in community-operators, channel v5, currently grafana-operator.v5.25.0.


Worth being explicit where the separation happens, because it is easy to read this the wrong
way: **Grafana does not separate anything here.** It never sees more than its own project. The
instance is unrestricted inside, one organisation, one datasource, and the limit sits in front of
it in the proxy chain on 9092. We did not configure Grafana to filter by namespace, we gave it a
datasource that cannot return anything else.

The supporting choices, each of which could be made differently:

- **Per project, not one shared Grafana with orgs.** Separation is then the cluster's, enforced by
RBAC and `--error-on-replace`, rather than Grafana's own org model. Everything else in this repo
separates by Kubernetes RBAC, so this keeps one mental model instead of two.
- **The CRs live in `user-projects`, not in a chart of their own.** A separate chart would hold a
second copy of the project list and the two would drift. `user-projects` already owns the
per-project loop.
- **A route with an `oauth-proxy` sidecar**, the service account as the OAuth client. Access is a
SubjectAccessReview for `services get` in the project's namespace, through `--openshift-sar` for
the cookie path and `--openshift-delegate-urls` for the bearer-token one. So whoever already has
`edit` in the project reaches the Grafana, no Keycloak client per project, and route access and
metric access test the same RBAC. Which group grants that `edit` is up to the project's values,
the chart does not fix a name.
- **Opt-in, off by default.** A project without a `grafana:` block renders exactly as before, no
pod and no quota used. Adding the template to the chart must not hand six existing projects a
Grafana they never asked for.
- **Dashboards are chart resources, not hand-built per project.** `project-overview` is one
provisioning file in `user-projects`, rendered into each project namespace. Because the
datasource behind it is scoped by the proxy, the same dashboard definition shows a different
project its own data without being edited. Write a panel once, every project that opts in gets it,
and nobody has to maintain six copies that drift.
- **No volume.** "No PVC" sounds like the metrics are not kept. They are, just not here: the
platform Prometheus on each workload cluster is configured for 15 days on a 50Gi volume, UWM
another 20Gi, and Grafana queries that through the tenancy port.

Grafana does keep state of its own, in a database: users, datasources, dashboards, preferences.
We run it without a volume anyway, which means everything we need is read from the provisioning
files at startup, and anything else is allowed to be lost. **That has a consequence for
projects: a dashboard built in the UI does not survive.** Dashboards belong in git. If a project
needs to keep its own, it adds them to the chart. An opt-in for editing in the UI existed for a
while and was dropped again in CCI-MOC/oac-apps#247.
- **No alerting in Grafana**, a separate decision from the volume. Alerting belongs to the
platform: Prometheus evaluates `PrometheusRule`, Alertmanager routes what fires, both already
running and both tenant-aware. Grafana alerting would be a second path next to the one OpenShift
ships, so it is switched off in the config (`unified_alerting.enabled: "false"`).
- **No operator.** The chart deploys Grafana itself and sets it up from provisioning files at
startup. `grafana-operator` v5 ran it until CCI-MOC/oac-apps#230, but it only ships in
`community-operators`, it is not flagged FIPS-compliant, and it pushed each project's datasource
token into Grafana's admin API over plain HTTP. Grafana listens on `127.0.0.1` only, so the
oauth-proxy in the same pod is the only way in.

### What we did not pick

#### One shared Grafana with Grafana organisations

Cheaper, one pod and one URL for everybody. Organisations are core Grafana and
work in the community edition, no Enterprise licence needed: dashboards,
folders, datasources, alerts and service accounts separate per organisation,
and `org_mapping` in Generic OAuth puts a Keycloak group into the right one.
Team Sync and per-datasource permissions are Enterprise, neither is needed
here.

Two reasons against:

1. Grafana would hold the credentials of several projects at once, so their
correct assignment inside Grafana becomes security relevant. A misconfigured
organisation or a shared datasource lets one project see another.

2. Provisioning files carry an `orgId`, but the organisations themselves are
not provisionable. They would have to be created through the Grafana API or
a job of our own, outside the files the chart writes, and the datasource of
every project would move into one pod.

If project count makes the per-project cost hurt, this is still the option to
come back to, with that gap as part of the price.

#### Per-cluster separation instead of per-namespace

Simpler, but it does not match OAC/MOC 2.0: the projects share
`oac-prod-workload0`, so the dividing line is the namespace. Not really a
choice, it is what the environment is.

#### AI Telemetry with the Prometheus Keycloak Proxy

This is how MOC 1.0 solved this using solutions developed by Chris Tate. It
separates more than we do and one level higher: hubs, tenants, clusters and
projects, Keycloak groups created per hub and per tenant, one proxy per hub
sharing a realm with the application. For a provider with several hubs and real
tenants that is the stronger model, and it ran in production.

The reason against is ownership, not technique. As far as I know it is no
official Red Hat feature or product, so support and maintenance would sit with
our team. For a component between projects and their data, carrying an
authorisation decision, that is a commitment to make deliberately. The
per-project Grafana leans on supported pieces instead: the OpenShift proxy
chain, the oauth-proxy sidecar, the cluster's RBAC.

If OAC/MOC 2.0 grows several hubs or needs tenants above the project level,
this is the design to look at again, with the support question answered first.

## Consequences

**The separation is enforced outside Grafana**, by the proxy and by RBAC, not by the application.
A dashboard with the wrong query does not leak across projects, because the proxy rejects it rather
than Grafana deciding to be careful.

Verified: the datasource points at 9092, it is `editable: false`, the service account's `view` is
bound to its own namespace, and only `auth-delegator` is cluster-wide. **Not** verified: that
`grafana-sa` gets nothing useful out of 9091, and that a hand-made query for another namespace is
refused. Both need impersonating the service account, which this account may not do on the workload
cluster. So this is a well-founded expectation, not a proven one.

**The cost is one Grafana pod plus one `oauth-proxy` sidecar per participating project**, against
that project's own quota. In numbers, from the chart:

| | requests | limits |
| --------------------- | ------------------- | ------------------- |
| Grafana | 100m CPU, 256Mi | 500m CPU, 512Mi |
| `oauth-proxy` sidecar | 50m CPU, 64Mi | 200m CPU, 256Mi |
| **per project** | **150m CPU, 320Mi** | **700m CPU, 768Mi** |

Deliberately small, and the sidecar needs its own entry because a container without resources picks
up the project `LimitRange` default of 1 CPU and 2Gi. No storage, see above.

If project count grows faster than expected, the shared-Grafana model comes back onto the table,
and with it the separation moving into the application.

**Two open questions nobody decided**, and they are not the same one. The pods sit in the project
namespace and consume from its quota, and what exactly they consume depends on whether the quota
caps requests, limits or both.

1. **Is the quota raised when Grafana is enabled?** If not, a project trades 150m CPU and 320Mi of
its own workload for the dashboard. The quota does not shrink, its remaining capacity does.
2. **Are those resources billed to the project?** Separate question: the pod consumes quota either
way, billing only decides who pays for it.

For raising the quota: observing your own workload is part of running on the platform, and making
people pay for it discourages the thing we want. Against billing it: 150m CPU is below the noise of
any billing unit. Both are policy, not technical, and better decided before the first project asks.

**It means also one URL per project**, every Grafana gets its own route in its own namespace.
There is no single address today where we can send everybody, so somebody has to tell a project
where its dashboard lifes. A landing page or a redirect could give one entry point without merging
the instances, we just have not built it.

**The upside sits on the same mechanism.** A dashboard written once in the chart works for every
project, because the scoping happens in the datasource and not in the dashboard. Adding a panel is
one change in one file, and a new project that opts in gets the current set with nothing to copy.

**We run two dashboard tools for two jobs.** Perses in the console on 9091 for the cluster-wide
view, Grafana per project on 9092. Intended, and still real operational work: two things to upgrade,
two things to explane. One correction to how I put it earlier: 9091 on a workload cluster shows that
cluster, not the fleet. The fleet view is the hub's.

**Dashboards are read only in the GUI.** They come from git, and an anonymous Editor could not be
held to a change, so editing was dropped in CCI-MOC/oac-apps#247.

**We maintain the Deployment ourselves** instead of letting an operator do it. In exchange nothing a
project uses depends on a community build.

**A project dashboard is only as good as the platform Prometheus under it**, and that is not part of
this chart. `cadvisor` lives in the platform instance, not in user workload monitoring, so the
container metrics the `project-overview` panels draw come from there. On a cluster where the platform
Prometheus mounts `emptyDir`, a project gets a dashboard that works and shows the age of the pod.
Nothing in Grafana reports this, and Argo is green either way.

That is a per-cluster property, so it has to be checked per cluster rather than assumed from this
decision. The three workload clusters are configured alike, `retention: 15d`, `retentionSize: 40GiB`
on a 50Gi volume, but what a project actually sees is the age of the volume. Measured 2026-11-03:
prod-workload0 holds 15.7 days, dev-workload0 holds 6.4, because that cluster was rebuilt a week
ago. The configuration is the ceiling, not the promise.

## Open points

1. **Nothing is built for a tenancy demo yet.** Showing two projects that cannot see each other is
the thing a reviewer will ask for, and it does not exist.
2. **The upstream gap in Perses is unreported.** An issue making the datasource port configurable
was drafted and never opened. If that lands, the per-project Grafana becomes optional rather than
necessary, which would be a simplification worth having.
3. **Per-project resource limits are small on purpose** and have not been tested under real
dashboard use.
4. **The quota question above needs an owner.** It is not blocking anything today, because few
projects have Grafana enabled, and it becomes awkward the first time somebody notices.
5. **Whether AI Telemetry is supportable** was never actually asked, I am going on what I know. If
somebody checks and it turns out there is a support path, the alternative above deserves a real
comparison rather than the short treatment it gets here.
6. **SQLite on `pure-fb-nfsv4` works, but not under a real project yet.** The concern was file
locking, SQLite being known to be fragile on NFS. Tested on dev-workload0 on 2026-10-20 with
`grafana/grafana:11.3.0`: the export mounts `nfsvers=4.1` with no `nolock`, the pod reports
`local_lock=none`, the database was created and the migrations ran, and after deleting the pod the
new one reopened the same file and logged `migrations completed, performed=602, skipped=6`. No
stale lock. Nothing writes to that database any more since editing was dropped, so this stays a
measurement rather than something we depend on.
7. **Nothing warns when the platform Prometheus below a project has no volume.** The dashboard looks
healthy and the panels are simply short. A check on the cluster, or a panel that shows the oldest
sample it can see, would make it visible. Neither exists.
8. **The negative access test is missing, and it is the one that matters.** Project A reads its own
data, project B reaches neither through a hand-made namespace query nor through 9091 with the
project's service account. We could not run it: impersonating a service account is Forbidden for
this account on the workload cluster. Somebody with those rights should, and the result belongs
in here.

## See also

- CCI-MOC/oac-apps#14, which implements this
- The ADR on central metric collection, which covers the other half of "why can somebody not see
this metric". That one is about what reaches the hub at all; this one is about who may read it.
Both answers are needed to work out where to look when a metric is missing.
- The ADR on running the Prometheus TSDB on FlashBlade NFS, for the volumes these dashboards read
through and why upstream does not stand behind that filesystem
- `pkg/controllers/uiplugin/prometheus-global-datasource.go` in the Cluster Observability Operator,
for the hardcoded Perses datasource port
- The MOC 1.0 metering and billing design, section "How AI Telemetry multi-tenancy and
authorization works", for the hub and tenant model built with the Prometheus Keycloak Proxy
Loading