diff --git a/docs/2026-10-20-dashboard-mco-capacity/README.md b/docs/2026-10-20-dashboard-mco-capacity/README.md new file mode 100644 index 0000000..c80168d --- /dev/null +++ b/docs/2026-10-20-dashboard-mco-capacity/README.md @@ -0,0 +1,184 @@ +--- +title: "Dashboard: MCO capacity and growth" +--- + +This document describes one dashboard as it exists in oac-apps +[`457b659`][main], 2026-10-20. It records why the dashboard exists and why +each metric is on it. The dashboard strategy (which tool for which audience, +naming, ownership) is the subject of a separate ADR, +[Dashboard strategy](../../adrs/2026-09-04-dashboard-strategy/), which is +Proposed and in review in a separate PR. + +## Question and audience + +**Question:** how fast does the hub's ACM Observability (MCO) store grow, and +is it losing data on the way in? + +**Audience:** hub operators. The dashboard queries hub-local metrics and needs +`cluster-monitoring-view`, which the `oac-simple-rbac` chart binds to the admin +group ([`allow-monitoring-view.yaml`][rbac]). It is not meant for projects. + +**Why it was created**, from CCI-MOC/oac-apps#139 (merged 2026-10-20): + +> Since MCO retains data now, how fast it grows is a question we can answer, +> and until now I measured it by hand each time. This puts it on a dashboard. + +MCO started retaining data once Thanos receive ran on both hubs +(CCI-MOC/oac-apps#136, #137; CCI-MOC/MOC-issues#611). + +## Tool and location + +One dashboard with two renderings. Both carry the same four panels and the +same queries. They exist because the console has two dashboard lists. + +| Rendering | Resource | Namespace | Where it shows up | Source | +|---|---|---|---|---| +| Perses | `PersesDashboard` `mco-capacity` (`perses.dev/v1alpha2`) | `openshift-cluster-observability-operator` | Observe > Dashboards (Perses), added by the COO Monitoring UI plugin | [`perses-dashboard-mco-capacity.yaml`][perses] | +| Console | ConfigMap `dashboard-mco-capacity`, label `console.openshift.io/dashboard: "true"` | `openshift-config-managed` | The older Observe > Dashboards list | [`console-dashboard-mco-capacity.yaml`][console] | + +- **Chart:** `charts/cluster-observability-operator`. The values + `capacityDashboard.perses.enabled` and `capacityDashboard.console.enabled` + switch each rendering, and both default to `true` ([`values.yaml`][values]). +- **Deployed to:** every cluster that runs the chart. That is both hubs, through + `hub-components`, and every managed cluster, through `all-managed-clusters` + ([`hub-components.yaml`][hubappset], [`all-managed-clusters.yaml`][managedappset]). + See open point 1. +- **Default time range:** 7 days. +- **Tests:** [`capacity_dashboard_test.yaml`][test] checks that both renderings + appear by default, that the console label and namespace are set, and that the + `acm_` prefix sits on the metric name and not on the `job` label. + +Red Hat documents the Perses route: "You can create Perses dashboards by +defining PersesDashboard custom resources by enabling dashboard-as-code +practices and GitOps workflows" ([COO UI plugins, Perses dashboards][coo-crd]). +The ConfigMap route in `openshift-config-managed` is not described in the +OCP 4.22 monitoring guides we checked. The chart relies on observed console +behaviour (open point 3). + +## Data source + +The hub's own platform monitoring stack: `thanos-querier` in +`openshift-monitoring`. Not MCO's Thanos, and not the fleet data. + +Why not MCO's own Grafana, from #139: + +> That one has a single datasource, the rbac-query-proxy, and it serves fleet +> data. The tsdb metrics in these panels sit in the hub's own prometheus and +> come back empty over the proxy. +> +> A second datasource is not possible either: the `grafana-datasources` secret +> is owned by the `MultiClusterObservability` resource, `mco-operator` is the +> only manager on it, and the CRD has no field to declare one. + +The COO documentation shows how a Perses datasource reaches "the platform +Thanos Querier instance run by the Cluster Monitoring Operator", on port 9091 +([COO, global Thanos datasource][coo-ds]). Port 9091 "provides access to all +the Thanos Querier endpoints" ([OCP 4.22, Accessing metrics, services][cmo-svc]), +which is what a fleet-operator dashboard wants. + +**The `acm_` prefix.** The metrics carry an `acm_` prefix on their name, but +the `job` label keeps the unprefixed value +`observability-thanos-receive-default`. From the chart comment: "ACM renames +the metrics it scrapes, so the name carries acm_ while the job label keeps the +real name." This is read off the cluster. We did not find it in the ACM 2.17 +observability guide. + +## Metrics + +All four panels look at Thanos receive, the MCO component that ingests what the +managed clusters send. Two panels show growth and two show storage. + +| Panel | PromQL (as in [`457b659`][main]) | Metric source | Needs the MCO allowlist? | Why it is there | +|---|---|---|---|---| +| Series held by receive | `sum(acm_prometheus_tsdb_head_series{job="observability-thanos-receive-default"})` | The TSDB of the Thanos receive pods in `open-cluster-management-observability`, scraped by the hub's platform Prometheus and renamed with `acm_` | No | How many active series receive holds, the size of the store. It is the first row of the hand measurements in #139 (dev hub 607,960, prod hub 342,205 on 2026-10-20) | +| Samples written per second | `sum(rate(acm_prometheus_tsdb_head_samples_appended_total{job="observability-thanos-receive-default"}[5m]))` | Same as above | No | Ingest rate, the second number from #139 (dev 6,546/s, prod 3,549/s) | +| Disk used per receive replica | `kubelet_volume_stats_used_bytes{namespace="open-cluster-management-observability",persistentvolumeclaim=~"data-observability-thanos-receive.*"}` | kubelet volume stats for the receive PVCs, a platform metric on the hub. One series per PVC | No | Shows when a receive volume fills up. A per-panel reason is not recorded beyond the "capacity and growth" purpose | +| Rejected samples, out of order or out of bounds | `sum(rate(acm_prometheus_tsdb_out_of_order_samples_total{job="observability-thanos-receive-default"}[5m]))` and `sum(rate(acm_prometheus_tsdb_out_of_bound_samples_total{job="observability-thanos-receive-default"}[5m]))` | Same TSDB as the first two panels | No | Data loss on ingest. Chart comment: "Receive has no out-of-order window, so anything counted here is data that never lands." | + +The dashboard does not read these metrics through MCO. It queries the hub's +own `thanos-querier`, so whether the MCO allowlist names them makes no +difference here, and the allowlist decision in +https://github.com/CCI-MOC/ops-docs/pull/36 does not affect this dashboard. + +The `[5m]` rate windows are under review: CCI-MOC/oac-apps#149 (open) found +that over a 14 day range the step grows past 5 minutes and the samples panel +"draws drops that are not in the data". It proposes `$__rate_interval` for +Perses and a fixed `30m` window for the console list, "because that one does +not resolve the variable". Update this table when #149 merges. + +## Thresholds + +None. The dashboard has no thresholds, colouring or linked alerts. The numbers +measured on 2026-10-20 in #139 are a reference point, not a limit: + +| | dev hub | prod hub | +|---|---|---| +| series | 607,960 | 342,205 | +| samples/s | 6,546 | 3,549 | +| metric names | 223 | 177 | + +Any value above zero in the rejected-samples panel means data that never lands. + +## Dependencies + +- **Cluster Observability Operator** with the Monitoring UI plugin and Perses + enabled (`uiPlugin.perses.enabled: true`), for the Perses rendering + ([`uiplugin.yaml`][uiplugin]). +- **ACM Observability (MCO) with Thanos receive running** on the cluster + (`charts/acm-observability`, CCI-MOC/oac-apps#119). Without it the panels are + empty. +- **`cluster-monitoring-view`** for the reader, or "the dashboards render but + every panel says Forbidden" ([`allow-monitoring-view.yaml`][rbac], + CCI-MOC/oac-apps#9). +- **MCO allowlist:** not needed (see Metrics). + +## Owner + +Not recorded in oac-apps. Both renderings were added in CCI-MOC/oac-apps#139. + +## Open points + +1. **Rendered on clusters without MCO.** The values comment says the dashboard + is "Off for a cluster without MCO, where the panels stay empty". But both + switches default to `true`, and no values file in oac-apps turns them off. So + the managed clusters also get the dashboard, with empty panels. Either + default it to off and enable it in the hub values, or record that empty + panels on managed clusters are accepted. +2. **History depth on the hub.** The dashboard opens on 7 days and #149 talks + about 14. The data comes from the hub's platform Prometheus. oac-apps sets no + `cluster-monitoring-config` on the hubs (`user-workload-monitoring` is not in + `hub-components`). So the hub's retention and whether it has a volume are not + recorded. Without a volume, the history ends at the age of the pod. +3. **Console ConfigMap route is undocumented.** We found no OCP 4.22 Red Hat + documentation for dashboards loaded from ConfigMaps in + `openshift-config-managed`. It works today, but a console upgrade could + change it without notice. +4. **Owner.** No owner is named. The strategy ADR (Proposed, in review in a + separate PR) is expected to decide who owns dashboards. + +## See also + +- [Dashboard strategy](../../adrs/2026-09-04-dashboard-strategy/) (Proposed, + in review in a separate PR): which tool for which audience, + dashboards as code, naming and ownership. +- https://github.com/CCI-MOC/ops-docs/pull/36: MCO metrics allowlist ADR. +- https://github.com/CCI-MOC/ops-docs/pull/37: per-project Grafana ADR, + section "Perses stays", on why Perses is the tool for the operator view. +- [Dashboard: Project overview](../2026-10-20-dashboard-project-overview/), the + other dashboard in oac-apps. +- CCI-MOC/oac-apps#139: the PR that added the dashboard, with screenshots of + both renderings. +- CCI-MOC/oac-apps#149: rate window fix (open). + +[main]: https://github.com/CCI-MOC/oac-apps/tree/457b659283f3f8da57bb44454628170058625a44 +[perses]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/templates/perses-dashboard-mco-capacity.yaml +[console]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/templates/console-dashboard-mco-capacity.yaml +[values]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/values.yaml +[uiplugin]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/templates/uiplugin.yaml +[test]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/tests/unit/capacity_dashboard_test.yaml +[hubappset]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/applicationsets/templates/hub/hub-components.yaml +[managedappset]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/applicationsets/templates/managed/all-managed-clusters.yaml +[rbac]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/oac-simple-rbac/templates/allow-monitoring-view.yaml +[coo-crd]: https://docs.redhat.com/en/documentation/red_hat_openshift_cluster_observability_operator/1-latest/html-single/ui_plugins_for_red_hat_openshift_cluster_observability_operator/index#coo-perses-dashboards-create-crd_perses-dashboard +[coo-ds]: https://docs.redhat.com/en/documentation/red_hat_openshift_cluster_observability_operator/1-latest/html-single/ui_plugins_for_red_hat_openshift_cluster_observability_operator/index#coo-perses-dashboards-datasources-global-thanos_perses-dashboard +[cmo-svc]: https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.22/html-single/accessing_metrics/index#cmo-services-resources_accessing-monitoring-apis-by-using-the-cli diff --git a/docs/2026-10-20-dashboard-project-overview/README.md b/docs/2026-10-20-dashboard-project-overview/README.md new file mode 100644 index 0000000..4f9c70f --- /dev/null +++ b/docs/2026-10-20-dashboard-project-overview/README.md @@ -0,0 +1,172 @@ +--- +title: "Dashboard: Project overview" +--- + +This document describes one dashboard as it exists in oac-apps +[`457b659`][main], 2026-10-20. It records why the dashboard exists and why +each metric is on it. The dashboard strategy (which tool for which audience, +naming, ownership) is the subject of a separate ADR, +[Dashboard strategy](../../adrs/2026-09-04-dashboard-strategy/), which is +Proposed and in review in a separate PR. + +## Question and audience + +**Question:** how much CPU and memory do the pods in my project use? + +**Audience:** the members of one project, in that project's own Grafana. They +see only their own namespace. Operators who want the cluster-wide view use the +console dashboards instead (see the +[MCO capacity dashboard](../2026-10-20-dashboard-mco-capacity/) and +https://github.com/CCI-MOC/ops-docs/pull/37, "Perses stays"). + +**Why it was created:** it is the dashboard that ships with the per-project +Grafana, added in CCI-MOC/oac-apps#14 (merged 2026-09-28). That PR gives the +reasons for Grafana itself: + +> Grafana is widely known and was already used in MOC 1.0, so existing +> dashboards onboard easy and old users get the same look and feel. +> +> Second reason is separation. Perses is locked to port 9091 and cannot +> separate tenants today, so startups sharing a workload cluster would see each +> other's metrics. + +Why a starter dashboard is a chart resource rather than something each project +builds, from https://github.com/CCI-MOC/ops-docs/pull/37: "the same dashboard +definition shows a different project its own data without being edited. Write +a panel once, every project that opts in gets it". Why the first two panels are +CPU and memory, and not something else, is not recorded. + +## Tool and location + +| | | +|---|---| +| Tool | Grafana 11.3.0, one instance per project, run by the community Grafana operator (`grafana.integreatly.org/v1beta1`) | +| Resource | `GrafanaDashboard` `project-overview` in `project-`, matched to the instance by the label `dashboards: project-` | +| Source | [`charts/user-projects/templates/grafana.yaml`][grafana], last part of the file | +| Switch | Per project, `grafana.enabled: true` in the `user-projects` values. Off by default | +| Deployed to | Managed clusters, through `all-managed-clusters` ([`all-managed-clusters.yaml`][managedappset]). Today only `project-gavin-test` on `oac-dev-workload0` sets the switch ([values][devvalues]). No prod project does | +| UID | `project--ovw`, fixed at 28 characters, because Grafana caps a UID at 40 characters and the readable form would break at 24 characters of project name (#14) | +| Default time range | 6 hours | +| Tests | [`grafana_test.yaml`][test] | + +Viewers cannot save changes. From the chart: "Dashboards belong in git as +GrafanaDashboard resources, because Grafana OSS cannot grant 'may save +dashboards' without also granting 'may create datasources'". So this dashboard +is changed only through oac-apps. CCI-MOC/oac-apps#150 (open) proposes an opt-in +editable Grafana. It leaves this dashboard as it is. + +## Data source + +One `GrafanaDatasource` named `thanos`, the only datasource the instance has. +It points at +`https://thanos-querier.openshift-monitoring.svc.cluster.local:9092`, the +tenancy port. It authenticates as the project's `grafana-sa` service account, +which has `view` in the project only, and adds `namespace=project-` to +every query. + +Red Hat documents port 9092 as providing "access to the `/api/v1/query`, +`/api/v1/query_range/`, `/api/v1/labels`, `/api/v1/label/*/values`, and +`/api/v1/series` endpoints restricted to a given project. Granting access +requires binding a user to the view cluster role in the project" +([OCP 4.22, Accessing metrics, services][cmo-svc]). The full proxy chain behind +9092, and why the namespace cannot be overridden, is in +https://github.com/CCI-MOC/ops-docs/pull/37 and not repeated here. + +**History depth.** This Grafana stores no metrics of its own. Both metrics live +in the cluster's platform Prometheus, so the history is that instance's +retention. On both hubs' managed clusters it is set to 15 days on a volume +(`platform.enabled: true` in the hub-wide `user-workload-monitoring` values, +CCI-MOC/oac-apps#131 and #134). Without that volume the platform Prometheus +mounts `emptyDir`, and the panels show only the age of the pod. CCI-MOC/oac-apps#132 +(open) adds a note about this to the chart values. + +## Metrics + +| Panel | PromQL (rendered for project ``) | Metric source | Needs the MCO allowlist? | Why it is there | +|---|---|---|---|---| +| CPU usage by pod | `sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="project-", container!=""}[5m]))` | cAdvisor in the kubelet, scraped by the platform Prometheus in `openshift-monitoring` | No | CPU per pod in the project. The choice of this metric is not recorded | +| Memory usage by pod | `sum by (pod) (container_memory_working_set_bytes{namespace="project-", container!=""})` | Same | No | Memory per pod in the project. The choice of this metric is not recorded | + +In the chart, the template inserts the project name in place of ``. The +legend shows one series per pod. + +Two details in the queries are deliberate: + +- **The namespace is named in the query**, even though the datasource already + forces it. From the commit that added it (CCI-MOC/oac-apps@0b135b3, + 2026-09-28): "The dashboard queries name the namespace as well rather than + relying on the port to inject it, so they stay scoped if the datasource is + ever repointed or the JSON gets copied." +- **`container!=""`** drops the series that cAdvisor reports for the pod as a + whole, so each pod is summed from its containers only and not counted twice. + The reason for the filter is not recorded in oac-apps. This is what the + filter does. + +The dashboard queries the cluster's local `thanos-querier` and not MCO, so the +MCO allowlist (https://github.com/CCI-MOC/ops-docs/pull/36) does not affect it. +Worth knowing anyway: `container_cpu_usage_seconds_total` is one of the +metrics #36 lists as measured locally but not collected at the hub. A fleet +version of this dashboard would need it on the allowlist. + +## Thresholds + +None. No thresholds, colouring or alerts. Alerting is "phase 2" in #14. +The project's own quota (`ResourceQuota`, default `requests.cpu: 10`, +`limits.memory: 40Gi`, [`user-projects/values.yaml`][values]) would be the +natural reference line, but it is not drawn. + +## Dependencies + +- **Grafana operator** (`charts/grafana-operator`, community catalog, channel + `v5`) on the cluster. It is only in the community catalog, which means "no + official support and no Red Hat certification" ([chart values][opvalues]). +- **Network access** from the project namespace to `thanos-querier` on 9092, and + from the operator namespace to Grafana on 3000, through + `restrict-tenant-networks` (CCI-MOC/oac-apps#31). +- **Platform Prometheus volume** on the cluster for any history beyond the age + of the pod (see Data source). +- **MCO allowlist:** not needed. + +## Owner + +Not recorded in oac-apps. The dashboard was added in CCI-MOC/oac-apps#14 with +the per-project Grafana. + +## Open points + +1. **Panel choice.** Why CPU and memory, and whether projects need more (GPU, + storage, network), is not recorded. GPU is a likely next question: + `requests.nvidia.com/gpu` is part of the project quota, and the DCGM exporter + runs on GPU clusters. +2. **Rate window.** The CPU panel uses `[5m]`. CCI-MOC/oac-apps#149 found that a + fixed `[5m]` draws drops that are not there once the step passes 5 minutes, + for the MCO capacity dashboard. With a 6 hour default range this panel is + less exposed, but a long range would hit the same effect. #149 does not touch + this file. +3. **Owner.** No owner is named. The strategy ADR (Proposed, in review in a + separate PR) is expected to decide who owns dashboards. +4. **Only on dev so far.** No prod project uses it yet. The tenancy filter was + to be "measured with a real token" on `gavin-test` first (CCI-MOC/oac-apps@9c08828). + Whether that test happened is not recorded. + +## See also + +- [Dashboard strategy](../../adrs/2026-09-04-dashboard-strategy/) (Proposed, + in review in a separate PR): which tool for which audience, + dashboards as code, naming and ownership. +- https://github.com/CCI-MOC/ops-docs/pull/37: per-project Grafana ADR, the + decision this dashboard depends on. +- https://github.com/CCI-MOC/ops-docs/pull/36: MCO metrics allowlist ADR. +- [Dashboard: MCO capacity and growth](../2026-10-20-dashboard-mco-capacity/), + the other dashboard in oac-apps. +- CCI-MOC/oac-apps#14: the PR that added the Grafana and this dashboard. +- CCI-MOC/oac-apps#132, #150: open follow-ups on the project Grafana. + +[main]: https://github.com/CCI-MOC/oac-apps/tree/457b659283f3f8da57bb44454628170058625a44 +[grafana]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/user-projects/templates/grafana.yaml +[values]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/user-projects/values.yaml +[test]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/user-projects/tests/unit/grafana_test.yaml +[devvalues]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/values/oac-dev-infra/oac-dev-workload0/user-projects.yaml +[managedappset]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/applicationsets/templates/managed/all-managed-clusters.yaml +[opvalues]: https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/grafana-operator/values.yaml +[cmo-svc]: https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.22/html-single/accessing_metrics/index#cmo-services-resources_accessing-monitoring-apis-by-using-the-cli