Skip to content

docs: document the two oac-apps dashboards, one page each - #2

Open
schwesig wants to merge 2 commits into
mainfrom
fm/ops-docs-dashboard-docs
Open

schwesig wants to merge 2 commits into
mainfrom
fm/ops-docs-dashboard-docs

Conversation

@schwesig

@schwesig schwesig commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Intent

Dashboards are built for the OAC clusters (Grafana, Perses, OpenShift Console Observe), today only in oac-apps (https://github.com/schwesig/OA_oac-apps, fork of CCI-MOC/oac-apps). The goal is to always know why a dashboard was created and why it is fed with which metrics:

"ich baue ja auch dashboards, ich wuerde gerne entweder ein ADR pro Dashboard oder ein ADT fuer mehrere Dashboards (Grafana, Perses, OpenSHift COnsole Ob serve) erstellen, damit man immer weiss, warum man was erstellt und mit diesen metriken etc gefuettert hat."

Agreed split ("ja passt, bisher nur in OAC-APPS"): one ADR for the dashboard strategy (which tool for which audience and purpose, data sources, dashboards as code, naming and ownership; referencing rather than repeating the existing per-project Grafana ADR in CCI-MOC#37 and the MCO allowlist ADR in CCI-MOC#36), plus one short document per dashboard in ops-docs docs/ (what question it answers and for whom, tool and repo location, data source, metrics with PromQL and why/where each metric comes from, thresholds, dependencies such as the MCO allowlist, owner, related ADRs). A separate ADR per dashboard only where a dashboard forces a real decision.
All work and PRs only on the fork schwesig/ops-docs; never on CCI-MOC.

What Changed

  • Adds docs/2026-10-20-dashboard-mco-capacity/README.md. It documents the MCO capacity and growth dashboard, which has a Perses PersesDashboard rendering and a console ConfigMap rendering. The page covers the question it answers, who it is for, its data source (the hub's own thanos-querier, not MCO's Grafana), each panel's PromQL with the metric's origin and why it is there, thresholds (none), dependencies, owner (not recorded) and open points. The open points include the pending [5m] rate-window fix in oac-apps#149 and the dashboard being rendered on clusters without MCO.
  • Adds docs/2026-10-20-dashboard-project-overview/README.md. It documents the per-project Grafana project-overview dashboard with the same structure: tenancy datasource on port 9092, the CPU and memory PromQL and why the namespace is named in each query, dependencies, and open points such as panel choice and the fact that it is only on dev so far.
  • Both pages pin oac-apps sources to commit 457b659 and say whether each dashboard needs the MCO allowlist (neither does). They reference the per-project Grafana ADR (adr: Separate project metrics with one Grafana per project on the tenancy port CCI-MOC/ops-docs#37) and the MCO allowlist ADR (adr: Collect central metrics using MultiClusterObservability allowlist keeping exclusion lists as a future change CCI-MOC/ops-docs#36) instead of repeating them. They also link to adrs/2026-09-04-dashboard-strategy/, marked Proposed. That ADR is not in this branch and is in review in a separate PR, so the link stays broken until that PR merges.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change only adds two docs pages that follow the repo's docs template and match the agreed split (the strategy ADR lives on a separate branch), and nothing executable is touched.

Testing

I built and served the ops-docs Eleventy site from this branch and browsed it over HTTP. Frontmatter validation passes. Both dashboard pages render fully and appear in the docs index. Each page covers question and audience, tool and location, data source, metrics with PromQL, thresholds, dependencies, owner, open points and related ADRs. Angle-bracket placeholders render as visible text, and the two pages link to each other correctly. The strategy-ADR link is a 404 on this branch, which you already accepted. There are no screenshots: the chrome-devtools-axi bridge never started (BRIDGE_NOT_READY, the Chrome target exited), so the evidence is the rendered HTML pages saved with their CSS. The temporary site, the HTTP server and node_modules were removed afterwards.

  • Live validation: ✅ go - 5 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Reader opens the Technical Documentation index in the built site and sees both new dashboard pages listed ✅ pass live Built with eleventy and served over HTTP; docs/index.html links /docs/2026-10-20-dashboard-mco-capacity/ and /docs/2026-10-20-dashboard-project-overview/ (rendered-site/docs/index.html)
Reader opens each dashboard page and finds why it exists, the tool and repo location, the data source, metrics with PromQL and where each comes from, thresholds, dependencies (MCO allowlist), owner an… ✅ pass live HTTP 200 for both pages. Rendered headings are complete and the metric tables hold the PromQL rows (rendered-site artifacts)
Adversarial: placeholders like <name> and <first 16 hex chars...> are not swallowed as HTML (markdown-it html:true) and no Liquid {{ }} breaks the build ✅ pass live The rendered HTML shows &lt;name&gt; inside code, has no raw <name>/<first tags and no {{ or {%, and eleventy built without error
Frontmatter of the new docs pages passes the repo's schema validator ✅ pass live npm run validate-frontmatter exit 0
Reader follows the cross-links: dashboard to dashboard resolves; the strategy ADR link points at the sibling PR path ✅ pass live The relative links between the two dashboard pages resolve (HTTP 200). ../../adrs/2026-09-04-dashboard-strategy/ returns 404 on this branch, which you accepted in review round 2 (it depends on the sib…
Evidence: Rendered MCO capacity dashboard page (built by Eleventy)
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Dashboard: MCO capacity and growth - MOC Operations Documentation</title>
  <link rel="stylesheet" href="/_assets/style.css">
</head>
<body>
  <header>
    <a href="/">
      <img src="/_assets/moc-wordmark-small.png" alt="MOC Alliance">
    </a>
    <nav>
      <ul>
        <li><a href="/">Home</a></li>
        <li><a href="/adrs/">ADRs</a></li>
        <li><a href="/docs/">Technical Documentation</a></li>
        <li><a href="/policies/">Policies</a></li>
        <li><a href="/production-changes/">Production Changes</a></li>
      </ul>
    </nav>
  </header>
  <main>
    <h1>Dashboard: MCO capacity and growth</h1>
    <p class="page-id">docs/2026-10-20-dashboard-mco-capacity</p>
    <p>This document describes one dashboard as it exists in oac-apps
<a href="https://github.com/CCI-MOC/oac-apps/tree/457b659283f3f8da57bb44454628170058625a44"><code>457b659</code></a>, 2026-10-20. It records why the dashboard exists and why
each metric is on it. The dashboard strategy (which tool for which audience,
naming, ownership) is the subject of a separate ADR,
<a href="../../adrs/2026-09-04-dashboard-strategy/">Dashboard strategy</a>, which is
Proposed and in review in a separate PR.</p>
<h2>Question and audience</h2>
<p><strong>Question:</strong> how fast does the hub's ACM Observability (MCO) store grow, and
is it losing data on the way in?</p>
<p><strong>Audience:</strong> hub operators. The dashboard queries hub-local metrics and needs
<code>cluster-monitoring-view</code>, which the <code>oac-simple-rbac</code> chart binds to the admin
group (<a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/oac-simple-rbac/templates/allow-monitoring-view.yaml"><code>allow-monitoring-view.yaml</code></a>). It is not meant for projects.</p>
<p><strong>Why it was created</strong>, from CCI-MOC/oac-apps#139 (merged 2026-10-20):</p>
<blockquote>
<p>Since MCO retains data now, how fast it grows is a question we can answer,
and until now I measured it by hand each time. This puts it on a dashboard.</p>
</blockquote>
<p>MCO started retaining data once Thanos receive ran on both hubs
(CCI-MOC/oac-apps#136, #137; CCI-MOC/MOC-issues#611).</p>
<h2>Tool and location</h2>
<p>One dashboard with two renderings. Both carry the same four panels and the
same queries. They exist because the console has two dashboard lists.</p>
<table>
<thead>
<tr>
<th>Rendering</th>
<th>Resource</th>
<th>Namespace</th>
<th>Where it shows up</th>
<th>Source</th>
</tr>
</thead>
<tbody>
<tr>
<td>Perses</td>
<td><code>PersesDashboard</code> <code>mco-capacity</code> (<code>perses.dev/v1alpha2</code>)</td>
<td><code>openshift-cluster-observability-operator</code></td>
<td>Observe &gt; Dashboards (Perses), added by the COO Monitoring UI plugin</td>
<td><a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/templates/perses-dashboard-mco-capacity.yaml"><code>perses-dashboard-mco-capacity.yaml</code></a></td>
</tr>
<tr>
<td>Console</td>
<td>ConfigMap <code>dashboard-mco-capacity</code>, label <code>console.openshift.io/dashboard: &quot;true&quot;</code></td>
<td><code>openshift-config-managed</code></td>
<td>The older Observe &gt; Dashboards list</td>
<td><a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/templates/console-dashboard-mco-capacity.yaml"><code>console-dashboard-mco-capacity.yaml</code></a></td>
</tr>
</tbody>
</table>
<ul>
<li><strong>Chart:</strong> <code>charts/cluster-observability-operator</code>. The values
<code>capacityDashboard.perses.enabled</code> and <code>capacityDashboard.console.enabled</code>
switch each rendering, and both default to <code>true</code> (<a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/values.yaml"><code>values.yaml</code></a>).</li>
<li><strong>Deployed to:</strong> every cluster that runs the chart. That is both hubs, through
<code>hub-components</code>, and every managed cluster, through <code>all-managed-clusters</code>
(<a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/applicationsets/templates/hub/hub-components.yaml"><code>hub-components.yaml</code></a>, <a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/applicationsets/templates/managed/all-managed-clusters.yaml"><code>all-managed-clusters.yaml</code></a>).
See open point 1.</li>
<li><strong>Default time range:</strong> 7 days.</li>
<li><strong>Tests:</strong> <a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/tests/unit/capacity_dashboard_test.yaml"><code>capacity_dashboard_test.yaml</code></a> checks that both renderings
appear by default, that the console label and namespace are set, and that the
<code>acm_</code> prefix sits on the metric name and not on the <code>job</code> label.</li>
</ul>
<p>Red Hat documents the Perses route: &quot;You can create Perses dashboards by
defining PersesDashboard custom resources by enabling dashboard-as-code
practices and GitOps workflows&quot; (<a href="https://docs.redhat.com/en/documentation/red_hat_openshift_cluster_observability_operator/1-latest/html-single/ui_plugins_for_red_hat_openshift_cluster_observability_operator/index#coo-perses-dashboards-create-crd_perses-dashboard">COO UI plugins, Perses dashboards</a>).
The ConfigMap route in <code>openshift-config-managed</code> is not described in the
OCP 4.22 monitoring guides we checked. The chart relies on observed console
behaviour (open point 3).</p>
<h2>Data source</h2>
<p>The hub's own platform monitoring stack: <code>thanos-querier</code> in
<code>openshift-monitoring</code>. Not MCO's Thanos, and not the fleet data.</p>
<p>Why not MCO's own Grafana, from #139:</p>
<blockquote>
<p>That one has a single datasource, the rbac-query-proxy, and it serves fleet
data. The tsdb metrics in these panels sit in the hub's own prometheus and
come back empty over the proxy.</p>
<p>A second datasource is not possible either: the <code>grafana-datasources</code> secret
is owned by the <code>MultiClusterObservability</code> resource, <code>mco-operator</code> is the
only manager on it, and the CRD has no field to declare one.</p>
</blockquote>
<p>The COO documentation shows how a Perses datasource reaches &quot;the platform
Thanos Querier instance run by the Cluster Monitoring Operator&quot;, on port 9091
(<a href="https://docs.redhat.com/en/documentation/red_hat_openshift_cluster_observability_operator/1-latest/html-single/ui_plugins_for_red_hat_openshift_cluster_observability_operator/index#coo-perses-dashboards-datasources-global-thanos_perses-dashboard">COO, global Thanos datasource</a>). Port 9091 &quot;provides access to all
the Thanos Querier endpoints&quot; (<a href="https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.22/html-single/accessing_metrics/index#cmo-services-resources_accessing-monitoring-apis-by-using-the-cli">OCP 4.22, Accessing metrics, services</a>),
which is what a fleet-operator dashboard wants.</p>
<p><strong>The <code>acm_</code> prefix.</strong> The metrics carry an <code>acm_</code> prefix on their name, but
the <code>job</code> label keeps the unprefixed value
<code>observability-thanos-receive-default</code>. From the chart comment: &quot;ACM renames
the metrics it scrapes, so the name carries acm_ while the job label keeps the
real name.&quot; This is read off the cluster. We did not find it in the ACM 2.17
observability guide.</p>
<h2>Metrics</h2>
<p>All four panels look at Thanos receive, the MCO component that ingests what the
managed clusters send. Two panels show growth and two show storage.</p>
<table>
<thead>
<tr>
<th>Panel</th>
<th>PromQL (as in <a href="https://github.com/CCI-MOC/oac-apps/tree/457b659283f3f8da57bb44454628170058625a44"><code>457b659</code></a>)</th>
<th>Metric source</th>
<th>Needs the MCO allowlist?</th>
<th>Why it is there</th>
</tr>
</thead>
<tbody>
<tr>
<td>Series held by receive</td>
<td><code>sum(acm_prometheus_tsdb_head_series{job=&quot;observability-thanos-receive-default&quot;})</code></td>
<td>The TSDB of the Thanos receive pods in <code>open-cluster-management-observability</code>, scraped by the hub's platform Prometheus and renamed with <code>acm_</code></td>
<td>No</td>
<td>How many active series receive holds, the size of the store. It is the first row of the hand measurements in #139 (dev hub 607,960, prod hub 342,205 on 2026-10-20)</td>
</tr>
<tr>
<td>Samples written per second</td>
<td><code>sum(rate(acm_prometheus_tsdb_head_samples_appended_total{job=&quot;observability-thanos-receive-default&quot;}[5m]))</code></td>
<td>Same as above</td>
<td>No</td>
<td>Ingest rate, the second number from #139 (dev 6,546/s, prod 3,549/s)</td>
</tr>
<tr>
<td>Disk used per receive replica</td>
<td><code>kubelet_volume_stats_used_bytes{namespace=&quot;open-cluster-management-observability&quot;,persistentvolumeclaim=~&quot;data-observability-thanos-receive.*&quot;}</code></td>
<td>kubelet volume stats for the receive PVCs, a platform metric on the hub. One series per PVC</td>
<td>No</td>
<td>Shows when a receive volume fills up. A per-panel reason is not recorded beyond the &quot;capacity and growth&quot; purpose</td>
</tr>
<tr>
<td>Rejected samples, out of order or out of bounds</td>
<td><code>sum(rate(acm_prometheus_tsdb_out_of_order_samples_total{job=&quot;observability-thanos-receive-default&quot;}[5m]))</code> and <code>sum(rate(acm_prometheus_tsdb_out_of_bound_samples_total{job=&quot;observability-thanos-receive-default&quot;}[5m]))</code></td>
<td>Same TSDB as the first two panels</td>
<td>No</td>
<td>Data loss on ingest. Chart comment: &quot;Receive has no out-of-order window, so anything counted here is data that never lands.&quot;</td>
</tr>
</tbody>
</table>
<p>The dashboard does not read these metrics through MCO. It queries the hub's
own <code>thanos-querier</code>, so whether the MCO allowlist names them makes no
difference here, and the allowlist decision in
https://github.com/CCI-MOC/ops-docs/pull/36 does not affect this dashboard.</p>
<p>The <code>[5m]</code> rate windows are under review: CCI-MOC/oac-apps#149 (open) found
that over a 14 day range the step grows past 5 minutes and the samples panel
&quot;draws drops that are not in the data&quot;. It proposes <code>$__rate_interval</code> for
Perses and a fixed <code>30m</code> window for the console list, &quot;because that one does
not resolve the variable&quot;. Update this table when #149 merges.</p>
<h2>Thresholds</h2>
<p>None. The dashboard has no thresholds, colouring or linked alerts. The numbers
measured on 2026-10-20 in #139 are a reference point, not a limit:</p>
<table>
<thead>
<tr>
<th></th>
<th>dev hub</th>
<th>prod hub</th>
</tr>
</thead>
<tbody>
<tr>
<td>series</td>
<td>607,960</td>
<td>342,205</td>
</tr>
<tr>
<td>samples/s</td>
<td>6,546</td>
<td>3,549</td>
</tr>
<tr>
<td>metric names</td>
<td>223</td>
<td>177</td>
</tr>
</tbody>
</table>
<p>Any value above zero in the rejected-samples panel means data that never lands.</p>
<h2>Dependencies</h2>
<ul>
<li><strong>Cluster Observability Operator</strong> with the Monitoring UI plugin and Perses
enabled (<code>uiPlugin.perses.enabled: true</code>), for the Perses rendering
(<a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/cluster-observability-operator/templates/uiplugin.yaml"><code>uiplugin.yaml</code></a>).</li>
<li><strong>ACM Observability (MCO) with Thanos receive running</strong> on the cluster
(<code>charts/acm-observability</code>, CCI-MOC/oac-apps#119). Without it the panels are
empty.</li>
<li><strong><code>cluster-monitoring-view</code></strong> for the reader, or &quot;the dashboards render but
every panel says Forbidden&quot; (<a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/oac-simple-rbac/templates/allow-monitoring-view.yaml"><code>allow-monitoring-view.yaml</code></a>,
CCI-MOC/oac-apps#9).</li>
<li><strong>MCO allowlist:</strong> not needed (see Metrics).</li>
</ul>
<h2>Owner</h2>
<p>Not recorded in oac-apps. Both renderings were added in CCI-MOC/oac-apps#139.</p>
<h2>Open points</h2>
<ol>
<li><strong>Rendered on clusters without MCO.</strong> The values comment says the dashboard
is &quot;Off for a cluster without MCO, where the panels stay empty&quot;. But both
switches default to <code>true</code>, and no values file in oac-apps turns them off. So
the managed clusters also get the dashboard, with empty panels. Either
default it to off and enable it in the hub values, or record that empty
panels on managed clusters are accepted.</li>
<li><strong>History depth on the hub.</strong> The dashboard opens on 7 days and #149 talks
about 14. The data comes from the hub's platform Prometheus. oac-apps sets no
<code>cluster-monitoring-config</code> on the hubs (<code>user-workload-monitoring</code> is not in
<code>hub-components</code>). So the hub's retention and whether it has a volume are not
recorded. Without a volume, the history ends at the age of the pod.</li>
<li><strong>Console ConfigMap route is undocumented.</strong> We found no OCP 4.22 Red Hat
documentation for dashboards loaded from ConfigMaps in
<code>openshift-config-managed</code>. It works today, but a console upgrade could
change it without notice.</li>
<li><strong>Owner.</strong> No owner is named. The strategy ADR (Proposed, in review in a
separate PR) is expected to decide who owns dashboards.</li>
</ol>
<h2>See also</h2>
<ul>
<li><a href="../../adrs/2026-09-04-dashboard-strategy/">Dashboard strategy</a> (Proposed,
in review in a separate PR): which tool for which audience,
dashboards as code, naming and ownership.</li>
<li>https://github.com/CCI-MOC/ops-docs/pull/36: MCO metrics allowlist ADR.</li>
<li>https://github.com/CCI-MOC/ops-docs/pull/37: per-project Grafana ADR,
section &quot;Perses stays&quot;, on why Perses is the tool for the operator view.</li>
<li><a href="../2026-10-20-dashboard-project-overview/">Dashboard: Project overview</a>, the
other dashboard in oac-apps.</li>
<li>CCI-MOC/oac-apps#139: the PR that added the dashboard, with screenshots of
both renderings.</li>
<li>CCI-MOC/oac-apps#149: rate window fix (open).</li>
</ul>

  </main>
  <footer>
    <p>
      Revision
      <a href="https://github.com/cci-moc/ops-docs/commit/eafd02a7e681e3cedc138754201ed1413742bb67">eafd02a7</a>
      @ 2026-10-20 18:01:09 +0200
    <span class="noprint"> |
    <a href="https://github.com/cci-moc/ops-docs/blob/main/./docs/2026-10-20-dashboard-mco-capacity/README.md?plain=1">View page source</a>
    </span>
    </p>
  </footer>
</body>
</html>
Evidence: Rendered Project overview dashboard page (built by Eleventy)
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Dashboard: Project overview - MOC Operations Documentation</title>
  <link rel="stylesheet" href="/_assets/style.css">
</head>
<body>
  <header>
    <a href="/">
      <img src="/_assets/moc-wordmark-small.png" alt="MOC Alliance">
    </a>
    <nav>
      <ul>
        <li><a href="/">Home</a></li>
        <li><a href="/adrs/">ADRs</a></li>
        <li><a href="/docs/">Technical Documentation</a></li>
        <li><a href="/policies/">Policies</a></li>
        <li><a href="/production-changes/">Production Changes</a></li>
      </ul>
    </nav>
  </header>
  <main>
    <h1>Dashboard: Project overview</h1>
    <p class="page-id">docs/2026-10-20-dashboard-project-overview</p>
    <p>This document describes one dashboard as it exists in oac-apps
<a href="https://github.com/CCI-MOC/oac-apps/tree/457b659283f3f8da57bb44454628170058625a44"><code>457b659</code></a>, 2026-10-20. It records why the dashboard exists and why
each metric is on it. The dashboard strategy (which tool for which audience,
naming, ownership) is the subject of a separate ADR,
<a href="../../adrs/2026-09-04-dashboard-strategy/">Dashboard strategy</a>, which is
Proposed and in review in a separate PR.</p>
<h2>Question and audience</h2>
<p><strong>Question:</strong> how much CPU and memory do the pods in my project use?</p>
<p><strong>Audience:</strong> the members of one project, in that project's own Grafana. They
see only their own namespace. Operators who want the cluster-wide view use the
console dashboards instead (see the
<a href="../2026-10-20-dashboard-mco-capacity/">MCO capacity dashboard</a> and
https://github.com/CCI-MOC/ops-docs/pull/37, &quot;Perses stays&quot;).</p>
<p><strong>Why it was created:</strong> it is the dashboard that ships with the per-project
Grafana, added in CCI-MOC/oac-apps#14 (merged 2026-09-28). That PR gives the
reasons for Grafana itself:</p>
<blockquote>
<p>Grafana is widely known and was already used in MOC 1.0, so existing
dashboards onboard easy and old users get the same look and feel.</p>
<p>Second reason is separation. Perses is locked to port 9091 and cannot
separate tenants today, so startups sharing a workload cluster would see each
other's metrics.</p>
</blockquote>
<p>Why a starter dashboard is a chart resource rather than something each project
builds, from https://github.com/CCI-MOC/ops-docs/pull/37: &quot;the same dashboard
definition shows a different project its own data without being edited. Write
a panel once, every project that opts in gets it&quot;. Why the first two panels are
CPU and memory, and not something else, is not recorded.</p>
<h2>Tool and location</h2>
<table>
<thead>
<tr>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td>Tool</td>
<td>Grafana 11.3.0, one instance per project, run by the community Grafana operator (<code>grafana.integreatly.org/v1beta1</code>)</td>
</tr>
<tr>
<td>Resource</td>
<td><code>GrafanaDashboard</code> <code>project-overview</code> in <code>project-&lt;name&gt;</code>, matched to the instance by the label <code>dashboards: project-&lt;name&gt;</code></td>
</tr>
<tr>
<td>Source</td>
<td><a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/user-projects/templates/grafana.yaml"><code>charts/user-projects/templates/grafana.yaml</code></a>, last part of the file</td>
</tr>
<tr>
<td>Switch</td>
<td>Per project, <code>grafana.enabled: true</code> in the <code>user-projects</code> values. Off by default</td>
</tr>
<tr>
<td>Deployed to</td>
<td>Managed clusters, through <code>all-managed-clusters</code> (<a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/applicationsets/templates/managed/all-managed-clusters.yaml"><code>all-managed-clusters.yaml</code></a>). Today only <code>project-gavin-test</code> on <code>oac-dev-workload0</code> sets the switch (<a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/values/oac-dev-infra/oac-dev-workload0/user-projects.yaml">values</a>). No prod project does</td>
</tr>
<tr>
<td>UID</td>
<td><code>project-&lt;first 16 hex chars of sha256(name)&gt;-ovw</code>, fixed at 28 characters, because Grafana caps a UID at 40 characters and the readable form would break at 24 characters of project name (#14)</td>
</tr>
<tr>
<td>Default time range</td>
<td>6 hours</td>
</tr>
<tr>
<td>Tests</td>
<td><a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/user-projects/tests/unit/grafana_test.yaml"><code>grafana_test.yaml</code></a></td>
</tr>
</tbody>
</table>
<p>Viewers cannot save changes. From the chart: &quot;Dashboards belong in git as
GrafanaDashboard resources, because Grafana OSS cannot grant 'may save
dashboards' without also granting 'may create datasources'&quot;. So this dashboard
is changed only through oac-apps. CCI-MOC/oac-apps#150 (open) proposes an opt-in
editable Grafana. It leaves this dashboard as it is.</p>
<h2>Data source</h2>
<p>One <code>GrafanaDatasource</code> named <code>thanos</code>, the only datasource the instance has.
It points at
<code>https://thanos-querier.openshift-monitoring.svc.cluster.local:9092</code>, the
tenancy port. It authenticates as the project's <code>grafana-sa</code> service account,
which has <code>view</code> in the project only, and adds <code>namespace=project-&lt;name&gt;</code> to
every query.</p>
<p>Red Hat documents port 9092 as providing &quot;access to the <code>/api/v1/query</code>,
<code>/api/v1/query_range/</code>, <code>/api/v1/labels</code>, <code>/api/v1/label/*/values</code>, and
<code>/api/v1/series</code> endpoints restricted to a given project. Granting access
requires binding a user to the view cluster role in the project&quot;
(<a href="https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.22/html-single/accessing_metrics/index#cmo-services-resources_accessing-monitoring-apis-by-using-the-cli">OCP 4.22, Accessing metrics, services</a>). The full proxy chain behind
9092, and why the namespace cannot be overridden, is in
https://github.com/CCI-MOC/ops-docs/pull/37 and not repeated here.</p>
<p><strong>History depth.</strong> This Grafana stores no metrics of its own. Both metrics live
in the cluster's platform Prometheus, so the history is that instance's
retention. On both hubs' managed clusters it is set to 15 days on a volume
(<code>platform.enabled: true</code> in the hub-wide <code>user-workload-monitoring</code> values,
CCI-MOC/oac-apps#131 and #134). Without that volume the platform Prometheus
mounts <code>emptyDir</code>, and the panels show only the age of the pod. CCI-MOC/oac-apps#132
(open) adds a note about this to the chart values.</p>
<h2>Metrics</h2>
<table>
<thead>
<tr>
<th>Panel</th>
<th>PromQL (rendered for project <code>&lt;name&gt;</code>)</th>
<th>Metric source</th>
<th>Needs the MCO allowlist?</th>
<th>Why it is there</th>
</tr>
</thead>
<tbody>
<tr>
<td>CPU usage by pod</td>
<td><code>sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=&quot;project-&lt;name&gt;&quot;, container!=&quot;&quot;}[5m]))</code></td>
<td>cAdvisor in the kubelet, scraped by the platform Prometheus in <code>openshift-monitoring</code></td>
<td>No</td>
<td>CPU per pod in the project. The choice of this metric is not recorded</td>
</tr>
<tr>
<td>Memory usage by pod</td>
<td><code>sum by (pod) (container_memory_working_set_bytes{namespace=&quot;project-&lt;name&gt;&quot;, container!=&quot;&quot;})</code></td>
<td>Same</td>
<td>No</td>
<td>Memory per pod in the project. The choice of this metric is not recorded</td>
</tr>
</tbody>
</table>
<p>In the chart, the template inserts the project name in place of <code>&lt;name&gt;</code>. The
legend shows one series per pod.</p>
<p>Two details in the queries are deliberate:</p>
<ul>
<li><strong>The namespace is named in the query</strong>, even though the datasource already
forces it. From the commit that added it (CCI-MOC/oac-apps@0b135b3,
2026-09-28): &quot;The dashboard queries name the namespace as well rather than
relying on the port to inject it, so they stay scoped if the datasource is
ever repointed or the JSON gets copied.&quot;</li>
<li><strong><code>container!=&quot;&quot;</code></strong> drops the series that cAdvisor reports for the pod as a
whole, so each pod is summed from its containers only and not counted twice.
The reason for the filter is not recorded in oac-apps. This is what the
filter does.</li>
</ul>
<p>The dashboard queries the cluster's local <code>thanos-querier</code> and not MCO, so the
MCO allowlist (https://github.com/CCI-MOC/ops-docs/pull/36) does not affect it.
Worth knowing anyway: <code>container_cpu_usage_seconds_total</code> is one of the
metrics #36 lists as measured locally but not collected at the hub. A fleet
version of this dashboard would need it on the allowlist.</p>
<h2>Thresholds</h2>
<p>None. No thresholds, colouring or alerts. Alerting is &quot;phase 2&quot; in #14.
The project's own quota (<code>ResourceQuota</code>, default <code>requests.cpu: 10</code>,
<code>limits.memory: 40Gi</code>, <a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/user-projects/values.yaml"><code>user-projects/values.yaml</code></a>) would be the
natural reference line, but it is not drawn.</p>
<h2>Dependencies</h2>
<ul>
<li><strong>Grafana operator</strong> (<code>charts/grafana-operator</code>, community catalog, channel
<code>v5</code>) on the cluster. It is only in the community catalog, which means &quot;no
official support and no Red Hat certification&quot; (<a href="https://github.com/CCI-MOC/oac-apps/blob/457b659283f3f8da57bb44454628170058625a44/charts/grafana-operator/values.yaml">chart values</a>).</li>
<li><strong>Network access</strong> from the project namespace to <code>thanos-querier</code> on 9092, and
from the operator namespace to Grafana on 3000, through
<code>restrict-tenant-networks</code> (CCI-MOC/oac-apps#31).</li>
<li><strong>Platform Prometheus volume</strong> on the cluster for any history beyond the age
of the pod (see Data source).</li>
<li><strong>MCO allowlist:</strong> not needed.</li>
</ul>
<h2>Owner</h2>
<p>Not recorded in oac-apps. The dashboard was added in CCI-MOC/oac-apps#14 with
the per-project Grafana.</p>
<h2>Open points</h2>
<ol>
<li><strong>Panel choice.</strong> Why CPU and memory, and whether projects need more (GPU,
storage, network), is not recorded. GPU is a likely next question:
<code>requests.nvidia.com/gpu</code> is part of the project quota, and the DCGM exporter
runs on GPU clusters.</li>
<li><strong>Rate window.</strong> The CPU panel uses <code>[5m]</code>. CCI-MOC/oac-apps#149 found that a
fixed <code>[5m]</code> draws drops that are not there once the step passes 5 minutes,
for the MCO capacity dashboard. With a 6 hour default range this panel is
less exposed, but a long range would hit the same effect. #149 does not touch
this file.</li>
<li><strong>Owner.</strong> No owner is named. The strategy ADR (Proposed, in review in a
separate PR) is expected to decide who owns dashboards.</li>
<li><strong>Only on dev so far.</strong> No prod project uses it yet. The tenancy filter was
to be &quot;measured with a real token&quot; on <code>gavin-test</code> first (CCI-MOC/oac-apps@9c08828).
Whether that test happened is not recorded.</li>
</ol>
<h2>See also</h2>
<ul>
<li><a href="../../adrs/2026-09-04-dashboard-strategy/">Dashboard strategy</a> (Proposed,
in review in a separate PR): which tool for which audience,
dashboards as code, naming and ownership.</li>
<li>https://github.com/CCI-MOC/ops-docs/pull/37: per-project Grafana ADR, the
decision this dashboard depends on.</li>
<li>https://github.com/CCI-MOC/ops-docs/pull/36: MCO metrics allowlist ADR.</li>
<li><a href="../2026-10-20-dashboard-mco-capacity/">Dashboard: MCO capacity and growth</a>,
the other dashboard in oac-apps.</li>
<li>CCI-MOC/oac-apps#14: the PR that added the Grafana and this dashboard.</li>
<li>CCI-MOC/oac-apps#132, #150: open follow-ups on the project Grafana.</li>
</ul>

  </main>
  <footer>
    <p>
      Revision
      <a href="https://github.com/cci-moc/ops-docs/commit/eafd02a7e681e3cedc138754201ed1413742bb67">eafd02a7</a>
      @ 2026-10-20 18:01:09 +0200
    <span class="noprint"> |
    <a href="https://github.com/cci-moc/ops-docs/blob/main/./docs/2026-10-20-dashboard-project-overview/README.md?plain=1">View page source</a>
    </span>
    </p>
  </footer>
</body>
</html>
Evidence: Rendered Technical Documentation index listing both pages
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Technical Documentation - MOC Operations Documentation</title>
  <link rel="stylesheet" href="/_assets/style.css">
</head>
<body>
  <header>
    <a href="/">
      <img src="/_assets/moc-wordmark-small.png" alt="MOC Alliance">
    </a>
    <nav>
      <ul>
        <li><a href="/">Home</a></li>
        <li><a href="/adrs/">ADRs</a></li>
        <li><a href="/docs/">Technical Documentation</a></li>
        <li><a href="/policies/">Policies</a></li>
        <li><a href="/production-changes/">Production Changes</a></li>
      </ul>
    </nav>
  </header>
  <main>
    <h1>Technical Documentation</h1>
    
    
<table>
  <tr>
    <th>ID</th><th>Title</th>
  </tr>

  
  <tr>
    <td>
      <a href="/docs/2026-10-20-dashboard-mco-capacity/">docs/2026-10-20-dashboard-mco-capacity</a>
    </td>
    <td>
      Dashboard: MCO capacity and growth
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/docs/2026-10-20-dashboard-project-overview/">docs/2026-10-20-dashboard-project-overview</a>
    </td>
    <td>
      Dashboard: Project overview
    </td>
  </tr>
  
</table>

  </main>
  <footer>
    <p>
      Revision
      <a href="https://github.com/cci-moc/ops-docs/commit/eafd02a7e681e3cedc138754201ed1413742bb67">eafd02a7</a>
      @ 2026-10-20 18:01:09 +0200
    <span class="noprint"> |
    <a href="https://github.com/cci-moc/ops-docs/blob/main/./docs/index.html?plain=1">View page source</a>
    </span>
    </p>
  </footer>
</body>
</html>
Evidence: Rendered page headings
mco-capacity: Question and audience | Tool and location | Data source | Metrics | Thresholds | Dependencies | Owner | Open points | See also
project-overview: Question and audience | Tool and location | Data source | Metrics | Thresholds | Dependencies | Owner | Open points | See also
- Outcome: ⚠️ 1 info across 1 run (4m22s)

Rendered MCO capacity dashboard page (full page, headless Firefox)
Rendered Project overview dashboard page (full page, headless Firefox)
Technical documentation index listing both dashboard pages

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 2 infos
  • ℹ️ docs/2026-10-20-dashboard-mco-capacity/README.md:9 - Both dashboard pages refer to the strategy ADR as the placeholder adrs/&lt;date&gt;-dashboard-strategy and call it "pending". The sibling branch fm/ops-docs-dashboard-adr already adds it as adrs/2026-09-04-dashboard-strategy. Once that ADR merges (or if this branch lands after it), replace the placeholder with a real relative link, ../../adrs/2026-09-04-dashboard-strategy/, so readers can follow it. The same placeholder appears at docs/2026-10-20-dashboard-mco-capacity/README.md:9 and :150, docs/2026-10-20-dashboard-mco-capacity/README.md:141 (open point 4), and docs/2026-10-20-dashboard-project-overview/README.md:9, :140 (open point 3) and :151.

🔧 Fix applied.
2 infos still open:

  • ℹ️ docs/2026-10-20-dashboard-mco-capacity/README.md:9 - Both dashboard pages refer to the strategy ADR as the placeholder adrs/&lt;date&gt;-dashboard-strategy and call it "pending". The sibling branch fm/ops-docs-dashboard-adr already adds it as adrs/2026-09-04-dashboard-strategy. Once that ADR merges (or if this branch lands after it), replace the placeholder with a real relative link, ../../adrs/2026-09-04-dashboard-strategy/, so readers can follow it. The same placeholder appears at docs/2026-10-20-dashboard-mco-capacity/README.md:9 and :150, docs/2026-10-20-dashboard-mco-capacity/README.md:141 (open point 4), and docs/2026-10-20-dashboard-project-overview/README.md:9, :140 (open point 3) and :151.
  • ℹ️ docs/2026-10-20-dashboard-mco-capacity/README.md:9 - The round-1 fix links to ../../adrs/2026-09-04-dashboard-strategy/. That path exists only on the sibling branch fm/ops-docs-dashboard-adr (adrs/2026-09-04-dashboard-strategy/README.md, Status: Proposed), so the link is a 404 on this branch until the ADR PR merges. The user chose this link explicitly in round 1. Nothing in pre-commit checks links, so CI will not fail on it. The point is merge order: merge the ADR PR first, or accept the broken links for a while. The same link appears at docs/2026-10-20-dashboard-mco-capacity/README.md:9 and :161, and at docs/2026-10-20-dashboard-project-overview/README.md:9 and :155.
⚠️ **Test** - 1 info
  • ℹ️ There are no screenshots of the rendered pages. The chrome-devtools-axi bridge failed on every attempt (BRIDGE_NOT_READY: the Chrome CDP target exited, or the port was held by a stale bridge). The rendered HTML pages with their CSS are saved as evidence instead. To get screenshots, make sure chrome-devtools-mcp can launch Chrome on this host (for example, check npx chrome-devtools-mcp@latest --help and stop any stale bridge), then re-run.
  • Live validation: ✅ go - 5 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Reader opens the Technical Documentation index in the built site and sees both new dashboard pages listed ✅ pass live Built with eleventy and served over HTTP; docs/index.html links /docs/2026-10-20-dashboard-mco-capacity/ and /docs/2026-10-20-dashboard-project-overview/ (rendered-site/docs/index.html)
Reader opens each dashboard page and finds why it exists, the tool and repo location, the data source, metrics with PromQL and where each comes from, thresholds, dependencies (MCO allowlist), owner an… ✅ pass live HTTP 200 for both pages. Rendered headings are complete and the metric tables hold the PromQL rows (rendered-site artifacts)
Adversarial: placeholders like <name> and <first 16 hex chars...> are not swallowed as HTML (markdown-it html:true) and no Liquid {{ }} breaks the build ✅ pass live The rendered HTML shows &lt;name&gt; inside code, has no raw <name>/<first tags and no {{ or {%, and eleventy built without error
Frontmatter of the new docs pages passes the repo's schema validator ✅ pass live npm run validate-frontmatter exit 0
Reader follows the cross-links: dashboard to dashboard resolves; the strategy ADR link points at the sibling PR path ✅ pass live The relative links between the two dashboard pages resolve (HTTP 200). ../../adrs/2026-09-04-dashboard-strategy/ returns 404 on this branch, which you accepted in review round 2 (it depends on the sib…
  • npm ci (in the worktree; node_modules removed afterwards)
  • npm run validate-frontmatter: exit 0
  • npx eleventy --output=/tmp/nm-site-01M5: built 12 files, including both dashboard pages
  • Served the built site with python3 -m http.server 8731 and fetched both dashboard pages with curl (HTTP 200) and the strategy ADR link (HTTP 404, as expected)
  • Checked the rendered HTML: both pages are listed in docs/index.html, the section headings are complete, the PromQL table rows are there, &lt;name&gt;-style placeholders are shown as escaped text inside code, and no {{/{% reached the page
  • Checked internal links: the two pages link to each other (both resolve) and both link to ../../adrs/2026-09-04-dashboard-strategy/ (404 on this branch)
  • Tried chrome-devtools-axi open/screenshot for screenshots: the bridge failed every time (BRIDGE_NOT_READY / CDP target gone)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Two dashboards exist in CCI-MOC/oac-apps at 457b659:

- MCO capacity and growth: one dashboard, rendered as a PersesDashboard
  and as a console ConfigMap with the same panels and queries.
- Project overview: the GrafanaDashboard shipped with the per-project
  Grafana in user-projects.

Each page uses the same sections: Question and audience, Tool and
location, Data source, Metrics, Thresholds, Dependencies, Owner, Open
points, See also. The Metrics table gives, per panel, the PromQL as in
the repo, where the metric comes from, whether it needs the MCO
allowlist, and why it is there. Where the reason is not recorded in
oac-apps, the page says so and lists it as an open point.

The dashboard strategy ADR is referenced as pending.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant