Skip to content

docs: how to create alerts on metric values and on log contents - #4

Open
schwesig wants to merge 3 commits into
mainfrom
fm/ops-docs-alerting-howto
Open

schwesig wants to merge 3 commits into
mainfrom
fm/ops-docs-alerting-howto

Conversation

@schwesig

Copy link
Copy Markdown
Owner

Intent

Write the documentation asked for by CCI-MOC/MOC-issues#339 ("Document how to create alerts on either metric values or log contents"; completion criteria: documentation written covering alert creation for both metrics and logs). It is a sub-issue of the epic CCI-MOC/MOC-issues#325 (OAC: Monitoring and logging).

"starte das Doku-Dokument für #339"

It is one document in ops-docs docs/, not an ADR. All work and PRs only on the fork schwesig/ops-docs; never on CCI-MOC (no PRs, comments, issues or links there).

What Changed

  • Adds docs/2026-10-20-alerting-howto/README.md, a how-to for OAC operators. It starts with a table that maps each kind of metric alert to its rule object and to where that rule is evaluated: an AlertingRule for a platform metric on one cluster, a PrometheusRule for a project metric on a workload cluster, and the hub's thanos-ruler-custom-rules ConfigMap for a metric across the fleet. For each path it gives a minimal YAML example, where the rule should live as code and how to check that the alert fires.
  • Documents alerts on log contents through Loki Operator AlertingRule objects that use LogQL: how to enable the LokiStack ruler, how to write a rule and how to check it. A warning explains that log alerts cannot run on OAC yet, because main deploys no logging stack and the open logging change has no spec.rules section.
  • Adds a "Where alerts end up today" table (no Alertmanager receivers are configured anywhere in oac-apps), a list of open points (receivers, enabling the Loki ruler, alert forwarding not verified) and a "See also" section. The page sets templateEngineOverride: md so the Prometheus {{ }} templates are not parsed by Liquid.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change adds a single documentation page. Both round-1 fixes are in place as the user instructed: the leaf-prometheus note at lines 195-198 keeps the ACM 2.17 link and adds no new quotes, and the API group is qualified at line 203. No new defects were found.

Testing

I built the real ops-docs Eleventy site from this branch into the evidence directory, served it on a local port and took a full-page screenshot of the new doc with headless Firefox. Frontmatter validation passes. The page renders the metric section (platform AlertingRule, project PrometheusRule, fleet-wide rule on the hub, how to check it fires) and the log section (enable the ruler, LokiStack AlertingRule, how to check it fires). The IMPORTANT and WARNING callouts render. The Prometheus {{ }} template comes through intact because the page skips Liquid. The page has no dead in-page anchors. Both fixes from round 1 show up in the rendered output. Offline, all YAML examples parse into the expected kinds and rules. That is not a live check, so the YAML scenario stays untested. Promtool, oc and a cluster are not available, so the PromQL and LogQL expressions were not run against a live Prometheus or Loki.

  • Live validation: ✅ go - 5 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Operator opens the docs site and finds the new alerting how-to listed and rendered ✅ pass live Eleventy build wrote docs/2026-10-20-alerting-howto/index.html; it is served with HTTP 200 and listed on /docs/; alerting-howto-full.png
Reader finds instructions for alerting on metric values (platform, project and fleet-wide) and on log contents, each with how to check that it fires ✅ pass live Rendered headings: Alert on a metric value (Platform metric, Project metric, Metric across the fleet, Check that a metric alert fires), Alert on log contents (Enable the ruler, Write the rule, Check t…
Rendering edge case: the Prometheus {{ $labels.instance }} template in an example is shown literally and does not break the Liquid build ✅ pass live Build succeeded; the rendered HTML contains labels.instance in the code block (templateEngineOverride: md works)
IMPORTANT callout uses plain section names and the published page has no dead in-page anchor links ✅ pass live No href="#..." in the rendered page; the callout reads 'See the sections "Where alerts end up today" and "Open points" below.' (alerting-howto-top.png)
Round-1 fixes: the leaf-prometheus scope sentence is present and the check command uses alertingrule.monitoring.openshift.io ✅ pass live Rendered HTML has the sentence 'evaluated only by the user workload Prometheus ... would never fire with the label' and oc get alertingrule.monitoring.openshift.io -n openshift-monitoring
A reader copying the YAML examples gets valid manifests with the intended kinds and rules ⏸️ untested no The prior payload did not establish a live result. The YAML was only parsed offline with js-yaml. No cluster, oc, promtool or logcli is available, so the manifests and the PromQL/LogQL expressions wer…

Full-page screenshot of the rendered alerting how-to
Top of the rendered page (title, IMPORTANT callout, path table)

Evidence: Rendered HTML page from the Eleventy build
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>How to create alerts on metric values and on log contents - MOC Operations Documentation</title>
  <link rel="stylesheet" href="/_assets/style.css">
</head>
<body>
  <header>
    <a href="/">
      <img src="/_assets/moc-wordmark-small.png" alt="MOC Alliance">
    </a>
    <nav>
      <ul>
        <li><a href="/">Home</a></li>
        <li><a href="/adrs/">ADRs</a></li>
        <li><a href="/docs/">Technical Documentation</a></li>
        <li><a href="/policies/">Policies</a></li>
        <li><a href="/production-changes/">Production Changes</a></li>
      </ul>
    </nav>
  </header>
  <main>
    <h1>How to create alerts on metric values and on log contents</h1>
    <p class="page-id">docs/2026-10-20-alerting-howto</p>
    <p>This document describes how an operator on the OAC clusters creates an alert
on a metric value and an alert on log contents. It covers what oac-apps
deploys at <a href="https://github.com/CCI-MOC/oac-apps/tree/1260a865c69c1851a75b781df53ae40ca41e6be1"><code>1260a86</code></a> (2026-10-20) and the logging change that is
still open in <a href="https://github.com/schwesig/OA_oac-apps/pull/2">schwesig/OA_oac-apps#2</a>. For each path it gives a
minimal YAML example, where the rule should live as code, how to check that it
fires, and where the alert goes.</p>
<div class="markdown-alert markdown-alert-important"><p class="markdown-alert-title"><svg class="octicon octicon-report mr-2" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v9.5A1.75 1.75 0 0 1 14.25 13H8.06l-2.573 2.573A1.458 1.458 0 0 1 3 14.543V13H1.75A1.75 1.75 0 0 1 0 11.25Zm1.75-.25a.25.25 0 0 0-.25.25v9.5c0 .138.112.25.25.25h2a.75.75 0 0 1 .75.75v2.19l2.72-2.72a.749.749 0 0 1 .53-.22h6.5a.25.25 0 0 0 .25-.25v-9.5a.25.25 0 0 0-.25-.25Zm7 2.25v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 9a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z"></path></svg>Important</p><p>Alerts are evaluated today, but they are not delivered to anyone. oac-apps
configures no Alertmanager receivers on any cluster, and log alerts cannot be
evaluated yet. See the sections &quot;Where alerts end up today&quot; and &quot;Open
points&quot; below.</p>
</div>
<h2>Which path applies where</h2>
<p>OAC has two infra hubs (<code>oac-dev-infra</code>, <code>oac-prod-infra</code>). Each runs ACM and
ACM Observability (MCO) and hosts the control planes of its workload clusters
(<code>oac-dev-workload0</code>, <code>oac-dev-workload1</code>, <code>oac-prod-workload0</code>, ...).</p>
<table>
<thead>
<tr>
<th>You want to alert on</th>
<th>Rule object</th>
<th>Where it is evaluated</th>
<th>Applies on OAC to</th>
</tr>
</thead>
<tbody>
<tr>
<td>A platform metric of one cluster (nodes, operators, kube-state-metrics)</td>
<td><code>AlertingRule</code> (<code>monitoring.openshift.io/v1</code>) in <code>openshift-monitoring</code></td>
<td>That cluster's platform Prometheus</td>
<td>Every cluster, hubs and workload clusters</td>
</tr>
<tr>
<td>A metric of a project on a workload cluster</td>
<td><code>PrometheusRule</code> (<code>monitoring.coreos.com/v1</code>) in the project namespace</td>
<td>User workload monitoring (Thanos Ruler) on that cluster</td>
<td>Workload clusters only. User workload monitoring is not enabled on the hubs</td>
</tr>
<tr>
<td>A metric across the fleet</td>
<td>Entry in the <code>thanos-ruler-custom-rules</code> ConfigMap on the hub</td>
<td>MCO's <code>observability-thanos-rule</code> on the hub</td>
<td>Both hubs, for every cluster that sends metrics to MCO</td>
</tr>
<tr>
<td>Log contents</td>
<td><code>AlertingRule</code> (<code>loki.grafana.com/v1</code>)</td>
<td>The LokiStack ruler</td>
<td><strong>Nowhere yet.</strong> Logging is not on <code>main</code>, and the open logging change does not enable the ruler</td>
</tr>
</tbody>
</table>
<p>Where each component comes from in oac-apps:</p>
<ul>
<li><strong>User workload monitoring</strong> is switched on by the <code>user-workload-monitoring</code>
chart (<a href="https://github.com/CCI-MOC/oac-apps/blob/1260a865c69c1851a75b781df53ae40ca41e6be1/charts/user-workload-monitoring/templates/enable.yaml"><code>enable.yaml</code></a>), which the <code>all-managed-clusters</code>
ApplicationSet sends to every managed cluster
(<a href="https://github.com/CCI-MOC/oac-apps/blob/1260a865c69c1851a75b781df53ae40ca41e6be1/applicationsets/templates/managed/all-managed-clusters.yaml"><code>all-managed-clusters.yaml</code></a>). It is not in
<code>hub-components</code> (<a href="https://github.com/CCI-MOC/oac-apps/blob/1260a865c69c1851a75b781df53ae40ca41e6be1/applicationsets/templates/hub/hub-components.yaml"><code>hub-components.yaml</code></a>), so the hubs run
platform monitoring only.</li>
<li><strong>MCO</strong> is deployed to both hubs by the <code>acm-observability</code> chart
(<a href="https://github.com/CCI-MOC/oac-apps/blob/1260a865c69c1851a75b781df53ae40ca41e6be1/charts/acm-observability/templates/multiclusterobservability.yaml"><code>multiclusterobservability.yaml</code></a>). It runs three Thanos rule
replicas and three Alertmanager replicas.</li>
<li><strong>Logging</strong> (Loki Operator, <code>cluster-logging</code> chart, LokiStack
<code>openshift-logging/logging-loki</code>) exists only on the branch of
<a href="https://github.com/schwesig/OA_oac-apps/pull/2">schwesig/OA_oac-apps#2</a>. The design is recorded in the
workload cluster logging ADR (<a href="https://github.com/schwesig/ops-docs/pull/1">schwesig/ops-docs#1</a>).</li>
</ul>
<h2>Alert on a metric value</h2>
<h3>Where the rule lives as code</h3>
<p>oac-apps contains no alerting rule today. A search of <a href="https://github.com/CCI-MOC/oac-apps/tree/1260a865c69c1851a75b781df53ae40ca41e6be1"><code>1260a86</code></a> for
<code>PrometheusRule</code>, <code>AlertingRule</code> and <code>custom_rules</code> finds nothing. ArgoCD
syncs every chart with <code>selfHeal: true</code>, so a rule created by hand with
<code>oc apply</code> stays until someone removes it, but it is not reviewed and is not
reproduced on a new cluster. Put rules in the chart that deploys the thing
they watch:</p>
<ul>
<li>A platform or project rule for one cluster: a template in the chart that
owns the component, so it reaches the same clusters as the component.</li>
<li>A fleet rule: the <code>acm-observability</code> chart, next to the
<code>observability-metrics-custom-allowlist</code> ConfigMap
(<a href="https://github.com/CCI-MOC/oac-apps/blob/1260a865c69c1851a75b781df53ae40ca41e6be1/charts/acm-observability/templates/metrics-custom-allowlist.yaml"><code>metrics-custom-allowlist.yaml</code></a>).</li>
</ul>
<p>No chart holds rules yet, so the first rule sets the pattern. See open point 4.</p>
<h3>Platform metric on one cluster</h3>
<p>Red Hat documents <code>AlertingRule</code>, not a <code>PrometheusRule</code> in
<code>openshift-monitoring</code>, for new platform rules: &quot;You must create the
AlertingRule object in the openshift-monitoring namespace. Otherwise, the
alerting rule is not accepted.&quot; It requires <code>cluster-admin</code>
(<a href="https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.22/html-single/managing_alerts/index#creating-new-alerting-rules_managing-alerts-as-an-administrator">OCP 4.22, Creating new alerting rules</a>).</p>
<pre><code class="language-yaml">apiVersion: monitoring.openshift.io/v1
kind: AlertingRule
metadata:
  name: oac-node-filesystem
  namespace: openshift-monitoring
spec:
  groups:
    - name: oac-node
      rules:
        - alert: OACNodeRootFilesystemAlmostFull
          expr: |
            node_filesystem_avail_bytes{mountpoint=&quot;/&quot;}
              / node_filesystem_size_bytes{mountpoint=&quot;/&quot;} &lt; 0.10
          for: 15m
          labels:
            severity: warning
          annotations:
            message: &quot;{{ $labels.instance }} has less than 10% free on /.&quot;
</code></pre>
<p>To change a rule that ships with the platform, do not copy it. Red Hat uses
<

... [18685 bytes truncated] ...

/configuring_user_workload_monitoring/index#enabling-the-platform-alertmanager-instance-for-user-defined-alert-routing_preparing-to-configure-the-monitoring-stack-uwm">OCP 4.22, Enabling the platform Alertmanager instance for
user-defined alert routing</a>). Log alerts follow the same
rule: they go to <code>openshift-monitoring</code> unless a user workload Alertmanager is
enabled (<a href="https://docs.redhat.com/en/documentation/red_hat_openshift_logging/6.6/html-single/configure/index#loki-about-alerting-recording-rules_configure-custom-logging-alerts">Logging 6.6</a>).</li>
<li><strong>Platform Alertmanager has no receivers.</strong> &quot;Alertmanager does not send
notifications by default&quot;
(<a href="https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.22/html-single/configuring_core_platform_monitoring/index#configuring-alert-notifications_configuring-alerts-and-notifications">OCP 4.22, Configuring alert notifications</a>). oac-apps does not
manage the <code>alertmanager-main</code> secret on any cluster, and the console shows
the <code>AlertmanagerReceiverNotConfigured</code> alert where none is set
(<a href="https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.22/html-single/configuring_core_platform_monitoring/index#configuring-alert-routing-console_configuring-alerts-and-notifications">Configuring alert routing in the console</a>).</li>
<li><strong>Forwarding to the hub.</strong> &quot;After you enable observability, alerts from your
OpenShift Container Platform managed clusters are automatically sent to
Alertmanager on your hub cluster&quot;
(<a href="https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#obs-forward-alerts">ACM 2.17, Forwarding your alerts</a>). oac-apps sets neither
<code>mco-disable-alerting</code> nor <code>mco-disable-uwl-alerting</code> on the
<code>MultiClusterObservability</code> resource (<a href="https://github.com/CCI-MOC/oac-apps/blob/1260a865c69c1851a75b781df53ae40ca41e6be1/charts/acm-observability/templates/multiclusterobservability.yaml"><code>multiclusterobservability.yaml</code></a>),
so forwarding is on by default
(<a href="https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#obs-disable-alert-forwarding">Disabling alert forwarding</a>,
<a href="https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#disable-uw-alert-forwarding">Disabling user workload alert forwarding</a>). Whether it
works on OAC has not been checked (open point 3).</li>
<li><strong>Hub Alertmanager has no receivers either.</strong> Receivers are added by
overriding &quot;the alertmanager-config secret in the
open-cluster-management-observability namespace&quot;
(<a href="https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#configure-obs-alerts">ACM 2.17, Configuring alerts for observability</a>). oac-apps
does not manage that secret. Silences go through <code>amtool</code> in the
<code>observability-alertmanager-0</code> pod
(<a href="https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#silence-alerts">ACM 2.17, Silencing alerts</a>).</li>
<li><strong>Hub alerts in the console.</strong> The COO Monitoring UI plugin can show the MCO
Alertmanager in the console through <code>spec.monitoring.acm</code>
(<a href="https://docs.redhat.com/en/documentation/red_hat_openshift_cluster_observability_operator/1-latest/html-single/ui_plugins_for_red_hat_openshift_cluster_observability_operator/index#coo-monitoring-ui-plugin-install_monitoring-ui-plugin">COO, Installing the Monitoring UI plugin</a>). oac-apps leaves it off,
<code>uiPlugin.acm.enabled: false</code>, &quot;until MCO is running&quot;
(<a href="https://github.com/CCI-MOC/oac-apps/blob/1260a865c69c1851a75b781df53ae40ca41e6be1/charts/cluster-observability-operator/values.yaml"><code>values.yaml</code></a>). Until it is on, fleet alerts are visible in
MCO's Grafana only.</li>
</ul>
<h2>Open points</h2>
<ol>
<li><strong>No receivers.</strong> Neither the platform Alertmanagers nor the MCO
Alertmanager have a receiver in oac-apps, so no alert reaches a person or a
ticket system. Someone has to decide where alerts go (mail, Slack, a pager)
and which Alertmanager owns that routing: per cluster, or centrally on the
hub through MCO forwarding. The secret then belongs in oac-apps through
External Secrets, like the other credentials.</li>
<li><strong>Log alerts need the ruler.</strong> The LokiStack in
<a href="https://github.com/schwesig/OA_oac-apps/pull/2">schwesig/OA_oac-apps#2</a> has no <code>spec.rules</code>. Enabling it needs a
label convention for <code>selector</code> and <code>namespaceSelector</code>, ruler capacity on
the <code>1x.demo</code> size used on dev, and a decision on whether project members
get <code>alertingrules.loki.grafana.com-v1-edit</code>. It should be added to that
change or to a follow-up once logging is enabled.</li>
<li><strong>Alert forwarding is not verified, and may be overwritten.</strong> MCO enables
forwarding by editing <code>cluster-monitoring-config</code> (and, for user workload
alerts, <code>user-workload-monitoring-config</code>) on each managed cluster. The
ACM 2.17 guide says these updates are reverted when forwarding is turned off
(<a href="https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.17/html-single/observability/index#obs-disable-alert-forwarding">Disabling alert forwarding</a>). The <code>user-workload-monitoring</code>
chart writes the same <code>config.yaml</code> key in both ConfigMaps with server-side
apply and <code>selfHeal: true</code> (<a href="https://github.com/CCI-MOC/oac-apps/blob/1260a865c69c1851a75b781df53ae40ca41e6be1/charts/user-workload-monitoring/templates/enable.yaml"><code>enable.yaml</code></a>,
<a href="https://github.com/CCI-MOC/oac-apps/blob/1260a865c69c1851a75b781df53ae40ca41e6be1/charts/user-workload-monitoring/templates/tuning.yaml"><code>tuning.yaml</code></a>). If ArgoCD reverts MCO's additions, forwarding
stops without an error. Check on a workload cluster whether the forwarding
settings survive a sync, and whether alerts from a workload cluster appear
in the hub Alertmanager. It is also not recorded whether the hubs forward
their own alerts.</li>
<li><strong>Where rules live.</strong> No chart holds alerting rules yet. The first rule
decides whether rules sit in the chart of the component they watch, as
suggested above, or in one dedicated chart per path.</li>
<li><strong>Project rule permissions.</strong> The <code>user-projects</code> chart does not bind
<code>monitoring-rules-edit</code>. Whether project members should create their own
metric and log alerts, or only through oac-apps, is not recorded.</li>
</ol>
<h2>See also</h2>
<ul>
<li><a href="https://github.com/CCI-MOC/ops-docs/pull/36">CCI-MOC/ops-docs#36</a>: MCO metrics allowlist ADR. Decides
which metrics a fleet rule can see.</li>
<li><a href="https://github.com/schwesig/ops-docs/pull/1">schwesig/ops-docs#1</a>: workload cluster logging ADR (Proposed).</li>
<li><a href="https://github.com/schwesig/OA_oac-apps/pull/2">schwesig/OA_oac-apps#2</a>: the logging implementation (open).</li>
<li><a href="https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.22/html-single/managing_alerts/index#monitoring-accessing-the-alerting-ui_managing-alerts-as-an-administrator">OCP 4.22, Managing alerts</a>: silences, alert details,
rule management.</li>
</ul>

  </main>
  <footer>
    <p>
      Revision
      <a href="https://github.com/cci-moc/ops-docs/commit/28e9ed791a5fb4420280704ba99bc130a57b3ea9">28e9ed79</a>
      @ 2026-10-20 20:25:31 +0200
    <span class="noprint"> |
    <a href="https://github.com/cci-moc/ops-docs/blob/main/./docs/2026-10-20-alerting-howto/README.md?plain=1">View page source</a>
    </span>
    </p>
  </footer>
</body>
</html>
Evidence: YAML examples parsed into rule objects

AlertingRule monitoring.openshift.io/v1 OACNodeRootFilesystemAlmostFull; PrometheusRule monitoring.coreos.com/v1 WebDeploymentUnavailable; ConfigMap thanos-ruler-custom-rules -> OACClusterNotSendingMetrics; LokiStack openshift-logging/logging-loki; AlertingRule loki.grafana.com/v1 WebLogsManyErrors

block 1: AlertingRule monitoring.openshift.io/v1 ns=openshift-monitoring rules=OACNodeRootFilesystemAlmostFull: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
block 2: PrometheusRule monitoring.coreos.com/v1 ns=project-example rules=WebDeploymentUnavailable: kube_deployment_status_replicas_available{deployment="web"} == 0
block 3: ConfigMap v1 ns=open-cluster-management-observability rules=
block 4: LokiStack loki.grafana.com/v1 ns=openshift-logging rules=
block 5: AlertingRule loki.grafana.com/v1 ns=project-example rules=WebLogsManyErrors: sum(count_over_time({kubernetes_namespace_name="project-example", kubernetes_container_name="web"} |= "ERROR" [5m])) > 10
nested custom_rules.yaml: OACClusterNotSendingMetrics: absent_over_time(cluster:cpu_usage_cores:sum{cluster="oac-dev-workload0"}[30m])
- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (10m2s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed ✅
  • ⚠️ docs/2026-10-20-alerting-howto/README.md:184 - The fleet section tells readers to add openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus to a project PrometheusRule so its alerts get forwarded to the hub. It does not say that with this label the rule is evaluated only by the user-workload Prometheus, which cannot query platform metrics. Here is how that goes wrong: a reader adds the label to the doc's own project example (kube_deployment_status_replicas_available{deployment=&#34;web&#34;} == 0, line 126). That is a kube-state-metrics series scraped by platform Prometheus, so the expression returns nothing and the alert never fires, with no error. Fix: add one sentence saying leaf-prometheus rules only see metrics scraped by user workload monitoring in that project, so the example rule would stop working with the label.
  • ℹ️ docs/2026-10-20-alerting-howto/README.md:191 - The check step uses oc get alertingrule -n openshift-monitoring. Once the Loki Operator is installed (workload clusters and the dev hub under the logging change), alertingrule exists in two API groups: monitoring.openshift.io and loki.grafana.com. The short name may then resolve to the Loki CRD and list nothing, even though the platform rule exists. The log section (line 304) already qualifies its group (alertingrule.loki.grafana.com). Use alertingrules.monitoring.openshift.io here so the two match.

🔧 Fix applied.
✅ Re-checked - no issues remain.

🔧 **Test** - 1 issue found → auto-fixed ✅
  • ⚠️ docs/2026-10-20-alerting-howto/README.md:16 - The IMPORTANT callout links to #where-alerts-end-up-today and #open-points. The ops-docs site renders Markdown with markdown-it plus github-alerts only, so headings get no id attributes, and on the published page these links do nothing (they work when viewed on GitHub). There are two ways to fix it: replace the links with plain section names in the doc, or add a heading-anchor plugin such as markdown-it-anchor to eleventy.config.js. The second is a site-wide change outside this docs-only scope.
  • Live validation: ✅ go - 4 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Reader opens the new doc on the ops-docs site and sees how to alert on metric values and on log contents ✅ pass live npm run build + served _site; alerting-howto-rendered.png
Reader finds the doc from the Technical Documentation index ✅ pass live docs-index.png
Adversarial: Prometheus {{ }} templates in the examples survive the Liquid template engine ✅ pass live Built HTML contains {{ $labels.instance }} verbatim; screenshot shows it in the AlertingRule example
Reader copies the YAML examples and gets well-formed objects ⏸️ untested no The previous payload did not establish a live result. The YAML was only parsed offline with js-yaml and was never applied to a real OpenShift/ACM/Loki cluster: no oc, kubectl or promtool is on PATH an…
Recorded review fixes: leaf-prometheus scope caveat and API-group-qualified alertingrule check ✅ pass live Rendered page, 'Metric across the fleet' and 'Check that a metric alert fires' sections in alerting-howto-rendered.png
Reader clicks the in-page links in the IMPORTANT callout and jumps to the referenced section ⏸️ untested no The previous payload did not establish a live click-through result. Reading the built HTML showed href="#where-alerts-end-up-today" and href="#open-points" but no id attributes on any heading, because…
  • git archive HEAD into a throwaway temp dir, then npm ci, npm run validate-frontmatter, npm run build (Eleventy 3.1.7)
  • Inspected the built _site/docs/2026-10-20-alerting-howto/index.html: callouts render as markdown-alert-important and markdown-alert-warning, the {{ $labels.instance }} template is kept, no [text][ref] reference links are left unresolved, and checked whether in-page anchor targets exist
  • Checked that _site/docs/index.html lists the new doc
  • Parsed every ```yaml block with js-yaml (including the nested custom_rules.yaml in the ConfigMap) and printed apiVersion, kind, name, namespace, labels and rule expressions
  • Read the leaf-prometheus note and the oc get alertingrule.monitoring.openshift.io check step to confirm the recorded review fixes
  • Served _site with python3 -m http.server and took headless Firefox screenshots (throwaway profile) of the doc page and the docs index

🔧 Fix applied.
✅ Re-checked - no issues remain.

  • Live validation: ✅ go - 5 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Operator opens the docs site and finds the new alerting how-to listed and rendered ✅ pass live Eleventy build wrote docs/2026-10-20-alerting-howto/index.html; it is served with HTTP 200 and listed on /docs/; alerting-howto-full.png
Reader finds instructions for alerting on metric values (platform, project and fleet-wide) and on log contents, each with how to check that it fires ✅ pass live Rendered headings: Alert on a metric value (Platform metric, Project metric, Metric across the fleet, Check that a metric alert fires), Alert on log contents (Enable the ruler, Write the rule, Check t…
Rendering edge case: the Prometheus {{ $labels.instance }} template in an example is shown literally and does not break the Liquid build ✅ pass live Build succeeded; the rendered HTML contains labels.instance in the code block (templateEngineOverride: md works)
IMPORTANT callout uses plain section names and the published page has no dead in-page anchor links ✅ pass live No href="#..." in the rendered page; the callout reads 'See the sections "Where alerts end up today" and "Open points" below.' (alerting-howto-top.png)
Round-1 fixes: the leaf-prometheus scope sentence is present and the check command uses alertingrule.monitoring.openshift.io ✅ pass live Rendered HTML has the sentence 'evaluated only by the user workload Prometheus ... would never fire with the label' and oc get alertingrule.monitoring.openshift.io -n openshift-monitoring
A reader copying the YAML examples gets valid manifests with the intended kinds and rules ⏸️ untested no The prior payload did not establish a live result. The YAML was only parsed offline with js-yaml. No cluster, oc, promtool or logcli is available, so the manifests and the PromQL/LogQL expressions wer…
  • npm ci (inside the worktree; node_modules removed afterwards)
  • npm run validate-frontmatter
  • npx eleventy --output=&lt;evidence&gt;/site (real site build)
  • Checked the built page: headings, callout classes, no in-page href=&#34;#...&#34; links, literal {{ $labels.instance }} in the code block, API group of the alertingrule commands, leaf-prometheus sentence
  • Served the built site with python3 -m http.server 18765 and confirmed the page and its entry on /docs/ return 200
  • firefox --headless --screenshot full-page capture of the rendered doc (chrome-devtools-axi bridge failed, so Firefox was used instead)
  • Parsed every fenced YAML block and the nested custom_rules.yaml with the repo's js-yaml
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant