From a22cbbbbe98d3ad5d019eaab2f37634f2cddd423 Mon Sep 17 00:00:00 2001 From: Sahil Batra Date: Wed, 1 Jul 2026 18:43:08 -0400 Subject: [PATCH] Update README alert rules section for stackdriver metric names Alert rules were rewritten to use stackdriver_* metrics (PR #58). Updated the README to list the new alert names, severities, and conditions in a table. Also fixed prometheus.yml description to reflect otel-collector scrape target removal. Co-Authored-By: Claude Opus 4.6 --- Readme.md | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/Readme.md b/Readme.md index 37bb08f..df0e583 100644 --- a/Readme.md +++ b/Readme.md @@ -82,7 +82,7 @@ scripts/ docs/ # Design docs and plans observability/ # Prometheus + Grafana monitoring stack ├── otel-collector.yml # OTel Collector: receive OTLP, export to Prometheus (local dev only) -├── prometheus.yml # Scrape config (stackdriver-exporter + OTel Collector) +├── prometheus.yml # Scrape config (stackdriver-exporter) ├── alert_rules.yml # Alert rules (error rates, latency, cost anomalies) ├── alertmanager.yml # Notification routing (Slack, PagerDuty) ├── docker-compose.observability.yml # Observability stack (stackdriver-exporter bridges Cloud Monitoring) @@ -380,17 +380,20 @@ docker compose -f docker-compose.observability.yml down ### Alert Rules -Alerts are defined in `observability/alert_rules.yml` and loaded by Prometheus: -- **HighPredictionErrorRate** — error rate > 5% for 5 minutes -- **NoPredictionsFlowing** — zero predictions for 10 minutes -- **HighPredictionLatency** — p95 latency > 1 second -- **FeatureFetchFailureSpike** — feature fetch failures > 10/5min -- **HighParseErrorRate** — parse error rate > 10% -- **PredictionBatchFailureSpike** — batch prediction failures spiking -- **DataflowWorkerCountSpike** — vCPU count > 2x 7-day average -- **HighPubSubBacklog** — undelivered messages > 10k - -Alertmanager routing is configured in `observability/alertmanager.yml` (default receiver has no notification targets — configure Slack/PagerDuty webhooks for production). +Alerts are defined in `observability/alert_rules.yml` and loaded by Prometheus. All rules use `stackdriver_*` metrics from the stackdriver-exporter: + +| Alert | Severity | Condition | +|---|---|---| +| `HighCloudRunErrorRate` | critical | Cloud Run 5xx rate > 5% for 5 min | +| `NoPredictionsFlowing` | critical | Zero `prediction_success` for 10 min | +| `HighPredictionLatency` | warning | Mean prediction latency > 1 second for 5 min | +| `FeatureFetchFailureSpike` | warning | `fetch_failure` + `fetch_retry` rate > 10/5min | +| `PredictionRetrySpike` | critical | `prediction_retry` rate > 5/5min | +| `ParseToFetchDropoff` | warning | > 10% of parsed messages not reaching fetch stage | +| `DataflowWorkerCountSpike` | warning | vCPU count > 2x 7-day average for 30 min | +| `HighPubSubBacklog` | warning | Undelivered messages > 10k for 15 min | + +View alert status at http://localhost:9090/alerts (inactive/pending/firing). Alertmanager routing is configured in `observability/alertmanager.yml` (default receiver has no notification targets — configure Slack/PagerDuty webhooks for production). ## Development