Skip to content

Update README observability alert rules for stackdriver metrics #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shlbatra merged 1 commit into from
Jul 1, 2026
Merged

Update README observability alert rules for stackdriver metrics #59

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 15 additions & 12 deletions Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ scripts/
docs/ # Design docs and plans
observability/ # Prometheus + Grafana monitoring stack
├── otel-collector.yml # OTel Collector: receive OTLP, export to Prometheus (local dev only)
├── prometheus.yml # Scrape config (stackdriver-exporter + OTel Collector)
├── prometheus.yml # Scrape config (stackdriver-exporter)
├── alert_rules.yml # Alert rules (error rates, latency, cost anomalies)
├── alertmanager.yml # Notification routing (Slack, PagerDuty)
├── docker-compose.observability.yml # Observability stack (stackdriver-exporter bridges Cloud Monitoring)
Expand Down Expand Up @@ -380,17 +380,20 @@ docker compose -f docker-compose.observability.yml down

### Alert Rules

Alerts are defined in `observability/alert_rules.yml` and loaded by Prometheus:
- **HighPredictionErrorRate** — error rate > 5% for 5 minutes
- **NoPredictionsFlowing** — zero predictions for 10 minutes
- **HighPredictionLatency** — p95 latency > 1 second
- **FeatureFetchFailureSpike** — feature fetch failures > 10/5min
- **HighParseErrorRate** — parse error rate > 10%
- **PredictionBatchFailureSpike** — batch prediction failures spiking
- **DataflowWorkerCountSpike** — vCPU count > 2x 7-day average
- **HighPubSubBacklog** — undelivered messages > 10k

Alertmanager routing is configured in `observability/alertmanager.yml` (default receiver has no notification targets — configure Slack/PagerDuty webhooks for production).
Alerts are defined in `observability/alert_rules.yml` and loaded by Prometheus. All rules use `stackdriver_*` metrics from the stackdriver-exporter:

| Alert | Severity | Condition |
|---|---|---|
| `HighCloudRunErrorRate` | critical | Cloud Run 5xx rate > 5% for 5 min |
| `NoPredictionsFlowing` | critical | Zero `prediction_success` for 10 min |
| `HighPredictionLatency` | warning | Mean prediction latency > 1 second for 5 min |
| `FeatureFetchFailureSpike` | warning | `fetch_failure` + `fetch_retry` rate > 10/5min |
| `PredictionRetrySpike` | critical | `prediction_retry` rate > 5/5min |
| `ParseToFetchDropoff` | warning | > 10% of parsed messages not reaching fetch stage |
| `DataflowWorkerCountSpike` | warning | vCPU count > 2x 7-day average for 30 min |
| `HighPubSubBacklog` | warning | Undelivered messages > 10k for 15 min |

View alert status at http://localhost:9090/alerts (inactive/pending/firing). Alertmanager routing is configured in `observability/alertmanager.yml` (default receiver has no notification targets — configure Slack/PagerDuty webhooks for production).

## Development

Expand Down
Loading