diff --git a/internal/grafana/README.md b/internal/grafana/README.md index 92867aabece9..ff1d5d1c70a0 100644 --- a/internal/grafana/README.md +++ b/internal/grafana/README.md @@ -6,7 +6,7 @@ This directory currently provisions: - Eleven dashboards: `Zero on Fly.io — service health` (`zero-fly-health`), `Zero slow queries` (`slow-queries`), and `Zero-cache HTTP edge (Fly.io)` (`ni7hhgd`) in the `Zero` folder; `Anthropic scrape overview` in the `Anthropic` folder; `Dotcom events` (`ni5k8zc`), `Dotcom events V1` (`adgumkhou3chsd`), `File effect outbox`, `MCP server (shared boards)` (`mcp-server`), `MCP app sessions` (`kotx6wj`), and `Bemo analytics` in the `Dotcom` folder; and `Cloudflare platform metrics` (`nivs2rl`), which sits at the root rather than in a folder. The events pipeline currently reports from **staging** — set the environment variable accordingly when a dashboard looks empty. - All 13 Grafana-managed alert rules (Zero replication/errors/backups, Fly.io container health, Anthropic cost/usage, file room DO exceptions) -- The folders `Zero` (dashboards + 9 rules), `Dotcom` (6 dashboards + 1 rule), and `Anthropic` (1 dashboard + 3 cost rules) +- The folders `Zero` (dashboards + 9 rules), `Dotcom` (6 dashboards + 1 rule), and `Anthropic` (1 dashboard + 3 rules, see below) - The `discord` and `email` notification templates (`templates/*.gotmpl`, defining `tldraw.discord.title` / `tldraw.discord.text` and `tldraw.email.subject` / `tldraw.email.message`), which compress alert notifications to severity + summary + links (no Silence link on Discord: silence URLs carry one matcher per label and blow Discord’s 2000-char message cap on multi-instance alerts, truncating mid-link) and explain `DatasourceNoData` as a likely metrics-source outage. Templates can't go through `gcx resources push` — gcx ignores the `notifications.alerting.grafana.app` API group — so they live in `templates/` instead of `resources/` and the deploy workflow PUTs them through the classic provisioning API. Each contact point's optional Title/Subject and Message fields reference these templates; those references are set by hand (contact points stay hand-managed), so the templates only take effect while a contact point points at them. There are two MCP dashboards, for two different servers. `MCP server (shared boards)` covers the public server on the sync worker at `POST /app/mcp` (`apps/dotcom/sync-worker`), which reads its protocol metrics from the `MEASURE` dataset. `MCP app sessions` covers the separate tldraw MCP app worker (`apps/mcp-app`) and its own `MCP_ANALYTICS` dataset. Neither one's panels belong on `Dotcom events` — that dashboard covers the sync worker's own events. @@ -19,6 +19,23 @@ Contact points, notification policies, and data sources are hand-managed in Graf Every provisioned dashboard carries the tags `provisioned` and `repo:tldraw/tldraw`, so anyone looking at it in Grafana knows where it's managed from. Add both tags when adopting a new dashboard into this tree. +## Anthropic rules + +The three rules in the `Anthropic` folder watch the Grafana Anthropic integration's scrape of the Anthropic Admin API. The two metrics behave very differently, and the rules are shaped around that: + +- `gen_ai_usage_tokens_total` is near real time and is labelled by API key, model, and token type (`input`, `output`, `cache_creation_5m`, `cache_creation_1h`, `input_cache_read`). About 70 series at the time of writing. This is the only intraday signal. +- `gen_ai_cost` is **month-to-date** spend in cents, with no API key label, updated **once a day at about 03:00 UTC**. It resets on the 1st, and has had no data at all on the 1st. Anything built on it describes the last completed day, sees a runaway up to 27 hours late, and is blind for a day or two at each month rollover. + +`AnthropicTokenBurst` (warning) fires when the summed rate across all series, excluding cache reads, exceeds 1000 tokens/sec over an hour (about 3.6M tokens/hour). Three deliberate choices, each learned the hard way: + +- **Summed, not per series.** A per-series rule makes one alert instance per key × model × token type, and every lightly used series has a tiny baseline that any real session clears. +- **No baseline.** Usage is bursty and mostly idle (the median hour is under 10 tokens/sec), so a 7-day average is close to zero and "5x the average" is a normal working hour. #10579 and #10582 tuned the multiplier and floor twice and it still paged on ordinary sessions. +- **Cache reads excluded.** They cost a tenth of uncached input and dominate agent and eval traffic. On 2026-09-01 and 09-02 the rule paged repeatedly on Opus 5 cache reads from the internal evals key, which spent about $50 that day. The floor is chosen so that workload stays quiet (it peaks around 500 non-cache-read tokens/sec) while a big-numbers bulk cold-outreach research run (`tldraw-internal`, `crm-dashboard-api`), roughly 2000 tokens/sec of uncached Sonnet 4.5 input at $20 to $25 an hour and about $500 over 2026-08-21 to 08-22, clears it within the hour. Those runs are expected work, so a page from this rule usually means "check whether someone started a bulk run" rather than "something is broken". + +`AnthropicDailyCostSpike` (warning) fires when the last completed day cost more than 1.5x the day before, with a $50 floor. `AnthropicHighCostThreshold` (critical) fires when the last completed day cost more than $1000. Both use `increase(gen_ai_cost[1d])` rather than `offset` subtraction so the monthly reset reads as a counter reset and counts from zero instead of going negative. The scrape gap on the 1st is still a blind spot: the first day of the month reads as roughly $0 because there is nothing to diff against. Because the metric only steps once a day, a firing rule stays firing until the next 03:00 update. + +All three are filter queries (`expr > threshold`), so the alert series only exists while the condition holds. `missingSeriesEvalsToResolve: 10` is what stops a single quiet minute from resolving and re-firing them. + ## Making a change 1. Edit the files under `resources/`. diff --git a/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/8451fbba-5835-56e2-b2f5-7f5e44c520b8.yaml b/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/8451fbba-5835-56e2-b2f5-7f5e44c520b8.yaml index 7aece59deb96..b678dbbbd44b 100644 --- a/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/8451fbba-5835-56e2-b2f5-7f5e44c520b8.yaml +++ b/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/8451fbba-5835-56e2-b2f5-7f5e44c520b8.yaml @@ -14,9 +14,10 @@ metadata: namespace: stacks-890472 spec: annotations: - description: Token processing rate of {{ $value | humanize }} tokens/sec is 5x - higher than the 7-day average for job {{ $labels.job }}. - summary: Anthropic API token rate anomaly detected. + description: 'Anthropic API usage over the last hour averaged {{ $value | humanize }} + tokens/sec excluding cache reads, above the 1000 tokens/sec floor (about 3.6M + tokens/hour). Check the scrape overview dashboard for which key and model.' + summary: Anthropic API token burst detected. execErrState: Ok expressions: prometheus_math: @@ -54,13 +55,9 @@ spec: type: prometheus uid: grafanacloud-prom expr: | - ( - rate(gen_ai_usage_tokens_total{job=~"integrations/anthropic/.*"}[1h]) - > - 5 * avg_over_time(rate(gen_ai_usage_tokens_total{job=~"integrations/anthropic/.*"}[1h])[7d:1h]) - ) - and - rate(gen_ai_usage_tokens_total{job=~"integrations/anthropic/.*"}[1h]) > 1000 + sum( + rate(gen_ai_usage_tokens_total{job=~"integrations/anthropic/.*", gen_ai_token_type!="input_cache_read"}[1h]) + ) > 1000 instant: true intervalMs: 1000 maxDataPoints: 43200 @@ -110,6 +107,6 @@ spec: severity: warning missingSeriesEvalsToResolve: 10 noDataState: Ok - title: AnthropicTokenRateAnomaly + title: AnthropicTokenBurst trigger: interval: 1m diff --git a/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/e75d2f6c-44cf-54bf-8a74-ced2e045fcb6.yaml b/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/e75d2f6c-44cf-54bf-8a74-ced2e045fcb6.yaml index dcd1a15f7a20..f617a720b682 100644 --- a/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/e75d2f6c-44cf-54bf-8a74-ced2e045fcb6.yaml +++ b/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/e75d2f6c-44cf-54bf-8a74-ced2e045fcb6.yaml @@ -14,8 +14,9 @@ metadata: namespace: stacks-890472 spec: annotations: - description: 'Cost over the last 24h of ${{ $value | humanize }} is more than - 1.5x the previous 24h.' + description: 'The last completed day cost ${{ $value | humanize }}, more than + 1.5x the day before. The cost metric updates once a day at about 03:00 UTC, + so this describes yesterday.' summary: Anthropic API daily cost spike detected. execErrState: Ok expressions: @@ -55,25 +56,12 @@ spec: uid: grafanacloud-prom expr: | ( - ( - sum(gen_ai_cost{job=~"integrations/anthropic/.*"}) - - - sum(gen_ai_cost{job=~"integrations/anthropic/.*"} offset 1d) - ) / 100 + sum(increase(gen_ai_cost{job=~"integrations/anthropic/.*"}[1d])) / 100 > - 1.5 * - ( - sum(gen_ai_cost{job=~"integrations/anthropic/.*"} offset 1d) - - - sum(gen_ai_cost{job=~"integrations/anthropic/.*"} offset 2d) - ) / 100 + 1.5 * sum(increase(gen_ai_cost{job=~"integrations/anthropic/.*"}[1d] offset 1d)) / 100 ) and - ( - sum(gen_ai_cost{job=~"integrations/anthropic/.*"}) - - - sum(gen_ai_cost{job=~"integrations/anthropic/.*"} offset 1d) - ) / 100 > 50 + sum(increase(gen_ai_cost{job=~"integrations/anthropic/.*"}[1d])) / 100 > 50 instant: true intervalMs: 1000 maxDataPoints: 43200 diff --git a/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/eec2ad9e-30b9-5c8f-802f-53e26ab8fdc9.yaml b/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/eec2ad9e-30b9-5c8f-802f-53e26ab8fdc9.yaml index 38d0e9eac256..1ebed9ec8a23 100644 --- a/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/eec2ad9e-30b9-5c8f-802f-53e26ab8fdc9.yaml +++ b/internal/grafana/resources/alertrules.v0alpha1.rules.alerting.grafana.app/eec2ad9e-30b9-5c8f-802f-53e26ab8fdc9.yaml @@ -14,8 +14,9 @@ metadata: namespace: stacks-890472 spec: annotations: - description: 'Cost over the last 24h of ${{ $value | humanize }} has exceeded - the $1000 threshold. Current cost: ${{ $value }}.' + description: 'The last completed day cost ${{ $value | humanize }}, above the + $1000 threshold. The cost metric updates once a day at about 03:00 UTC, so + this describes yesterday.' summary: Anthropic API cost threshold exceeded. execErrState: Ok expressions: @@ -54,11 +55,7 @@ spec: type: prometheus uid: grafanacloud-prom expr: | - ( - sum(gen_ai_cost{job=~"integrations/anthropic/.*"}) - - - sum(gen_ai_cost{job=~"integrations/anthropic/.*"} offset 1d) - ) / 100 > 1000 + sum(increase(gen_ai_cost{job=~"integrations/anthropic/.*"}[1d])) / 100 > 1000 instant: true intervalMs: 1000 maxDataPoints: 43200