[pull] main from tldraw:main - #729
Merged
Merged
Conversation
…nd fix the cost rules at month rollover (#10654) This PR fixes a bug where `AnthropicTokenRateAnomaly` kept paging on ordinary agent and eval traffic after two rounds of threshold tuning (#10579, #10582). It also fixes the two Anthropic cost rules going NoData at the start of each month, and documents how the three rules work. ### Before The token rule compared every `gen_ai_usage_tokens_total` series (key × model × token type, about 70 of them) against 5x its own 7-day average. Two things made that unfixable by tuning: - Usage is bursty and mostly idle (median hour under 10 tokens/sec), so any baseline is near zero and a normal working hour clears 5x. - Cache reads counted the same as uncached input despite costing a tenth as much. On Sep 1 and 2 the rule fired repeatedly on Opus 5 cache reads from the internal evals key, a known workload that spent about $50 that day. The cost rules subtracted `gen_ai_cost offset 1d` from the current value. That metric is month-to-date, not lifetime, and updates once a day at about 03:00 UTC. At the rollover the subtraction has nothing to diff against, so both rules have been in "Normal (NoData)" since Sep 1. Their descriptions also claimed to measure "the last 24h", which the metric cannot do. ### After `AnthropicTokenBurst` (renamed from `AnthropicTokenRateAnomaly`, same uid) sums the token rate across all series, excludes `input_cache_read`, and fires above a flat 1000 tokens/sec with no baseline. Checked against the last 14 days of data in Grafana: it is true only during the Aug 21 to 22 Sonnet 4.5 incident and today's uncached Sonnet burst, and stays quiet during the evals traffic (which peaks around 500 non-cache-read tokens/sec). The cost rules use `increase(gen_ai_cost[1d])`, which treats the monthly reset as a counter reset and counts from zero. Checked at Aug 21, Aug 22, Aug 24 and Sep 2: the spike rule fires on the two incident days and nowhere else. The descriptions now say they describe the last completed day. The scrape gap on the 1st is still a blind spot and is noted in the README. The README gains an "Anthropic rules" section covering the two metric semantics and the reasoning behind each rule, so the next tuning attempt starts from that instead of rediscovering it. ### Implementation notes - The alertname label changes with the rename. Notification policies are hand-managed and the Anthropic rules carry no receiver label, so they route through the default policy either way. Any silence keyed on the old name stops applying, which is the intent. - The $1000/day critical threshold and the $50 floor are unchanged. The Aug 22 day was $325, so $1000 has never come close; lowering it is a separate decision. - Nothing was pushed to Grafana from a local machine. The deploy workflow pushes on merge. ### Change type - [x] `bugfix` ### Test plan 1. `gcx resources validate -p internal/grafana/resources` passes locally (alert rules report as "skipped: client-side check only", expected per the README). 2. Each new PromQL expression was run against the live datasource at the timestamps above with `gcx metrics query`. 3. After merge, confirm in Grafana that `AnthropicTokenBurst` shows a single instance and the two cost rules leave NoData at the next 03:00 UTC update. ### Code changes | Section | LOC change | | -------------- | ---------- | | Config/tooling | +36 / -37 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )