Stop page loads from spending the whole Cloud Armor rate-limit budget

[rcurranmoz]

An operator hit a bare 429 Too Many Requests page twice on 2026-09-17 simply by refreshing the dashboard. This fixes the cause on both sides: the page asks for far less, and the limit stops being able to blank the app.

What the load-balancer logs actually show

The denied minute was exactly 100 allowed, then 6 denied — hangar-armor's threshold is 100/60s per IP, and it counts the HTML document and hashed assets alongside API calls. Breakdown of those 100:

Path	Count
/api/fleet/pool-sources	35
/api/fleet/android-pool-sources	14
other API (28 distinct endpoints)	39
document + assets + favicon	12

A repeat 12 minutes later had the identical signature. In both cases the document was denied last — the API burst spent the budget, then the navigation was refused. That ordering is why the symptom is a blank error page rather than a degraded card.

So two unbatched per-pool loops were 49% of the entire budget. Both per-pool computations were already cached server-side, so the cost was purely request count — which is exactly what a per-IP limiter charges for.

Changes

Batch the per-pool loops. New /fleet/pool-sources-batch and /fleet/android-pool-sources-batch build on the same cache keys as the single-pool routes, so per-pool payloads are byte-identical and no card changes behaviour. Cold pools are computed with bounded parallelism rather than serially. The four call sites — Pools.tsx (pinned, linux/windows, android) and Overview.tsx (monitored) — now issue one request each: 49 → 3.

Cache android_pool_sources. It was the one uncached path: a live Taskcluster workers call (limit=1000) plus a task fetch per running task, per pool, 14 times per Pools load. Now cached like its hardware sibling.

In-flight GET dedupe in api.ts. Components that mount together request the same endpoint — Layout + Overview both want /fleet/summary; Workers + CommandPalette both want /fleet/pools. Identical concurrent GETs share one response; nothing is cached past settlement, so no semantics change.

Retry with jittered backoff on 429/502/503/504, honouring Retry-After. A throttled API call now degrades one card instead of erroring the page.

Split the Terraform limit in two. Denying an /api call degrades a card inside a loaded page; denying the document replaces the whole dashboard with Cloud Armor's error page. Those deserve different ceilings: API 1200/60s (~24 heavy page loads/min after batching), app shell 3000/60s.

IAP already restricts this origin to @mozilla.com, so these limits are DoS hygiene, not access control. And enforce_on_key = "IP" means corp VPN/NAT users share one budget — which is why the headroom matters more as the audience grows, not less.

Net effect: a heavy load goes ~100 → ~54 requests while the /api ceiling goes 100 → 1200.

Recorded but deliberately NOT changed here

The three OWASP rules are unreachable — 2000 XSS, 2001 SQLi, 2002 RFI. Cloud Armor enforces the first matching rule and stops; these rate-limit rules match every request. Verified in the LB logs — every request reports enforcedSecurityPolicy priority 1000, including a scanner walking /cgi-bin and /docSQL, which gets a 429 rather than a WAF 403. Today the rate limiter is the only Cloud Armor control actually running.

Activating them means reordering, which carries real false-positive risk and wants preview = true first to measure. That's its own change; there's a comment in lb.tf so the next reader isn't surprised.

Verification

• 28 backend tests pass (12 new in tests/test_pool_sources_batch.py).
• The new tests pin the fix rather than document it: reverting just the android cache wrapper fails test_android_single_route_is_now_cached.
• npm run build, terraform fmt -check, terraform validate all clean.
• eslint (6 problems) and ruff (7) at exact parity with main on every touched file — diffed per-file rather than trusting a total.

The request-count reduction is derived from the changed call sites plus the measured before-state; the after-number comes from LB logs post-deploy:

gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status=429' \
  --project=relops-dashboard --freshness=1d --limit=100 \
  --format="value(httpRequest.remoteIp,httpRequest.requestUrl)"

No window where traffic is denied

The plan removes one rule and adds two, which raises the fair question of what happens in between. The policy's own default rule is 2147483647 / allow ("Default allow"), so during the swap a request that matches no rate-limit rule falls through to allow. The failure direction is fail-open — briefly un-rate-limited, never denied — and IAP authenticates throughout, since it lives on the backend service and is independent of the security policy.

Worth one post-apply check: the new rules plan as preview = (known after apply). They must come back false. If either landed as true the rule would log instead of enforce, which is harmless but would mean no rate limiting at all.

Deploy note

The code half auto-deploys on merge. The Terraform half is a manual apply and is the part that makes the blank page impossible — merging alone roughly halves consumption but leaves the 100/min ceiling in place. A good plan touches only google_compute_security_policy.hangar; if it wants to change iap{} or the Cloud Run image, stop.

🤖 Generated with Claude Code