Context
A stress test with a single client saturated the DB connection pool and caused a 500 cascade. Root cause was the server holding pooled connections across non-DB I/O (LLM inference, NATS streaming) — tracked/fixed separately. This issue is about the complementary concern: one client should not be able to monopolize shared capacity, handled where it belongs.
Decision: rate limiting belongs at the edge, not in the app
Per-client throttling (per IP / per account / per token) is cheapest and most robust at the API gateway / load balancer / CDN layer (e.g. Cloudflare, an ingress rate-limit, or an API gateway), where a rejected request never consumes app resources deciding to reject it. Building it into the Rust server would burn the very resources we're trying to protect.
The server's own responsibilities (separate work) are: release shared resources promptly (don't hold DB connections across I/O) and degrade gracefully under saturation (finite pool-acquire timeout → 503 + Retry-After instead of hanging to a 500). Those are being addressed in the connection-hold PR.
Scope for this issue (ops/infra)
- Choose the enforcement point (gateway / LB / CDN) for the deployment.
- Define per-client limits (requests/sec and, importantly, concurrent in-flight requests, since the expensive paths are concurrency-bound, not rate-bound).
- Tighter limits on expensive endpoints (upload, redact) than on cheap reads.
- Return
429 with Retry-After.
Explicitly out of scope
In-application rate-limiting middleware — the goal is resilience against abuse via infra, not building a limiter into the API tier. Revisit only if edge enforcement proves insufficient.
Context
A stress test with a single client saturated the DB connection pool and caused a 500 cascade. Root cause was the server holding pooled connections across non-DB I/O (LLM inference, NATS streaming) — tracked/fixed separately. This issue is about the complementary concern: one client should not be able to monopolize shared capacity, handled where it belongs.
Decision: rate limiting belongs at the edge, not in the app
Per-client throttling (per IP / per account / per token) is cheapest and most robust at the API gateway / load balancer / CDN layer (e.g. Cloudflare, an ingress rate-limit, or an API gateway), where a rejected request never consumes app resources deciding to reject it. Building it into the Rust server would burn the very resources we're trying to protect.
The server's own responsibilities (separate work) are: release shared resources promptly (don't hold DB connections across I/O) and degrade gracefully under saturation (finite pool-acquire timeout → 503 + Retry-After instead of hanging to a 500). Those are being addressed in the connection-hold PR.
Scope for this issue (ops/infra)
429withRetry-After.Explicitly out of scope
In-application rate-limiting middleware — the goal is resilience against abuse via infra, not building a limiter into the API tier. Revisit only if edge enforcement proves insufficient.