Evaluate in-application rate limiting / concurrency caps on the server

[martsokha]

Context

Companion to #260 (edge/infra rate limiting). That issue argues per-client throttling belongs at the gateway/LB/CDN. This issue tracks the separate question of whether the server itself should also carry rate-limiting or concurrency controls — and if so, of what kind — since edge limits and in-app limits protect against different failure modes.

Why in-app limits might still be warranted

• Deployments without a capable edge. Self-hosted installs may have no gateway doing per-client limits; the server would then be the only line of defense.
• Resource-shaped, not rate-shaped, limits. The incident that started this was concurrency exhaustion of the DB pool, not raw request rate. An edge rate-limiter (requests/sec) does not directly bound concurrent in-flight expensive operations. A server-side bounded-concurrency semaphore on the expensive paths (upload, redact/inference) caps simultaneous heavy work regardless of arrival rate — something the edge cannot see.
• Fairness by identity. The server knows the authenticated account/token; it can enforce per-account concurrency fairness that an IP-based edge limiter cannot.

Options to weigh

1. Bounded concurrency on expensive endpoints (a tokio::sync::Semaphore per heavy route, or a shared work budget) → excess requests get 503 + Retry-After fast instead of piling onto the pool. Smallest, most targeted; complements the connection-hold fixes.
2. Per-account/token token-bucket middleware (in-memory or NATS/Redis-backed for multi-instance) → true in-app rate limiting. Heavier; needs shared state to work across horizontally-scaled instances.
3. Rely on edge only (Rate limiting / abuse protection at the edge (infra, not app) #260) and keep the server limited to graceful degradation (finite pool-acquire timeout → 503).

Recommendation to discuss

Prefer (1) bounded concurrency on upload + redact as the in-app measure — it directly bounds the resource that actually gets exhausted and needs no shared store — and leave general per-client rate limiting to the edge (#260). Full in-app token-bucket (2) only if we must support edge-less deployments.

Depends on / relates to: the connection-hold release work (releasing DB connections across I/O) and #260.