Release DB connections across slow I/O; fail fast on pool exhaustion

[martsokha]

The incident

A stress test with a single spammy client saturated the 10-connection DB pool and cascaded to 500s. Logs showed Connection acquisition took longer than expected elapsed=28s followed by request timeout exceeded → 500.

Root cause: handlers and the detection worker held a pooled connection across slow non-DB work (LLM inference, NATS object I/O, image processing). With a small pool and no acquire timeout, a few concurrent requests pinned every connection and everyone else hung until the request timeout killed them. An audit found this pattern in six places.

Release the connection across slow I/O

Every offender now scopes the pre-flight DB work, drops the connection before the slow phase, and re-acquires only for the final commit — the pattern already used by upload_file, chat send_message, and the connection-sync service.

Path	Was held across	Now
redact_detection (handler)	audit load + LLM anonymize + 4 object round-trips	connection released before all of it
DetectionWorker::detect	LLM analyze + NATS get/put	manages its own connection per phase
Avatar upload (account + workspace)	image decode/encode + NATS put	authorize, then release before processing
Detection-audit reads + redaction review	NATS get	resolve file row, release, then load
bulk_delete_files purge loop	N sequential NATS deletes under one connection	release batch connection; re-acquire a short-lived one per purge

To support this cleanly, RunBlobStore audit loading is split into a connection-bound resolve step (resolve_audit_file / resolve_review_file) and a connection-free load step (load_audit). This also deduplicates the object load/decode logic that was copied across load_analyzed_document and load_review_audit, and removes the now-unused load_analyzed_document.

Fail fast on pool exhaustion

• POSTGRES_CONNECTION_TIMEOUT now defaults to 10s (was unset = wait forever), comfortably below the 30s request timeout, so a starved request fails promptly instead of hanging.
• A pool wait timeout now maps to a retryable 503 Service Unavailable (new ErrorKind::ServiceUnavailable) instead of a misleading 500; a backend create/recycle timeout stays a 500.

Scope notes

• Rate limiting is intentionally not in this PR. Per the discussion, per-client throttling belongs at the edge/infra (Rate limiting / abuse protection at the edge (infra, not app) #260); the server's job is to hold shared resources briefly and degrade gracefully. In-app concurrency caps are tracked separately (Evaluate in-application rate limiting / concurrency caps on the server #261).
• A Retry-After header on the 503 would be a nice follow-up (the error path has no header mechanism today).

Testing

Full gate green: cargo check / clippy --all-targets --all-features --workspace -D warnings / fmt --check, unit tests (pool-config defaults updated to assert the finite timeout), doc build.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 26 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 16bcc530-f82a-43e0-b02f-237da30e1868

📥 Commits

Reviewing files that changed from the base of the PR and between 222838a and f5198ff.

📒 Files selected for processing (13)

• crates/nvisy-postgres/src/client/pg_config.rs
• crates/nvisy-postgres/src/error.rs
• crates/nvisy-postgres/src/lib.rs
• crates/nvisy-server/src/handler/accounts.rs
• crates/nvisy-server/src/handler/detection_audits.rs
• crates/nvisy-server/src/handler/detections.rs
• crates/nvisy-server/src/handler/error/http_error.rs
• crates/nvisy-server/src/handler/error/pg_error.rs
• crates/nvisy-server/src/handler/files.rs
• crates/nvisy-server/src/handler/redactions.rs
• crates/nvisy-server/src/handler/workspaces.rs
• crates/nvisy-server/src/service/detection/worker.rs
• crates/nvisy-server/src/service/run_blob_store.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}