feat: paginate delete blockers

[grantfitzsimmons]

Fixes #7515

Authored by @melton-jason

Description

This PR addresses a major performance bottleneck in Specify's delete blockers endpoint, which affects large databases. Currently, when fetching delete blockers for a resource, Specify attempts to load all referencing records at once into a single Python object using Django's Collector. For highly-referenced resources (particularly Agent, which is referenced by createdByAgent and modifiedByAgent across 515+ relationships) this eager collection can:

• Cause request timeouts as the server struggles to load tens or hundreds of thousands of related records
• Tie up Gunicorn workers, preventing them from responding to other incoming requests
• Exhaust system memory, triggering the Linux OOM killer and crashing the Specify process (reproducible locally with 8 GiB of RAM allocated to Docker)
• Cause memory thrashing as the kernel aggressively swaps, degrading performance across the entire instance

At least two production instances have experienced kernel-invoked process restarts due to this issue. Even when requests don't crash, they can hold the Python GIL long enough to block other threads.

Solution: Paginated delete blockers

This PR introduces pagination support for the delete blockers endpoint. Instead of loading all referencing records at once, related record IDs are fetched in configurable chunks (limit/offset). The new approach:

• Returns the first page of delete blockers in under 1 second for highly-referenced agents (previously timed out or crashed)
• Fetches all delete blockers in ~3 seconds when needed, still a dramatic improvement over Django's Collector
• Splits results into immediate blockers (PROTECT / protect_with_blockers) and deferred blockers (CASCADE), enabling the frontend to prioritize what's shown
• Preserves the original endpoint at a legacy URL (old_delete_blockers) for backward compatibility

Changes

New endpoint behavior (/api/specify/delete_blockers/<model>/<id>/):

• Accepts limit (default: 20) and offset (default: 0) query parameters
• Returns JSON with two arrays:
  • results: immediate blockers — relationships using PROTECT or protect_with_blockers
  • next: deferred blockers — relationships using CASCADE
• Each entry includes table, field, ids, offset, limit, and a complete flag indicating whether all matching records for that relationship were fetched
• Passing limit=0 fetches all results (unpaginated)

Future work

Further frontend changes are needed to adopt the paginated approach in the UI. Additional backend optimizations (e.g., query-level improvements) will be detailed in follow-up issues.

Developer Testing instructions

1. Paginated endpoint: GET /api/specify/delete_blockers/agent/<id>/?limit=20&offset=0 — should return the first 20 IDs per blocking relationship in under 1 second
2. complete flag: Verify complete: true when all related records fit within limit, and complete: false otherwise
3. Legacy compatibility: GET /api/specify/old_delete_blockers/<model>/<id>/ should return the full, unpaginated result set identical to the previous behavior
4. No blockers: A resource with zero references should return empty results and next arrays
5. Stress-test with a highly-referenced Agent in a database of comparable size to Geneva, Edinburgh, or Berlin to confirm no timeouts or OOM conditions occur

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9e58bd40-7f61-41da-aa0d-bc93a6a43037

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch issue-7515-2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[grantfitzsimmons]

@melton-jason Could you base this instead on v7_12_0_7_base?