[stopachka]

A database outage can permanently stop the singleton WAL listener. During the October 18 outage, its cleanup path timed out acquiring a connection for the optional kick-wal insert. That exception escaped the listener loop before channel cleanup and retry, so a replacement server had to restore the listener.

Treat SQL failures from the optional wakeup as retryable while completing shutdown and closing the old channels. If the slot-state check cannot reach the database, wait for the next acquisition interval. Cancellation, interruption, and unexpected errors still propagate.

The regression exercises a failed cleanup insert, a failed subsequent slot check, successful reacquisition, closed channels, and explicit shutdown. It produces five failures against the original source and passes with this fix. The full singleton test namespace passes (18 tests, 50 assertions), and clj-kondo reports no warnings.

This draft has not been deployed. It repairs recovery after database failures; it does not prevent the Aurora OOM that initiated this incident.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 6aa0eddf-6e38-4709-90c5-e0e147dfaf89

📥 Commits

Reviewing files that changed from the base of the PR and between b97a116 and 90b9a31.

📒 Files selected for processing: 2

• server/src/instant/reactive/invalidator.clj
• server/test/instant/reactive/invalidator_singleton_test.clj

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

The invalidator now logs retryable SQL failures during WAL shutdown and active-slot checks. Active-slot checks defer acquisition after these failures. Tests cover listener recovery, shutdown completion, and propagation of interruption, cancellation, and unexpected exceptions.

Changes: Invalidator recovery

Layer / File(s)	Summary
Database retry handling and recovery tests server/src/instant/reactive/invalidator.clj, server/test/instant/reactive/invalidator_singleton_test.clj	The invalidator logs SQL failures that are not caused by interruption or cancellation. Shutdown continues after retryable WAL failures, and active-slot checks defer acquisition after retryable query failures. Tests cover listener reacquisition, worker shutdown, channel closure, explicit shutdown, and propagation of non-retryable exceptions.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: dwwoelfel

Merge Risk: ⚪ Minimal · up to 90b9a

The change keeps the WAL listener retrying after database outages and still lets interruption, cancellation, and unexpected errors propagate. No merge-blocking risk was identified in the supplied files.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly and concisely summarizes the main change: keeping the WAL listener retrying after database outages.
Description check		The description directly explains the database outage failure, the retry behavior, exception handling, test coverage, and scope of the changes.
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}