Skip to content

Fiddle: release messages on shutdown and delete expired messages in the queue processor - #172

Merged
krishan711 merged 2 commits into
mainfrom
queue-shutdown-housekeeping
Oct 21, 2026
Merged

krishan711 merged 2 commits into
mainfrom
queue-shutdown-housekeeping

Conversation

@krishan711

@krishan711 krishan711 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Description

Two improvements to the message queues, prompted by how the SQL queue behaves in production:

Graceful shutdown

  • New abstract MessageQueue.release_message: hands a claimed message back to be claimed again straight away without using an attempt (SQL and Cosmos keep postCount; SQS and AQS make it visible again).
  • MessageQueueProcessor.run/run_batches now handle SIGTERM/SIGINT (and request_shutdown()): the message being processed finishes, claimed messages not yet started are released, and the idle sleep stops immediately. A second signal falls back to the default handler.
  • A message whose processing task is cancelled is released instead of waiting for its lease to expire (which also used up an attempt).
  • Previously every deploy killed the in-flight job, which then sat for 300s until its lease expired and counted as a failed attempt.

Housekeeping

  • New abstract MessageQueue.delete_expired_messages: SqlMessageQueue deletes succeeded/deduplicated rows older than retentionSeconds; other queues remove messages on completion and return 0.
  • The processor calls it every deleteExpiredMessagesIntervalSeconds (default 600), logging rather than raising on failure.
  • SqlMessageQueue.get_message_count no longer deletes rows; shouldSkipRemovingNonRetained is removed (breaking).

Also adds shouldProcessInParallel to run_batches so consumers can try processing a claimed batch concurrently.

Tests: release on every backend (including after a lost lease), shutdown finishing the current message and releasing the rest, a real SIGTERM stopping an idle run, a cancelled message being released without an attempt, and the processor deleting expired rows. 141 queue/lock/database tests pass locally against sqlite, Postgres, Cosmos, ElasticMQ and Azurite.

Note: make type-check on main currently reports 35 Unpack errors in core/store/*; they are identical without this change.

Checklist:

  • I have updated the CHANGELOG with a summary of my changes

@krishan711
krishan711 merged commit 0e88258 into main Oct 21, 2026
4 checks passed
@krishan711
krishan711 deleted the queue-shutdown-housekeeping branch October 21, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant