Release in-flight jobs when the async supervisor's shutdown times out

[hannesfostie]

What we hit

We run Solid Queue 1.7.0 in async mode (bin/jobs, so workers are threads inside the supervisor's process) on Kubernetes. Whenever a deploy replaced the pod while a long job was running, the job ended up failed with SolidQueue::Processes::ProcessPrunedError about five minutes later, instead of going back to its queue. We had 14 of these over two days on our staging cluster, and each one lined up with a rollout.

The old pod's logs always ended the same way: the worker logged Release claimed job for one of its two claimed jobs, then nothing more, and no Shutdown Supervisor(async). The new pod then pruned that worker's registration and failed both jobs, including the one that had just been logged as released.

Why

On TERM, AsyncSupervisor#perform_graceful_termination stops the threads and waits shutdown_timeout. Each worker waits the same shutdown_timeout for its pool and then deregisters. Deregistering destroys its process row and releases its claimed executions in one transaction (after_destroy :release_all_claimed_executions).

The supervisor starts its wait slightly before the workers start theirs, so with a job that outlives the timeout it usually runs out first. It then calls exit! while the worker is still inside that transaction, and the release it had started is rolled back. exit! also skips the supervisor's own shutdown callbacks, so nothing else deregisters the threads. Their rows stay, and later the claimed jobs are failed instead of released.

The existing test for this case accepted either outcome and described the race in a comment. In fork mode the same situation ends with the jobs back in their queues: the fork supervisor quits its forks and then deregisters itself, which deregisters them too. That's also what #422 describes as the expected outcome of a deploy.

The change

When the graceful wait runs out with threads still alive, the async supervisor now deregisters itself before exit!. This deregisters the threads still running and releases their claimed jobs, as in fork mode.

The deregistration goes through a freshly loaded copy of the supervisor's registration. The supervisor's own copy shares the threads' registrations in memory (has_many_inversing), and a thread that has already deregistered leaves a destroyed, frozen record behind. My first version used the in-memory copy, and it failed intermittently on PostgreSQL with FrozenError: can't modify frozen attributes.

QUIT is unchanged. The async lifecycle test documents that it exits without cleanup, and I didn't want to change that here. The README's signal section says jobs in flight are returned to their queues on QUIT as well, which only holds in fork mode, so that may be worth a follow-up.

Tests

test "term supervisor exceeding timeout while there are jobs in-flight" in the async lifecycle test now expects the job to be ready and a clean termination, like its fork-mode counterpart, instead of accepting either claimed or ready. That's the reproduction:

• Without the change it failed 3 out of 3 runs on SQLite and 3 out of 3 on PostgreSQL.
• With it, the async lifecycle file passed 5 out of 5 runs on each, and the full suite passes on both.

I couldn't run MySQL locally, because mysql2 doesn't build on my machine.

🤖 Generated with Claude Code

[hannesfostie]

One thing I didn't ask Claude to add here is a timing change. We could make the supervisor's timeout wait for shutdown_timeout plus a few seconds or so of grace period. It wouldn't be a perfect fix on its own, but I think it may be worth considering.

Happy to add that if you think it's worth it

[hannesfostie]

As best as I can tell, CI failures will be fixed after #813 makes it in. I can rebase when that happens.