Repository navigation
fix: refuse to start when a fixed-size pool can't hold the subscribers' connections - #230
Merged
Merged
Conversation
Benchmark gate✅ gate passed
Gated (fails the build): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For 0.15.0, no issue:
broker.start()now raises when a fixed-size engine pool has fewer connections than the subscribers hold.Why
Each subscriber holds
1 + max_workerspooled connections for as long as it runs: the fetch connection and one writer per worker. The LISTEN connection is a raw asyncpg connection outside the pool, so it doesn't count.Below that total,
start()used to return normally. The loops then waited on each other for connections, each checkout timed out afterpool_timeout(30 s by default), and the subscribers retried forever with a stream of reconnect ERROR logs while consuming nothing. Below the subscribers' own total this is always a configuration bug, so it now fails at startup with a message that gives the number needed and what the pool allows.The check
OutboxBroker._check_pool_capacity()runs first instart(). It sums1 + max_workersoverself.subscribers, so subscribers added through routers count too. It raisesRuntimeErrorwhen the engine's pool is aQueuePool(including the async adapted pool) andpool_size + max_overflowis below that sum.It's skipped for:
NullPool,StaticPooland any other pool class;max_overflow=-1, which is also whatpool_size=0becomes;TestOutboxBroker, which replacesstart().Following the rule from the grilling, it doesn't guess how many connections the application's own sessions need. The docs ask for room on top.
The private attribute: SQLAlchemy has no public accessor for the overflow limit.
pool.overflow()is the current count, andstatus()doesn't include the limit. So the check reads_max_overflow:QueuePool.recreate()passes it back to the constructor, so a quiet rename is unlikely.getattr, and anything that isn't anintskips the check. A future change can only turn the check off, never break startup.test_queue_pool_exposes_the_overflow_limit_the_pool_check_readsasserts that the attribute exists and matchesmax_overflow. CI runs it on both the newest and the floor SQLAlchemy, so a rename fails our CI instead of disabling the check silently.The subscriber reports its own count through a small
_pooled_connectionsproperty, so the number lives next to the code that opens those connections.Tests
The
start()test and the router test failed before the change: an undersized pool started without an error. The other cases cover a pool that fits exactly,NullPool, unlimited overflow,pool_size=0, a broker without an engine, and the attribute pin.Also in this PR: a flaky test from #223
test_loop_mode_buffered_deletes_flush_before_the_lease_expiresfailed about 1 run in 15 locally. Its margins were thin: a 0.4 s lease, while a row could wait about 0.3 s queued in memory before its handler ran, so under CPU load a lease expired and a row was handled twice.The second commit widens the margins to a 1.0 s lease and 60 rows, so the run still takes longer than one lease. With the age bound from #223 removed, the widened test fails 3 out of 3. With it in place, it passes 30 out of 30. No library code changes in that commit.
Docs
The connection-budget section of
subscriber.md, the production checklist and the troubleshooting entry used to say an undersized pool doesn't blockstart(). They now describe the startup error. The runtime starvation symptom stays documented for the case where the subscribers fit but the application's own sessions take the remaining connections.Verification
just bench-checkpasses. It starts the broker for real, so the check accepts the benchmark's default pool.ruffis clean,tyshows the same 6 diagnostics asmain, and the docs build.Release note (0.15.0, breaking for misconfigured setups)