Skip to content

fix: refuse to start when a fixed-size pool can't hold the subscribers' connections - #230

Merged
lesnik512 merged 2 commits into
mainfrom
fix/startup-pool-check
Oct 17, 2026
Merged

lesnik512 merged 2 commits into
mainfrom
fix/startup-pool-check

Conversation

@lesnik512

Copy link
Copy Markdown
Member

For 0.15.0, no issue: broker.start() now raises when a fixed-size engine pool has fewer connections than the subscribers hold.

Why

Each subscriber holds 1 + max_workers pooled connections for as long as it runs: the fetch connection and one writer per worker. The LISTEN connection is a raw asyncpg connection outside the pool, so it doesn't count.

Below that total, start() used to return normally. The loops then waited on each other for connections, each checkout timed out after pool_timeout (30 s by default), and the subscribers retried forever with a stream of reconnect ERROR logs while consuming nothing. Below the subscribers' own total this is always a configuration bug, so it now fails at startup with a message that gives the number needed and what the pool allows.

The check

OutboxBroker._check_pool_capacity() runs first in start(). It sums 1 + max_workers over self.subscribers, so subscribers added through routers count too. It raises RuntimeError when the engine's pool is a QueuePool (including the async adapted pool) and pool_size + max_overflow is below that sum.

It's skipped for:

  • NullPool, StaticPool and any other pool class;
  • an unlimited max_overflow=-1, which is also what pool_size=0 becomes;
  • a broker without an engine;
  • TestOutboxBroker, which replaces start().

Following the rule from the grilling, it doesn't guess how many connections the application's own sessions need. The docs ask for room on top.

The private attribute: SQLAlchemy has no public accessor for the overflow limit. pool.overflow() is the current count, and status() doesn't include the limit. So the check reads _max_overflow:

  • It has the same name and meaning from 2.0.0 (our floor) to 2.1.5 (the newest release), and SQLAlchemy's own QueuePool.recreate() passes it back to the constructor, so a quiet rename is unlikely.
  • It's read with getattr, and anything that isn't an int skips the check. A future change can only turn the check off, never break startup.
  • test_queue_pool_exposes_the_overflow_limit_the_pool_check_reads asserts that the attribute exists and matches max_overflow. CI runs it on both the newest and the floor SQLAlchemy, so a rename fails our CI instead of disabling the check silently.

The subscriber reports its own count through a small _pooled_connections property, so the number lives next to the code that opens those connections.

Tests

The start() test and the router test failed before the change: an undersized pool started without an error. The other cases cover a pool that fits exactly, NullPool, unlimited overflow, pool_size=0, a broker without an engine, and the attribute pin.

Also in this PR: a flaky test from #223

test_loop_mode_buffered_deletes_flush_before_the_lease_expires failed about 1 run in 15 locally. Its margins were thin: a 0.4 s lease, while a row could wait about 0.3 s queued in memory before its handler ran, so under CPU load a lease expired and a row was handled twice.

The second commit widens the margins to a 1.0 s lease and 60 rows, so the run still takes longer than one lease. With the age bound from #223 removed, the widened test fails 3 out of 3. With it in place, it passes 30 out of 30. No library code changes in that commit.

Docs

The connection-budget section of subscriber.md, the production checklist and the troubleshooting entry used to say an undersized pool doesn't block start(). They now describe the startup error. The runtime starvation symptom stays documented for the case where the subscribers fit but the application's own sessions take the remaining connections.

Verification

  • 692 tests pass in docker at 100% coverage.
  • just bench-check passes. It starts the broker for real, so the check accepts the benchmark's default pool.
  • ruff is clean, ty shows the same 6 diagnostics as main, and the docs build.
  • The check runs once, at startup, so there's no runtime cost.

Release note (0.15.0, breaking for misconfigured setups)

broker.start() now raises RuntimeError when the engine uses a fixed-size pool whose pool_size + max_overflow is below the connections the subscribers hold (1 + max_workers each). Those setups never consumed: they timed out on pool checkout and retried forever. Raise the pool size, or lower max_workers.

@github-actions

Copy link
Copy Markdown

Benchmark gate

✅ gate passed

scenario msg/s delete/msg WALrec/msg WALB/msg fpi upd del dead_tup
consumer/w1/b10 1280 1.000 6.69 913 242 5000 5000 10000
consumer/w1/b100 1365 1.000 7.09 1143 243 5000 5000 10000
consumer/w2/b10 1501 1.000 6.70 915 242 5000 5000 10000
consumer/w2/b100 1950 1.000 7.09 1143 243 5000 5000 10000
consumer/w4/b10 1616 1.000 6.78 981 261 5000 5000 10000
consumer/w4/b100 1922 1.000 7.09 1144 244 5000 5000 10000
consumer/w1/b100/tfbs100 7534 0.010 6.06 1117 243 5000 5000 10000
consumer/w1/b100/interval1-10 1349 1.000 7.09 1143 243 5000 5000 10000
producer/w1/b100 4488 0.000 3.04 597 1 0 0 0
producer/w1/b100/ppt1 846 0.000 4.04 637 1 0 0 0

Gated (fails the build): delete_calls + tuple counters (upd/del/ins) + the producer's insert_calls, exact; select_calls and other_calls within +2; wal_records within a 10% band; fetch_calls within 2x. msg/s, WAL bytes and total calls are informational (timing/FPI noise).

@lesnik512
lesnik512 merged commit af6a565 into main Oct 17, 2026
13 checks passed
@lesnik512
lesnik512 deleted the fix/startup-pool-check branch October 17, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant