Skip to content

pg: a timed-out connection is shut down, and freed only once its armed recv has returned - #292

Merged
MDA2AV merged 3 commits into
mainfrom
fix/pg-redis-timeout-dispose
Oct 12, 2026
Merged

MDA2AV merged 3 commits into
mainfrom
fix/pg-redis-timeout-dispose

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Now pg-only. The Redis half went with ioxide.redis, which #337 drops; main is merged in, and the Pg suite passes (7).

A pg command that timed out left its connection's socket open, with a recv still armed against a receive buffer that had already been freed.

On a command timeout the pool disposed the connection while its reader was parked on a ring recv: close(fd), then NativeMemory.Free of the send and receive buffers. close() does not end a socket that an io_uring operation still holds. So the socket stayed ESTABLISHED and the recv stayed armed until the server finally answered, and the kernel then wrote the late reply into the freed receive buffer. In the reproduction, the allocator had already handed that buffer (same address) and the same fd number to the replacement connection, so the reply landed in a live connection's memory. Pick disposing a connection that one loop had marked broken could hit the other loop's operation the same way.

Dispose now marks the connection broken and, while a loop is parked on an operation, calls shutdown(fd, SHUT_RDWR) and returns. That completes the operation, and the loop that exits last runs Dispose again, which closes the fd and frees the buffers. Both entry points and Pick already refuse a broken connection, so nothing touches the buffers in between. It reuses the existing _reading/_sending/IsBroken state. Two comments that said closing the fd cancels the stuck recv are corrected.

Tests

New:

  • pg: a timed-out connection's socket ends at the timeout, not when the late reply comes (pg_sleep(30))

It uses a pool of 2 and a 1000 ms command timeout. The test requires the timed-out connection's socket to leave ESTABLISHED within 10 s. The sibling connection must stay ESTABLISHED and the pool must still answer afterwards. The socket states come from a new harness helper, TcpSockets, which reads /proc/net/tcp and counts only this process's sockets.

main this PR
timed-out connection's socket still ESTABLISHED after 10 s (until the server's late reply) gone at the timeout
late reply completed into the freed receive buffer (_received 65 at ~5.2 s in a pg_sleep(5) probe) never arrives; the recv ended at the shutdown

Suites: Pg 7 with main merged in (2026-10-12); Unit 63 when first run.

Two behaviour notes:

  • A connection that breaks on its own now frees itself once both loops have exited, instead of waiting for the pool to pick and dispose it. The pool's later Dispose is then a no-op.
  • Dispose called after the reactor has stopped, while a loop was still parked, now leaves that connection's fd and buffers to process exit rather than freeing them under the parked operation.

Bench: still to do. This is a pg module change, so it gets a wrk before/after, which I'll post here once the box is quiet; other suites are running on it now. The only hot-path cost is one bool check when a loop exits.

…ts armed recv has returned

A command timeout disposed the connection while its reader was parked on a ring recv: close(fd),
then the buffers freed. close() does not end a socket an io_uring op still holds, so the socket
stayed ESTABLISHED and the recv stayed armed until the server finally answered, and the kernel then
wrote that reply into the freed receive buffer - which the allocator had already handed to the
replacement connection (same address, same fd number). Pick's dispose of a connection broken by one
loop could hit the other loop's op the same way.

Dispose now shuts the socket down while a loop is parked on an op, which completes it, and the loop
that exits last closes the fd and frees the buffers. Dispose marks the connection broken, so it
takes no new commands in between.

"pg:" and "redis: a timed-out connection's socket ends at the timeout, not when the late reply
comes" time a command out on the sidecar (pg_sleep(30), BLPOP 30) and require the pool connection
that timed out to leave ESTABLISHED within 10 s, while its sibling stays ESTABLISHED and the pool
still answers. Confirmed both ways: both fail on the unmodified source (still ESTABLISHED after
10 s) and pass with the fix.
@MDA2AV

MDA2AV commented Oct 7, 2026

Copy link
Copy Markdown
Owner Author

Bench, before/after:

sample main this PR
Clients/Pg 371,955 req/s, 5.41 us/req (5.41-5.47) 364,504 req/s, 5.52 us/req (5.46-5.68)
Clients/Redis 371,641 req/s, 5.13 us/req (5.11-5.13) 373,397 req/s, 5.10 us/req (5.07-5.27)

Within the noise floor: Pg's median is 2% higher, but the first round measured the same (5.46 vs 5.47 us), and Redis, which has the identical change, is flat.

Method: bench/any.sh (wrk), 2 reactors, 64 connections, 8 threads, 10 s per cell, server pinned to P-cores 0-7,10-15, all cells saturated. 3 rounds, each branch run right after main (f298af0) in the same round, so drift lands on both. Median (min-max) of the 3.

@MDA2AV MDA2AV mentioned this pull request Oct 12, 2026
@MDA2AV MDA2AV changed the title pg, redis: a timed-out connection is shut down, and freed only once its armed recv has returned pg: a timed-out connection is shut down, and freed only once its armed recv has returned Oct 12, 2026
@MDA2AV
MDA2AV merged commit 23bf02d into main Oct 12, 2026
1 check passed
@MDA2AV MDA2AV mentioned this pull request Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant