Investigate intermittent TCP stalls with 5,000 active loopback connections on macOS #61

Status

Deferred at maintainer request on 2026-11-10; unresolved. Keep this issue open. No transport repair or production-capacity acceptance has been established, and further TCP load/capture/system-setting experiments are deferred while framework work continues.

See the progress and cross-platform comparison summary. The sections below retain historical experiments and their original limits; they are not an instruction to resume the investigation. Windows/WSL2 passes do not establish bare-metal Linux, cross-host or release acceptance.

Problem

Some connections intermittently stop completing requests in a 5,000-active-connection macOS loopback workload. The native Asio control also reproduces the symptom. This does not establish a cause in the workload, Asio, arknet or the OS TCP stack.

Reproduction context

• macOS 27.0.1 (26A434), arm64, Darwin 27.0.0; AppleClang 21, C++20 Release.
• Boost 1.90.0 / Boost.Asio 1.38.0; TLS off.
• Tool: benchmarks/asio_connections.cpp, source 66289bbfb34cc1c68ea438fd572fbe3cb675acac (diagnostic change in fix(benchmarks)!: preserve TCP kernel sample onset #60).
• One IO context / one IO worker per process; 5,000 loopback TCP connections.
• 1,024-byte application payload, four outstanding requests per connection; 4,096-byte initial/max receive buffer.
• Connection rate 1,000/s, at most 32 concurrent connects; traffic hold duration 3 s; request/connect timeout 5 s, workload deadline 40 s, cleanup allowance 5 s.
• Limits: 20,000 in-flight requests, 64 MiB in-flight bytes, 512 MiB buffer budget. Operation trace, socket-operation trace and TCP kernel samples were all enabled.
• Process descriptor soft limit was 1,048,575. All 5,000 connections were established/accepted in each run; no system-limit tuning was performed.

Completed observations

A fixed three-run campaign retained all outcomes: 5 unfinished connections, all connections completed, then 3 unfinished connections. These observations are not a failure-rate or capacity estimate.

Eight failing peer pairs matched by identity and reversed socket endpoints. Schema 2 retained 21 server queries before the corresponding last client query, addressing the earlier loss of pre-cancellation server history.

• Retained client queries: established connection, loss-recovery flag, congestion window 0, reported retransmission timeout 0, 4,364 bytes in the TCP send buffer, positive peer receive window.
• Corresponding server queries before cancellation: established connection, empty TCP send buffer, positive congestion window.
• Completed client write bytes minus completed server read bytes: 4,364 bytes, equal to four 1,091-byte framed requests.
• Both processes completed cleanup. These counters do not establish where bytes were lost/stalled or why progress stopped.

Earlier schema 1 tail-only evidence and the new schema 2 first-four/latest-four evidence remain distinct. Raw JSON, packet captures and logs remain local and are not release assets or public downloads.

Current investigation

Resumed by maintainer request on 2026-11-05. Root cause remains unresolved; no transport repair or long-run/capacity acceptance is claimed.

• Diagnostic source: fe83d005b5c318da34938e06a1384f9d4f84a3f9 in test(benchmarks): observe stalled sockets outside the IO worker #74, based on accepted d8ff34162cd1b490683f579ef23431068d8b7d8b. Existing local Boost 1.92.0 / Asio 1.38.2 differs from the historical dependency versions; no controlled version comparison was performed.
• Independent, non-consuming socket observations retained empty readable queues and matched both peers before cancellation. A separate bounded three-trial capture established all 5,000 connections in each trial; two trials completed and one retained a single unfinished pair. All workload/capture processes exited without an outer kill.
• For that pair, all four outstanding 1,091-byte requests were positively observed on lo0 before the cancellation marker. Receiver SACK records acknowledged later segments while cumulative acknowledgement remained before the first segment; repeated transmissions of that first segment were also observed. This narrows the next investigation to TCP delivery/recovery and does not locate a loss point or establish an OS defect.
• Capture reports zero kernel drops, but received and saved record counts differ. The conservative completeness gate therefore fails: absent packets or absent ACKs cannot be used as causal evidence. A later complete SACK block is not certified as pre-cancellation. Raw captures, identities, logs and analysis remain local.

MSS diagnostic control

A separate local-only connected-socket experiment requested TCP_MAXSEG 1460 after connection establishment. All 5,000 sockets on both peers read back 1460 before application traffic. Two trials completed; a later trial established all 5,000 but left seven peer pairs unfinished, including five client-side stalled senders whose retained samples still reported MSS 1460, an empty readable queue and congestion window zero. Setting MSS 1460 is therefore not an established remedy, and a fall to 1188 is not required for that observed stalled state. No library default was changed.

The batch is not a balanced statistical comparison: two other planned trials failed before reaching 5,000 connections with a local-address allocation error and are invalid for the established-connection-stall comparison. An earlier fixed-port batch stopped on address-in-use before its MSS arm started; its initial failed workload and setup failure are both retained locally. These setup outcomes must not be counted as successful traffic tests, merged with the original stall or silently replaced.

The experimental option and all raw results remain local; #74 contains only the independently tested observer/clock diagnostics. Root cause and a transport repair remain pending.

Recovery timer and post-failure observation

Two independent local diagnostics used Boost 1.92.0 / Asio 1.38.2 with the original five-second request failure criterion. Neither is a transport repair.

• The public macOS libproc socket query exposes retransmission, persist, keepalive and TIME_WAIT timers. A controlled zero-window connection verified that a reported retransmission timeout of zero can coexist with an active persist timer. The four fields do not expose every internal recovery timer, and their raw values cannot be interpreted as remaining milliseconds without the unavailable timer base.
• A bounded native-Asio campaign stopped at its first traffic failure: the first trial completed, the second established all 5,000 connections and left one pair unfinished. Eight pre-cancellation samples on its client sender retained 4,364 send-buffer bytes, congestion window zero, a positive peer receive window and a nonzero persist timer. This rules out the claim that all recovery timers were absent in that observed state; it does not establish why recovery did not progress.
• A separate first-failure trial established all 5,000 connections and retained 15 unfinished peer pairs. After the original request timeout was latched, an additional ten-second observation admitted no new requests and kept existing reads active. All 60 outstanding requests remained outstanding throughout 11 live sample rounds: admitted/sent/received counts stayed 550,861 / 550,861 / 550,801. Both processes cleaned up, and the client still exited with failure. This shows no later application progress within that bounded interval, not an indefinitely stuck connection or a broken timer callback.
• Local diagnostic source is f732e59414b6afcbae47cb9cdea745db7b2006a1, without the earlier TCP_MAXSEG experiment. Its binary SHA256 is 54f1b613c6f1ae87581edd7aef883ea3b58941717ec228152a3305c999cd4425. Raw observations remain local. No matching public source for the running XNU 13432 kernel was found; older Apple source is explanatory context, not proof of this kernel's implementation.

The owner-approved one-shot SACK control stopped at its negotiation preflight: the helper wrote the host setting from 1 to 0 and verified 0, but a newly created loopback client still reported the public SACK-enabled option (TCP_CONNECTION_INFO returned all 112 bytes, established state, options 7). The guard therefore aborted before launching any 5,000-connection workload; the server option query was not reached after the client mismatch. The helper restored 1, the independent watchdog verified restoration, and a separate read confirmed 1. This is an inconclusive control, not a SACK-disabled workload result or transport fix. No additional setting attempts or retries were made. Root cause, targeted repair and repair validation remain pending.

All-options-disabled traffic isolation

A public per-socket TCP_NOOPT export control, with host SACK restored to1, returned established options0 on both peers. Its Boolean readback was raw8, not normalized1; the initial assertion error stopped before connect and was retained separately. This control removes timestamps and window scaling as well as SACK, so it is not a pure SACK experiment.

Exactly one native-Asio traffic trial then used the same5,000-connection,1,024-byte/window4, one-context/worker and original5s request criterion. Both peers passed5,000 complete established-options0 checks, with zero option failures. Local diagnostic source was bd95573e5c715cff31a8c9b051813d3d201fe980; binary SHA256 d98cf56bcad2ea49cbd5bf6668d89a12b7661966e19044208db36f0dea58fdff. No host setting was changed.

• One request exceeded5s. Its response completed after5.468465s on the client's own clock,430.798ms after failure was latched. During the bounded observation, admission stayed441,491, responses advanced441,490 to441,491 and outstanding requests1 to0.
• The affected server sender retained positive congestion window7,168 bytes, an active retransmission timer, persist0 and MSS512; send-buffer bytes progressed780 to268. This differs from the original zero-congestion-window / persist / no-ten-second-progress state.
• Both processes retained exit1 and cleaned up without an outer kill. Final4,999 finished reflects frozen post-failure admission, not proof of a permanently missing response. A late response does not convert the original timeout into success.

This trial reproduces a five-second timeout without TCP options, but does not reproduce or exclude the original persistent stall. No targeted transport repair or acceptance is established; no retries followed.

A separate offline comparison of the retained original first-hole repetitions found increasing TSval values above previously observed peer echoes. It does not support stale/retrograde timestamps on those observed repetitions, does not establish receiver ts_recent, and does not exclude PAWS. The incomplete-capture limitation remains unchanged.

Remaining work

• Review retained evidence and select independent socket/sequence observations as a bounded discriminator.
• Match both peers and bracket socket observations with clock/cancellation markers; retain the incomplete-capture limitation.
• Establish the failed sequence range's delivery/recovery cause and validate a targeted repair or justified intervention; do not infer this from aggregate TCP counters.
• Preserve fixed run counts, source/binary identities and failures; do not retry to green or increase timeouts to hide the problem.
• Obtain cross-host/Linux acceptance when an appropriate test environment is available.

Related diagnostic changes: #58, #59, #60. No fix or 5,000-active-connection production-capacity claim is made.