Skip to content

test(benchmarks): add non-Asio controls for connection stalls - #76

Open
nick2781 wants to merge 1 commit into
mainfrom
test/connection-stall-controls
Open

nick2781 wants to merge 1 commit into
mainfrom
test/connection-stall-controls

Conversation

@nick2781

@nick2781 nick2781 commented Nov 9, 2026

Copy link
Copy Markdown
Member

What changed

Diagnostic tools for the 5,000-connection loopback stall in #61. Default behavior of every existing tool is unchanged.

  • arknet_poll_connection_control (new, POSIX only, benchmarks/poll_connections.cpp): a single-thread TCP peer without Asio or strands that speaks the native Asio control's wire protocol, so either role can be paired with arknet_asio_connection_control.
    • --events poll|kqueue: poll(2), or on Apple/BSD a loop that registers events the way the Boost.Asio kqueue reactor does.
    • --write-pacing burst|serial, --read-pacing drain|serial: reproduce the native control's serialized async_write and one-frame-per-completion async_read_until cadence.
    • --socket-setup default|asio: SO_NOSIGPIPE and ioctl(FIONBIO) as in Boost.Asio socket_ops.
    • --send-queue-watch on: reports kernel send queues unchanged for one second, with Apple TCP_CONNECTION_INFO.
    • Stalled-connection snapshots, per-call read/write histograms and a result event recording every mode.
  • arknet_asio_connection_control: --session-strands on|off, --write-buffers split|joined and --io-histogram off|on. The default write path keeps its two-buffer std::array; --io-histogram cannot be combined with --socket-operation-trace.
  • connections_io_histogram.hpp (new) and an io_count_stream wrapper in connections_socket_trace.hpp that preserves handler associations like the existing trace wrapper.
  • CMake registers benchmark_poll_connections_help and benchmark_poll_connections_self_test on Unix. benchmarks/CONNECTIONS.md, CONNECTIONS_CN.md and the testing tables in docs/project/testing*.md describe the tools.

Why

#61 had no way to separate Asio-specific behavior from the transport. With these controls, same-host comparisons showed that a non-Asio peer can reproduce the stall, that it appears only at 5,000 connections, and that it coincides with lost loopback packets and retransmission. The measurements and their limits are in the 2026-11-09 comment. This PR does not fix #61 and makes no capacity claim.

Validation

Local, macOS 27.0.1 (26A434) arm64, Apple clang 17.0.0, Boost 1.92.0, Release, TLS off, at the head of this PR:

  • cmake -G Ninja -DARKNET_BUILD_BENCHMARKS=ON -DARKNET_BUILD_TESTS=ON with ARKNET_CHECK_TESTS set to the 14 connection checks, then cmake --build ... --target arknet_check: all passed — connection_kernel_samples, benchmark_connections_help, connection_load_protocol, connection_load_resources, benchmark_connections_backpressure, benchmark_asio_connections_help, benchmark_asio_connections_self_test (now also covering the three new switches and the double-wrap rejection), connection_operation_trace, connection_socket_operation_trace, benchmark_poll_connections_help, benchmark_poll_connections_self_test (four event/pacing combinations), benchmark_asio_socket_observer, benchmark_asio_connections_processes, connection_socket_observer.
  • poll_connections.cpp compiles without warnings under -Wall -Wextra -Wpedantic -Wshadow -Wconversion.
  • node --test tests/ci/*.test.mjs: 93 passed, 1 skipped, 0 failed; node --test docs/site/scripts/ci-scope.test.mjs: 6 passed.
  • Manual pairings: poll ↔ native Asio in both directions, and kqueue, serial-pacing and Asio-socket-setup modes at 300 connections, all succeeded.
  • Not run locally: the Linux/Windows/sanitizer matrix, clang-format 18, and the documentation site check (docs/site dependencies are not installed here).

Limits

  • Diagnostic only; no library code changes.
  • Local formatting used clang-format 22.1.8 with scripts/format_spacing.sh; clang-format 18 is verified by CI.
  • --events kqueue, --socket-setup asio's SO_NOSIGPIPE and tcp_info apply on Apple/BSD; elsewhere the self-test runs the poll(2) combinations and tcp_info reports unsupported.
  • The 5,000-connection campaigns in Investigate intermittent TCP stalls with 5,000 active loopback connections on macOS #61 were run with intermediate local revisions of these tools; raw results remain local.

🤖 Generated with Claude Code

Add arknet_poll_connection_control, a POSIX peer without Asio or strands
that speaks the native control's wire protocol. It can use poll(2) or an
Asio-like kqueue loop, reproduce the native control's serialized read and
write cadence and socket setup, and report stalled kernel send queues with
Apple TCP_CONNECTION_INFO.

Add --session-strands, --write-buffers and --io-histogram to the native
Asio control; defaults keep its existing behavior. Document both tools.

These controls separate Asio-specific behavior from the transport in #61.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate intermittent TCP stalls with 5,000 active loopback connections on macOS

1 participant