feat(benchmarks): report connection port resources

benchmarks/CONNECTIONS.md

@@ -98,13 +98,13 @@ The buffer preflight reservation is `N * max-buffer-bytes + max-inflight-bytes +
 
 Large local client runs also consume ephemeral source ports. Closed TCP connections can remain in `TIME_WAIT`, so consecutive runs can exhaust the available ports and fail with `EADDRNOTAVAIL` even below the FD and memory limits. Record the OS port range and TCP states; allow previous connections to retire before an independent run, or distribute clients across suitable hosts/addresses. Preserve the failed run rather than counting it as a completed connection target. A local client's port limit is not the server's connection capacity.
 
-Before every batch, check free source-port headroom against the connection target, including ports held in `TIME_WAIT`. Wait for natural release or fail preflight when headroom is insufficient; a previous process's clean exit does not imply that its source ports are reusable.
+Before every batch, record the source-port range and available TCP-state observations. The configured range width is not free capacity: reserved ports, local addresses, destination tuples, other applications and reuse rules also matter. A namespace/host-wide `TIME_WAIT` count is not this process's count and must not be subtracted from the range to invent free headroom. If usable headroom cannot be established, record it as unavailable and wait for an independent run or fail preflight. A previous process's clean exit does not imply that its source ports are reusable. A ramp that fails with `EADDRNOTAVAIL` before reaching its target is separate from a TCP stall after all target connections have been established.
 
 There is one main scheduling thread and one cleanup-watchdog thread in addition to the configured IO workers. It does not create a thread for each connection. Ctrl+C or SIGTERM requests a failed run with normal cleanup. If the run, final output or cleanup cannot finish by the overall deadline plus cleanup grace, the watchdog requests immediate OS process termination: `TerminateProcess` with code 124 on Windows, or `SIGKILL` on POSIX. POSIX shells commonly display 137 for signal 9, but it is a signal termination, not a portable exit-code guarantee. This path means `clean_shutdown=false`; no complete success summary is produced. Termination does not depend on another log record, a C-runtime flush or sanitizer exit handlers.
 
 ## Output and interpretation
 
-Stdout is JSON Lines: a `resource_startup` event for each process, a server `ready` event, periodic `sample` events and one final `result` event. The native Asio control also emits `resource_startup`. Startup identifiers retain each tool's existing convention: decimal strings in the arknet tool and integers in the native control. This uses the existing initial resource sample, before connection admission; it does not add a sampling pass or change the cadence. Consumers should select events by `event` and ignore additional fields. Stderr contains argument/startup errors. Watchdog termination may leave output incomplete, so also record each process's exit code. Save each process's streams separately outside the Git repository.
+Stdout is JSON Lines: a `resource_startup` event for each process, a server `ready` event, periodic `sample` events, an optional `connect_resource_failure` before client cleanup, and one final `result` event. The native Asio control also emits `resource_startup`. Startup identifiers retain each tool's existing convention: decimal strings in the arknet tool and integers in the native control. This reuses the initial process-resource sample before connection admission; separate read-only port queries run at startup and the first connect failure without changing the periodic cadence. Consumers should select events by `event` and ignore additional fields. Stderr contains argument/startup errors. Watchdog termination may leave output incomplete, so also record each process's exit code. Save each process's streams separately outside the Git repository.
 
 | Result fields | Interpretation |
 | --- | --- |
@@ -122,6 +122,9 @@ Stdout is JSON Lines: a `resource_startup` event for each process, a server `rea
 | `resource_sample_elapsed_s`, `resource_cpu_interval_s` | CPU-query midpoint relative to the start of this run, and the actual interval used for its CPU percentage. The interval is null when CPU percentage is unavailable. Neither is a cross-host timestamp. |
 | `resource_phase` | `running` for startup and periodic observations during ramp/hold/protocol drain; `after_cleanup` for the final resource reading, after endpoints and IO workers have stopped. Final current memory/counts are not load-time values. |
 | `resource_unavailable` | Object mapping each missing resource field to its diagnostic reason. A real zero stays numeric; reasons are diagnostic text, not stable error codes. |
+| `network_resources` (startup) | `ephemeral_ports` contains inclusive `first`/`last` values, `range_source` and `range_scope`. macOS reads `net.inet.ip.portrange.first`/`last` through `sysctlbyname` (host scope); Linux reads `/proc/sys/net/ipv4/ip_local_port_range` (network-namespace scope). Unsupported or failed queries return null with an `unavailable` reason including numeric OS errors when a system query fails. No external command or setting change is performed. |
+| `tcp_time_wait`, `time_wait_source`, `time_wait_scope` inside network resources | Linux reads only `/proc/net/sockstat`'s `TCP: tw` counter, scoped to all processes in the current network namespace. Other platforms explicitly report unavailable. These counters are neither per-process nor a count of occupied ephemeral ports for the selected destination; queries are not atomic with connection counters. |
+| `connect_failure_resources` (`connect_resource_failure` event and result) | The main thread emits the failure event before client cleanup and repeats its captured snapshot in the final result. First connect failure captured before cleanup, with its numeric `error_code`, `error_category`, observed `established` count and `network_resources`. The `phase` remains `first_connect_failure_before_cleanup` even though the result is printed after cleanup. Later failures do not replace it. Null means no connect failure was observed; it does not explain a later traffic stall. Diagnostic capture failure is explicit. |
 | `sample_interval_ms` | Requested sampling cadence, included in the final result. Actual CPU intervals are reported separately. |
 | `content_errors`, `sequence_errors`, `send_errors`, `timeouts`, `resource_rejections` | Validation, transport, deadline or configured-budget failures. `shutdown_cancelled_sends` is separate from normal send errors. |
 | `clean_shutdown`, `outcome`, `first_error` | Cleanup status and `target_complete`, `observation_complete` or `failed`. Clean shutdown alone does not imply delivery success or a target was reached. |
@@ -153,7 +156,7 @@ The JSON changes are additive; existing fields retain their types and measuremen
 
 Configure with `-DARKNET_BUILD_TESTS=ON` to also build `arknet_connection_protocol_test`. It checks identity preservation, corrupted/truncated frame rejection, overflow and mergeable RTT bucket bounds. `arknet_check` includes those checks and `--help`; the POSIX full-output-pipe process check verifies watchdog termination without relying on output consumption. Deployment load runs remain local/manual.
 
-`arknet_connection_resources_test` checks the actual resource output fields, numeric zero versus missing metrics and reasons, diagnostic escaping, and independent periodic/whole-run time windows. The standalone [process resource probe](process_resources.md) covers the platform counters themselves.
+`arknet_connection_resources_test` checks the actual resource output fields, numeric zero versus missing metrics and reasons, diagnostic escaping, independent periodic/whole-run time windows, validated port-range/TIME_WAIT parsing, native query errors and retention of the first pre-cleanup connect-failure snapshot. It uses injected failure counters, not port exhaustion. The standalone [process resource probe](process_resources.md) covers the process counters themselves.
 
 Before increasing scale, run both idle and traffic with a few connections, IPv4 and IPv6, and inspect both final results. Also verify a refused connection, mismatched run ID, target deadline, duplicate identity and an invalid buffer/window budget fail with a nonzero exit. Keep the command lines, executable/source identity, machine configuration, all failures and both process logs with each measurement. This tool's test protocol does not replace an ark application, authentication, reconnect recovery or cross-machine production acceptance.
 

benchmarks/CONNECTIONS_CN.md

@@ -98,13 +98,13 @@ Windows 下由 CMake 提供系统网络与进程内存库。构建或使用 arkn
 
 本地大量建连还会占用客户端临时端口。TCP 连接关闭后可能仍处于 `TIME_WAIT`；连续运行会耗尽可用端口，即使 FD 和内存尚未达到限额，也可能报 `EADDRNOTAVAIL`。记录系统端口范围和 TCP 状态，等上一轮连接释放后再独立运行，或将客户端分布到合适的主机与地址。失败记录应保留，不能算作完成了目标连接数；本地客户端的端口限制也不代表服务端容量。
 
-每轮开始前，都应将空闲源端口余量与连接目标比较，并计入 `TIME_WAIT` 占用。余量不足时等待自然释放，或让预检失败；上一轮进程清理成功，不代表源端口已经可以复用。
+每轮开始前，都应记录源端口范围和可用的 TCP 状态观察。配置范围的宽度不等于空闲容量：保留端口、本地地址、目标四元组、其他应用及复用规则都会影响可用性。命名空间／主机范围的 `TIME_WAIT` 数量不是当前进程的数量，不能从端口范围中直接扣减来推算空闲余量。无法确认可用余量时应记录为 unavailable，等待独立运行时机或让预检失败。上一轮进程清理成功，不代表源端口已经可以复用。尚未达到目标连接数就出现 `EADDRNOTAVAIL` 的建连失败，与全部目标连接建立后的 TCP 停滞必须分别记录。
 
 除指定的 IO 工作线程外，每进程还有一条主调度线程和一条清理监控线程，不会为每条连接创建线程。Ctrl+C 或 SIGTERM 会请求失败退出，并执行正常清理。如果总期限加清理宽限期后，运行、最终输出或清理仍未完成，监控线程请求操作系统立即终止进程：Windows 使用 `TerminateProcess`，退出码为 124；POSIX 使用 `SIGKILL`。POSIX shell 通常将信号 9 显示为 137，但这不是跨平台统一退出码的保证。该路径表示 `clean_shutdown=false`，不产生完整成功摘要；终止过程不依赖再写日志、C 运行时的 flush 或 sanitizer 退出处理。
 
 ## 输出与指标
 
-Stdout 使用 JSON Lines：每个进程的 `resource_startup`、服务端的 `ready`、定期 `sample` 和最终 `result`。原生 Asio 对照程序也输出 `resource_startup`。启动标识沿用各工具已有的格式：arknet 程序使用十进制字符串，原生对照程序使用整数。启动记录复用建连准入前已有的首次资源采样，不新增采样，也不改变采样周期。消费方应按 `event` 选择事件，并忽略新增字段。Stderr 用于参数或启动错误。清理超时中止可能留下不完整输出，因此还需记录进程退出码。分别保存各进程输出，原始结果放在 Git 仓库外。
+Stdout 使用 JSON Lines：每个进程的 `resource_startup`、服务端的 `ready`、定期 `sample`、客户端清理前可选的 `connect_resource_failure` 和最终 `result`。原生 Asio 对照程序也输出 `resource_startup`。启动标识沿用各工具已有的格式：arknet 程序使用十进制字符串，原生对照程序使用整数。启动记录复用建连准入前已有的首次进程资源采样；另在启动和首次建连失败时执行只读端口查询，不改变周期采样频率。消费方应按 `event` 选择事件，并忽略新增字段。Stderr 用于参数或启动错误。清理超时中止可能留下不完整输出，因此还需记录进程退出码。分别保存各进程输出，原始结果放在 Git 仓库外。
 
 | 字段 | 口径 |
 | --- | --- |
@@ -122,6 +122,9 @@ Stdout 使用 JSON Lines：每个进程的 `resource_startup`、服务端的 `re
 | `resource_sample_elapsed_s`、`resource_cpu_interval_s` | CPU 查询中点相对本次运行开始的秒数，以及 CPU 占用率实际使用的区间长度。CPU 占用率不可用时，区间为 null；这些值不能作为跨机器时间戳。 |
 | `resource_phase` | 启动和周期记录为 `running`，周期记录可能处于建连、保持或协议排空阶段；最终记录为 `after_cleanup`，在端点和 IO 工作线程停止后读取。最终内存和数量指标不是负载阶段的值。 |
 | `resource_unavailable` | 以缺失资源字段名为键、诊断原因为值的对象。真实零值仍是数字；原因文本用于诊断，不是稳定错误码。 |
+| 启动记录的 `network_resources` | `ephemeral_ports` 含闭区间的 `first`／`last`，并附 `range_source` 和 `range_scope`。macOS 通过 `sysctlbyname` 读取 `net.inet.ip.portrange.first`／`last`，范围为整台主机；Linux 读取 `/proc/sys/net/ipv4/ip_local_port_range`，范围为当前网络命名空间。不支持或查询失败时为 null，`unavailable` 记录原因，系统查询失败时包含数字 OS 错误。不调用外部命令，不修改设置。 |
+| 网络资源内的 `tcp_time_wait`、`time_wait_source`、`time_wait_scope` | Linux 仅读取 `/proc/net/sockstat` 的 `TCP: tw`，涵盖当前网络命名空间的所有进程；其他平台明确报告 unavailable。这既不是本进程计数，也不是特定目标已占用的临时端口数；系统查询与连接计数不是原子快照。 |
+| `connect_resource_failure` 事件和最终记录的 `connect_failure_resources` | 主线程在客户端清理前输出失败事件，并在最终记录中重复保留的快照。保留清理前首次建连失败的数字 `error_code`、`error_category`、当时观察到的 `established` 和 `network_resources`。虽然最终结果在清理后输出，`phase` 仍为 `first_connect_failure_before_cleanup`。后续错误不覆盖它。Null 表示未观察到建连失败，不解释后续流量停滞；诊断采样自身失败也会明确记录。 |
 | `sample_interval_ms` | 最终结果中的计划采样间隔。实际 CPU 采样区间单独报告。 |
 | `content_errors`、`sequence_errors`、`send_errors`、`timeouts`、`resource_rejections` | 内容、序号、发送、期限或预算失败；停止过程中取消的发送单独记为 `shutdown_cancelled_sends`。 |
 | `clean_shutdown`、`outcome`、`first_error` | 清理状态，以及 `target_complete`、`observation_complete` 或 `failed`。清理完成本身不代表交付成功或达到连接目标。 |
@@ -151,7 +154,7 @@ JSON 只新增字段，原有字段的类型和测量阶段保持不变。对比
 
 配置 `-DARKNET_BUILD_TESTS=ON` 后可以构建 `arknet_connection_protocol_test`，检查身份保真、损坏与截断消息拒绝、长度溢出及可合并延迟桶边界。`arknet_check` 包含这些检查及 `--help`；POSIX 的满输出管道进程用例会检查监控线程在无人读取输出时仍可终止。部署负载仍在本地手动运行。
 
-`arknet_connection_resources_test` 检查实际资源输出字段、真实零值与缺失指标及其原因、诊断文本转义，以及周期和全程时间范围的独立性。系统计数器本身由独立的[进程资源检查程序](process_resources_CN.md)验证。
+`arknet_connection_resources_test` 检查实际资源输出字段、真实零值与缺失指标及其原因、诊断文本转义、周期和全程时间范围的独立性，以及端口范围／TIME_WAIT 解析、原生查询错误和首次清理前建连失败快照的保留。使用注入错误计数，不耗尽端口。进程计数器本身由独立的[进程资源检查程序](process_resources_CN.md)验证。
 
 扩大规模之前，先用少量连接检查 idle、traffic、IPv4 和 IPv6 的双端最终计数，再验证拒绝连接、错误运行标识、目标期限、重复身份及非法缓冲或窗口预算均非零退出。每次测量保留命令、可执行文件与源码身份、机器配置、全部失败和双端日志。本工具的测试协议不能替代 ark 应用、鉴权、重连恢复或跨机器生产验收。
 

benchmarks/asio_connections.cpp

@@ -591,6 +591,7 @@ struct application
                            stopped_at{}, drained_at{};
     clock_type::time_point sampled = began;
     connection_load::resource_observer resources{began};
+    connection_load::connect_failure_observer connect_resources;
 };
 
 session::session(application& owner, asio::io_context& context)
@@ -1239,6 +1240,7 @@ void application::launch() {
                 if (!stopping && error)
                 {
                     ++connect_errors;
+                    connect_resources.capture(error.value(), error.category().name(), established.load());
                     fail("connect callback", error.message());
                 }
                 else if (!stopping)
@@ -1466,6 +1468,7 @@ void application::run() {
     {
         if (failed)
         {
+            connect_resources.write_event(std::cout, o.role, o.run, o.client);
             stop();
         }
         else if (clock_type::now() >= until_deadline)
@@ -1543,6 +1546,7 @@ void application::result() const {
             std::cout << ",\"socket_observer_enabled\":true";
         }
     }
+    connect_resources.write_fields(std::cout);
     std::cout << "}\n";
     if (!failure.empty() || !clean || !complete)
     {

benchmarks/connections.cpp

@@ -311,6 +311,7 @@ struct counters
         outstanding{}, shutdown_cancelled_sends{};
     std::atomic<std::int64_t> work_begin{}, work_end{};
     connection_load::latency_distribution latency;
+    connection_load::connect_failure_observer connect_resources;
     std::mutex error_mutex;
     std::string first_error;
 
@@ -531,6 +532,8 @@ class client_run
                     }
                     if (!values_.closing)
                     {
+                        values_.connect_resources.capture(
+                            error.value(), error.category().name(), values_.established.load());
                         values_.fail("connect: " + error.message());
                     }
                     return;
@@ -572,6 +575,7 @@ class client_run
         {
             --values_.connecting;
             ++values_.connect_failures;
+            values_.connect_resources.capture(0, "connect_submission_rejected", values_.established.load());
             values_.fail("connect submission rejected");
         }
     }
@@ -987,6 +991,7 @@ void result(const options& settings, const counters& values, std::string_view ou
         << ",\"buffer_budget_bytes\":" << settings.buffer_budget << ",\"sample_interval_ms\":" << settings.sample_ms
         << ",\"peak_pending_send_bytes\":" << values.peak_pending_bytes;
     connection_load::write_resource_fields(std::cout, resources, true);
+    values.connect_resources.write_fields(std::cout);
     std::cout << ",\"first_error\":" << quoted(values.first_error)
               << ",\"latency_observations\":" << values.latency.total();
     for (const auto percent : {50, 95, 99})
@@ -1143,6 +1148,7 @@ int run(const options& settings) {
             }
         }
         active_before_stop = values.active;
+        values.connect_resources.write_event(std::cout, settings.role, settings.run, settings.client, true);
         client.stop();
     }
     runtime.stop();