[danceratopz]

Draft: refreshed CI and final confirmation pending

The root transport fix is shipped and is now the project baseline: hive-python-api#18 merged, ethereum-hive v0.1.0 is published on PyPI, and execution-specs#3472 updated forks/amsterdam to use ethereum-hive>=0.1.0,<1.0.0.

Description

This PR adds complementary simulator-side detection for the silent Hive result loss observed during the TIME_WAIT ephemeral-port exhaustion incident behind sporadic Nethermind consume-enginex dashboard failures on 2026-07-18/19. The package fix pools Hive API connections and safely retries connection establishment; this PR makes any remaining reporting loss visible and fails the pytest run.

• Exact session-wide result accounting: each worker tracks unique pytest node IDs that ran, successfully reported a Hive result, or were skipped before their Hive test started. The last worker aggregates the counts after the session-scoped EngineX Hive suite finishes. A healthy run must account for every collected test exactly; both undercounts and overcounts fail.
• Distributed interruption handling: each worker persists its interruption state and executed-test count. The aggregate assertion is skipped if any worker was interrupted or fewer tests ran than were collected, avoiding false reported-test-count-check failures after -x, --maxfail, or another partial run.
• Visible failure on genuine result loss: a count mismatch reports a failing reported-test-count-check meta-test to Hive and raises HiveReportedTestCountError, so missing results cannot silently shrink the dashboard total.
• Single-shot non-idempotent calls: start_test, end_test, and register_multi_test_client are not retried at the application layer. This avoids duplicating a Hive request when a response-side disconnect is surfaced as requests.ConnectionError after the server may already have processed the POST. Safe connection-establishment retries remain the responsibility of ethereum-hive v0.1.0.

The accounting helpers live in plugins/pytest_hive/reporting.py. The pytest_hive plugin and test directories are regular packages to avoid pytest prepend-mode import collisions.

Validation

• Original A/B validation with Hive --dev and Nethermind: 12,888 Cancun tests at -n 4, identical verdicts, 12,888/12,888 reported, and peak TIME_WAIT toward the Hive API reduced from 9,684 to 66 with the pooled package.
• Pre-release end-to-end validation of v0.1.0 with Hive --dev, consume enginex, tests@v20.0.2, geth, and Nethermind: 7,848/7,848 passed.
• Focused reporting and EngineX tests: 25 passed, including a real pytest -n 2 -x regression covering cross-worker early-stop behavior.
• Full testing package suite: 2,182 passed, 29 skipped, 4 xfailed, and 2 xpassed.
• Full static analysis: just static passed, including Ruff, mypy, codespell, vulture, actionlint, and ethereum-spec-lint.
• execution-specs CI is rerunning on the force-pushed squashed commit.

Dependency status

The package-side work and dependency update are complete:

• Root fix: hive-python-api#18 — pooled process-wide Hive API session with connect-only retries.
• Release preparation: hive-python-api#19.
• Release: ethereum-hive v0.1.0 tag and PyPI package.
• execution-specs dependency update: execution-specs#3472, merged into forks/amsterdam.

Related issues and PRs

The original incident was simulator-side Hive API ephemeral-port exhaustion, not a Nethermind client defect. The root transport fix is hive-python-api#18; this PR provides complementary reporting hardening and silent-loss detection.

Checklist

• Ran fast static checks to avoid CI failures; see Code Standards and Verifying Changes: just static
• PR title has the form <type>(<area>): <title>, where <type> and <area> come from an appropriate C-<type>, respectively A-<area>, label. The title should match the target squash commit message.

Cute animal picture

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.90%. Comparing base (e5b6e3b) to head (d1c8015).

Additional details and impacted files

@@               Coverage Diff                @@
##           forks/amsterdam    #3335   +/-   ##
================================================
  Coverage            93.90%   93.90%           
================================================
  Files                  624      624           
  Lines                36905    36905           
  Branches              3326     3326           
================================================
  Hits                 34655    34655           
  Misses                1569     1569           
  Partials               681      681           

Flag	Coverage Δ
unittests	93.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[danceratopz]

Closing in favour of #3879. The root cause (one new TCP connection per hive API call, which exhausted ephemeral ports) is fixed in hive-python-api#18 (ethereum/hive-python-api#18), released in ethereum-hive v0.1.0 (https://pypi.org/project/ethereum-hive/0.1.0/), which this repo already requires. Since then, every consume-enginex run on hive.ethpandaops.io (178 runs from 2026-09-21 to 2026-10-21, all clients) reported the full test count of its fixture release, and the latest geth, reth and Nethermind runs have no "Test was terminated by host" results. The extra result-count check here is no longer worth its size. #3879 keeps the one remaining fix from this PR: end each hive test only once.