Skip to content

fix(netwatch): recover failed UDP rebinds without terminating I/O - #235

Draft
ttizze wants to merge 1 commit into
n0-computer:mainfrom
ttizze:fix/udp-rebind-recovery
Draft

ttizze wants to merge 1 commit into
n0-computer:mainfrom
ttizze:fix/udp-rebind-recovery

Conversation

@ttizze

@ttizze ttizze commented Sep 25, 2026 •

Copy link
Copy Markdown

Description

A failed explicit UDP rebind leaves the socket closed without setting is_broken or waking idle I/O. Subsequent I/O cannot recover. Simply setting the flag still exposes rebind errors to async receive callers, which can permanently stop a QUIC endpoint driver before the address becomes usable again.

Keep recovery in netwatch: mark failed rebinds broken, wake pending I/O, and retry with backoff from 100 ms to a 5 s cap. Async I/O remains pending; nonblocking sends return WouldBlock during recovery. A socket-owned timer wakes both send and receive waiters, with no spawned retry task. Explicit close() cancels recovery and leaves the socket closed. An explicit rebind bypasses the delay so a new network change can trigger an immediate attempt.

All poll-based I/O uses the same recovery path. UdpSender::poll_send acquires its socket read guard after polling writable readiness, so a rebind never attempts a write lock while that guard is held.

Related: #200, #201, and n0-computer/iroh#4558. This follows the request on the iroh PR to put recovery and regression tests in the UDP wrapper. It addresses both the retry-state defect and the concern about unbounded rebind frequency.

API Changes

No new public API. During failed-rebind recovery, async I/O waits and nonblocking sends return WouldBlock. Explicit rebind() still returns the original bind error. close() cancels automatic recovery.

Validation

  • cargo test --locked -p netwatch --lib: 21 passed on macOS.
  • The three new regression tests fail with the production fix removed.
  • Tests cover waking an idle receiver, waking both directions, capped retry frequency even with repeated sends/receive polls, data delivery after recovery, and explicit close cancelling a pending retry.
  • A backport to netwatch 0.19.3 passes 18 tests, including recovery using a real occupied UDP port.
  • With the previous iroh-side production fix removed, a relay-free QUIC probe recovers an existing connection and establishes a new connection after clearing a 500 ms rebind failure, without another network-change notification.
  • cargo clippy -p netwatch --all-targets completes with the existing unused_async warning in interfaces/bsd.rs:40; -D warnings fails on that unchanged warning.

Reproducible iroh integration tests (iroh 1.1.0, no iroh-side production changes): validation branch. Run cargo test --locked -p iroh --lib failed_rebind; all three pass with the netwatch backport pinned in that branch.

Notes & open questions

Retries continue at the capped interval until recovery or explicit close. Prepared with Codex at the author's request; human review remains pending. This PR does not establish the trigger of any historical production outage.

Change checklist

  • Self-review.
  • Documentation updates where relevant.
  • Tests.
  • All API changes documented.
  • This PR was created by a human that thought critically about the proposed change and wrote an as clear and concise description as they could.
  • This PR isn't slop, and is carefully crafted to have the intended effect.
  • cargo make passes locally (targeted checks above).

@flub

flub commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

isn't this a dup of #201 but with way more code changes?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🚑 Needs Triage

Development

Successfully merging this pull request may close these issues.

2 participants