flux-network: half-close the write side and cap accepted connections - #142
Closed
Bronek wants to merge 4 commits into
Closed
flux-network: half-close the write side and cap accepted connections#142Bronek wants to merge 4 commits into
Bronek wants to merge 4 commits into
Conversation
Bronek
marked this pull request as draft
August 28, 2026 14:28
shutdown_write_when_drained ends a connection's outbound stream without giving up what the peer still has to say: the write side shuts as soon as the queued bytes are written, and the connection stays registered and readable. The peer reads the end of the stream, the bytes it sends afterwards still arrive as Message, and its own close still arrives as Disconnected. Both transports half-close, which is what ADR 0002 asks of them. Sends to such a token are rejected and queue nothing, so the queue only shrinks from the request onward. A hard close still ends the connection at any point, and a token that is draining as well closes when its queue empties rather than half-closing. Assisted-by: Claude Code:claude-opus-5
A group can bound how many connections it holds: max_connections is enforced in the accept loop, which accepts the connection arriving at a full group and drops it where it stands — no registration, no bytes, no event — and drains the rest of the backlog, as an edge-triggered listener requires. refused_connections reports how often that happened, and a warning names the group at the ten-second cadence the backlog warning already uses, never once per refusal. The cap counts what the group accepted and still holds, a connection it is closing included: draining and half-closed connections hold their places until they are gone. Outbound endpoints and listeners are not accepted connections and count for nothing. Nothing above the transport takes part: an HttpService on a capped group never hears of the connection its group refused. Assisted-by: Claude Code:claude-opus-5
Three rules the half-close tests left free to be negated: that the hard close wins over the half-close whichever was asked for first, that a broadcast passes over a half-closed member rather than writing to it, and that a reconnecting outbound endpoint gets its write side back. Each new test fails if its rule is reversed — the drain gate's branches swapped, the per-member broadcast filter dropped, the reset in disconnect_index deleted — and each runs over both transports, none of the three being a matter of which one carries the bytes. The recorder the file already had now keeps every lifecycle event with its token, which is what lets a test say which connection was closed and which one a payload arrived on. The write side of a shut connection is one the peer can read the end of the stream from, whether or not it has; the rustdoc of WriteSide::Shut said it already had. Assisted-by: Claude Code:claude-opus-5
Admission compares a count the group keeps, incremented when an accepted connection is inserted and decremented at each of the three places one is removed, so refusing a connection at the cap costs the same however many connections the network holds. A refusal flood is the case the cap exists for, and a scan of every connection per arrival made that cost grow with the network, unrelated groups and outbound endpoints included. The count is a private invariant of the group that owns the policy. The tests pin every path that moves it — peer disconnect, local disconnect, removal and group close — and that two listeners of one group share one cap. Assisted-by: Claude Code:claude-opus-5
Bronek
force-pushed
the
bronek/api_server-a5
branch
from
September 3, 2026 13:53
dcf97ae to
086d549
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR builds on #141. It adds two transport-level controls to
StreamNetwork: write-side half-close and a per-group cap on accepted connections. Four commits.Write-side half-close
shutdown_write_when_drained(token)writes everything already queued, shuts the connection's write side, and keeps reading. The peer sees the end of the outbound stream, while later inbound bytes and the peer's eventual close still arrive as normal events. TCP and Unix sockets share this behavior.Sends are rejected from the half-close request onward, before their payload is serialized, so the queue can only shrink. Broadcasts skip half-closing and half-closed connections. A hard close still wins in either call order, and an outbound endpoint starts a reconnect with its write side open again.
A TCP peer that never drains is bounded by
TCP_USER_TIMEOUT; a Unix peer has no corresponding bound.Connection admission
ConnectionGroupConfig::max_connectionslimits the accepted connections a group holds.Noneleaves the group uncapped;Some(0)is rejected. A connection arriving at the cap is accepted and immediately closed without registration, bytes or an event, while the accept loop continues to drain the listener backlog.Draining and half-closed accepted connections continue to hold their places. Listeners and outbound endpoints do not count, and every listener in one group shares the same allowance.
Each group maintains its accepted count, making admission and refusal O(1) regardless of the network's size. Peer close, local disconnect, removal and group close release the corresponding places.
refused_connections(group)reports the cumulative refusals. Warnings identify the group and are limited to one every ten seconds. Refusal remains entirely below the service layer, so anHttpServicenever receives an event for that connection.Verification
Half-close tests cover immediate and queued shutdown, continued reads, send and broadcast filtering, hard-close precedence, and reconnects over TCP and Unix sockets. Admission tests cover the cap boundary, draining and half-closed connections, backlog draining, every release path, shared caps across listeners, outbound endpoints, group isolation and HTTP behavior.
The loopback tests still guess unused TCP ports. That race is fixed later in the stack when
listenreports the endpoint it bound; rerun the job if it appears in CI.Workspace tests, Clippy with warnings denied, the nightly formatting check and
cargo docpass at this tip with no new warnings.Base: #141. Next: B, HTTP lingering close, request deadlines and status framing.
Refer: #143
Assisted-by: Claude Code:claude-fable-5
Assisted-by: Codex:gpt-5