[mgravell]

Problem

The MoveTo and Recycle handoff paths drained by waiting for "no caller work" before replacing the connections. Under steady traffic that never happens: there is always a command in flight, so the drain waited out the entire announced window and recycled at the deadline, failing whatever was in flight at that instant and leaving no time to establish the replacement.

Measured live on a Redis Enterprise endpoint rebind (single proxy policy, maintMovingEndpointType=Auto) with about 450 ops/s on the connection:

event	before	after
MOVING received	+5.8s	+5.8s
MoveTo decided	+6.6s	+6.6s
connections replaced	+20.9s	+6.7s
ConnectionRestored	+20.9s	+6.8s
failed commands	8	0
"did not establish a replacement within the announced 15000ms"	yes	no

The timing matched the code to the tenth of a second: 15s window, 0.8s jitter, DrainThenRecycleAsync waiting out the remaining 14.2s.

Change

PhysicalBridge gets a quiesce mode for the duration of the drain:

• new caller messages (single, batch, sync and async) go to the backlog instead of the socket, regardless of backlog policy, since the connection is live and the message would otherwise have been written;
• the backlog processor does not write while quiescing;
• the drain waits only for commands already written (HasCallerWorkInFlight), not for the backlog;
• held work is written to the replacement once it is established, as across any reconnect. A FailFast backlog policy no longer abandons it for a MaintenanceHandoff, which is a replacement we chose rather than a failure;
• internal traffic (keep-alives) is not held.

The retirement path (RetireAsync) already did the equivalent by marking the server unselectable first; that is not usable for a handoff on a single-endpoint database, hence the bridge-level hold.

Tests

• New MovingUnderSteadyLoadRecyclesPromptlyAndFailsNothing: steady load against the in-process server, MOVING with a 10s window and a named successor; asserts the recycle follows within a fraction of the window and no command fails. Fails on main (recycle at ~10s), passes here.
• Maintenance* (168), *Backlog*, *Connect*, RetirementUnderMaintenanceTests all pass against the docker topology. Full net10.0 suite: 6875 total, 1 failure, FailoverTests.SubscriptionsSurvivePrimarySwitchAsync (primaryChanged >= 12 got 10), which fails the same way on unmodified main on this machine (1 pass in 4 runs), so not from this change.
• Live: the table above, against RS 8.0.22 through the fault injector's endpoint_rebind.

[mgravell]

Self-review notes (independent review pass over the diff, recorded here so they don't get lost). The quiesce/drain approach holds up in principle; the gaps are at the edges of the window. Six behavioral items and two doc items, roughly in severity order:

1. A blocking command holds the whole bridge for the window. HasCallerWorkInFlight makes the drain hostage to the slowest in-flight command: one XREAD BLOCK 30000 / WAIT / slow command keeps it true, so all new traffic sits in the backlog for the rest of the maintenance window (sync callers eat their sync timeout; async callers see the full-window latency spike), and the blocking command is failed by the recycle at the deadline anyway. On main, traffic kept flowing on the live connection during the drain — so this is a regression precisely for workloads with blocking reads present. Probably wants a cap on the drain wait (or excluding commands that cannot complete inside the window) rather than full quiescence.

2. The drain doesn't fully close the race it exists for. A writer that read _quiescing == false at the gate but hasn't yet registered its message in _writtenAwaitingResponse is invisible to the poll: the drain can observe "no caller work", recycle, and the resumed writer hits the disposed connection — the exact failure class this PR is meant to remove. Nothing excludes a mid-write racer because the drain never takes _singleWriter; a brief acquire/release of it at recycle time would close the window (and MovingUnderSteadyLoadRecyclesPromptlyAndFailsNothing is rarely-flaky until it's closed).

3. HoldForHandoff bypasses the "never ever queue a QUIT" invariant. ServerEndPoint.Close sends QUIT through TryWriteAsync without SetInternalCall; during a quiesce that gets backlogged and later written to the freshly established replacement, which the server then closes. QueueOrFailMessage guards exactly this, and HoldForHandoff duplicates its queue branch minus the guard — routing the hold through that branch fixes the bug and removes the duplication in one move.

4. The hold overrides BacklogPolicy.FailFast once the connection actually drops. The quiesce check precedes the IsConnected check, so if the old connection dies mid-drain, new FailFast-policy commands are still backlogged — converting the policy's immediate NoConnectionAvailable into a long hang if the replacement then fails to establish. The "connection is live" premise in the comment should be enforced: hold only while connected.

5. A held enqueue racing EndQuiesce stalls until the heartbeat. HoldForHandoff enqueues without Set()/StartBacklogProcessor; if EndQuiesce saw an empty backlog just before the enqueue lands, nothing kicks the processor until OnHeartbeat (~1s) — an avoidable latency spike in a PR whose point is handoff latency. (QueueOrFailMessage has the same shape, but it's only reachable disconnected, where OnFullyEstablished re-kicks; here the bridge is connected and nothing else will.)

6. The batch path holds internal calls. TryEnqueue's quiesce path has no IsInternalCall exemption, while TryWriteSync/TryWriteAsync both have it — the "internal traffic is not held" invariant is enforced in two of the three entry points. Narrow window today, but an asymmetry worth fixing while it's cheap.

7. Orphaned XML doc. HasCallerWorkInFlight landed between CheckBacklogForTimeouts and its <summary>, so the predicate now carries "Crawls from the head of the backlog queue, consuming anything that should have timed out" (wrong for a read-only check) and CheckBacklogForTimeouts has none — the comment needs to move back down.

8. docs/ServerMaintenanceEvent.md still describes the old behavior. The user-visible changes (new commands held during the drain regardless of backlog policy; FailFast no longer aborting held work on the handoff disconnect) should land there too.

Items 3–8 are mechanical; 1 and 2 need a design decision (capped drain + _singleWriter at recycle being the likely shape) before this merges.

[mgravell]

Addressed in bc52360, item by item:

1. Blocking command holds the bridge - the drain is now capped at one second (DrainCap) as well as by the window. Ordinary commands clear in a round trip; anything still in flight after a second is a blocking or stuck command that would not finish inside the window either, so past the cap the connections are replaced regardless and it fails as the server's close would have failed it. New test MovingWithSlowCommandsInFlightRecyclesAtTheDrainCap (3s latency on every command, recycle asserted well under 3s, traffic resumes on the replacement).
2. Mid-write racer - the recycle goes through PhysicalBridge.TryRecycleWhenDrainedAsync, which takes _singleWriter, re-checks HasCallerWorkInFlight under it, and recycles while holding it. A writer that reaches the lock after the recycle finds its connection replaced and queues for the replacement instead of writing to a dead one (ConnectionReplacedWhileWaiting). That re-route is deliberately narrow: only a connection a handoff recycled (_recycledForHandoff, cleared on full establishment), never for bypassBacklog, never for internal traffic. The first cut applied to any changed connection and broke Issue3167Tests.WriteLosingItsConnectionIsReportedAsAClosure, which is the right test to have caught it: an ordinary closure is still reported as a closure.
3. QUIT - HoldForHandoff is gone; the hold is QueueOrFailMessage(message, holdForHandoff: true), so the never-queue-QUIT guard applies. QUIT is also excluded from the hold check itself and goes to the live old connection.
4. FailFast once dropped - ShouldHoldForHandoff requires IsConnected && !NeedsReconnect; after the old connection drops, the ordinary disconnected rules and the policy apply.
5. Enqueue racing EndQuiesce - a held enqueue that observes _quiescing == false kicks the processor itself (KickBacklogProcessor: _backlogAutoReset.Set() + StartBacklogProcessor()), which EndQuiesce uses too.
6. Batch path - held only when the batch has a caller-facing non-QUIT message, and then as a unit, since a batch is written as a unit; an all-internal batch is not held.
7. Orphaned doc - moved back to CheckBacklogForTimeouts.
8. Docs - docs/ServerMaintenanceEvent.md describes the hold, its bound, the policy exemption and the FailFast behaviour.

Verification: Moving (11), Maintenance (169), Backlog, Connect, Issue3167 and the Release analyzer build all pass; full net10.0 suite 6876 total with the one known-flaky FailoverTests.SubscriptionsSurvivePrimarySwitchAsync (fails 3 of 4 on unmodified main here). Live, with this revision: MOVING +5.9s, handoff +6.4s, restored +6.4s, 0 failed at ~450 ops/s.

[mgravell]

Self-review notes, round 2 (independent pass over the updated head, bc52360). First the good news: none of round 1 resurfaced — the QUIT-via-QueueOrFailMessage guard, hold-only-while-connected, the backlog kick, the batch-path internal exemption, and both doc items all check out. The new machinery (drain cap, lock-held recycle, ConnectionReplacedWhileWaiting) brings its own crop, though, and two items are at least as serious as what they replaced:

1. The drain pass re-kills its own replacement. RecycleConnection → OnDisconnected → TryConnect assigns a fresh physical synchronously inside the recycle, so when a pass is retried (interactive done, subscription still draining), the next pass 20ms later sees the brand-new interactive connection with no caller work in flight (handshake traffic is internal) and recycles it — and again every 20ms until the subscription drains or the cap fires (the cap path interactive?.RecycleConnection(reason) has the same defect). Each extra kill pushes failConnectCount past 1, so OnDisconnected stops reconnecting immediately and the endpoint waits on the heartbeat/retry policy: the handoff manufactures a connectivity gap plus spurious ConnectionFailed events. The comment "a bridge recycled on an earlier pass has nothing left to recycle, so a partial pass is safe to repeat" is false — needs per-bridge recycled tracking, or match against the connection that existed when the quiesce began.

2. The raced-writer requeue fails the exact command it exists to save. All three ConnectionReplacedWhileWaiting sites (sync ~1011, async fast path ~1539, awaited ~1579) requeue via QueueOrFailMessage(message) without holdForHandoff: true, so under FailFast the command falls through to FailDueToNoConnection — never written, connection replaced by choice, yet failed. Contradicts the PR's own invariant ("a FailFast policy no longer abandons held work for a MaintenanceHandoff").

3. Requeue ordering. A writer parked on the lock when the quiesce began gets requeued behind commands the caller issued after it (caller issues A, B, C; A parks; B and C are held; A is appended last; replacement executes B, C, A). The AVOID REORDERING discipline (TryPushToBacklog onlyIfExists) exists precisely for this — requeue at the head, or re-route racing writers through the hold check before they reach the lock.

4. The fixed 1s cap force-fails commands that would have finished. A BLPOP/XREAD BLOCK with a 5s timeout inside a 15s window completed on main (drain waited the remaining window) and is now failed at 1s. The XML doc's justification ("anything still in flight after this will not finish inside the window either") doesn't hold for blocking commands — and with quiesce in place, the cost of waiting longer is only latency on backlogged work, not failures. The cap should be window-derived/configurable, or at minimum generous.

5. The cap-expiry and lock-timeout fallback recycle without the write lock — reopening the mid-WriteMessageInsideLock loss the lock discipline was built to prevent (message fails with a write error, lost outright under FailFast, instead of being queued for the replacement). Also _recycledForHandoff = current on that path is a plain write with no lock/fence pairing against the lock-protected readers.

6. The "1s" bound isn't one. Each drain attempt first awaits _singleWriter.TryTakeAsync() with the full mutex timeout (default 5s), and the cap is only checked between attempts — one slow lock holder means a single attempt eats ~5s per bridge, sequentially for interactive then subscription.

7. EndQuiesce in the shared finally can leave the subscription quiesced forever. If interactive?.EndQuiesce() throws (e.g. _backlogAutoReset.Set() on a bridge disposed mid-handoff → ObjectDisposedException), subscription?.EndQuiesce() never runs; a still-connected subscription bridge with _quiescing stuck true backlogs every caller message while the processor refuses to write (while (IsConnected && !_quiescing)) — commands hang until timeout, indefinitely. Guard the Set, or give each bridge its own try/finally.

8. Cleanups: the ConnectionReplacedWhileWaiting guard is copy-pasted at the three lock sites where one check at the top of WriteMessageInsideLock would cover them all (plus the backlog processor's own stale snapshot); and ServerEndPoint.HasCallerWorkInFlight() is now uncalled — leftover from the round-1 polling design.

Shape of the fix for 1/2/3/5: track the quiesce-start connection per bridge and only ever recycle that instance, under the lock on every path, and make the requeue path a holdForHandoff head-insert. 4 and 6 are policy choices (cap value/derivation, cap-vs-lock-timeout interaction) worth deciding deliberately rather than inheriting from constants.

[mgravell]

Round 2 addressed in the latest commit:

1. Re-killing the replacement - BeginQuiesce records the connection current at that moment (_quiesceTarget), and TryRetireQuiesceTargetAsync only ever retires that instance, tracking per bridge whether it has already done so. A replacement that OnDisconnected → TryConnect establishes is never touched however many passes the loop takes; a target that went away by itself returns "nothing to retire". The false comment is gone with the method it was on.
2. Requeue under FailFast - the requeue path no longer goes through QueueOrFailMessage at all; it is BacklogRequeueFront, which queues unconditionally (held work), with the same counter accounting as BacklogEnqueue.
3. Ordering - a _backlogFront queue that every backlog operation (BacklogTryDequeue, BacklogTryPeek, BacklogHasItems, HasCallerWork, the pending counts) drains first; raced writers land there, ahead of what the hold queued after them.
4. Cap - now half of maintRelaxedTimeout (5s by default), bounded by the window, and the hold relaxes timeouts for its duration (ExtendRelaxedWindow(cap + relaxed)), so a blocking command with a timeout up to the cap completes and nothing held is timed out by the hold. The rationale for that particular derivation is in the DrainCap remarks: window-derived would be wrong in both directions, and this is the one knob an application already uses to say how patient it is during maintenance. Tests cover both sides: a 3s command is answered and the recycle follows it; a 9s one is failed at the cap and traffic resumes.
5. Unlocked recycles - none left. The forced pass goes through the same lock-held method; if the lock cannot be obtained even with the grace, the connection is left to the server's close (trace says so) rather than recycled around a writer. _recycledForHandoff is only written under the lock now.
6. Lock wait bound - TryTakeAsync takes a cancellation token owned by the drain (cap + 1s grace), so the mutex timeout no longer decides anything here; the drain loop's own clock does.
7. EndQuiesce - each bridge in its own try/finally (an exception is reported via OnInternalError), and the Set() in KickBacklogProcessor swallows ObjectDisposedException.
8. Cleanups - both HasCallerWorkInFlight predicates removed (the bridge reads target.HasCallerMessagesAwaitingResponse() under the lock directly); the three lock sites share one TryRequeueForHandoff helper. I kept the check at the call sites rather than inside WriteMessageInsideLock: the callers flush and SetIdle on the connection after a Success, which must not happen for a requeue, and the backlog processor's own loop re-reads this.physical per message under the lock, so its snapshot is already current.

Verification as before: Moving (12), Maintenance (170), Backlog, Connect, Issue3167 and the Release analyzer build pass; full net10.0 suite 6877 with only the known-flaky failover test. Live with this revision: rebind MOVING +5.7s, handoff +6.5s, 0 failed; maintenance mode MIGRATING/MIGRATED/MOVING chain, handoff 0.6s after MOVING, 0 failed, both bridges involved.

[mgravell]

Self-review notes, round 3. Round 2's nine items all check out on the new head — per-bridge recycle tracking, the holdForHandoff requeue, the head-insert, the window-derived cap, and the lock discipline all landed. The remaining findings are narrower races in the new machinery rather than design-level problems, but the first one is the worst failure mode of any round so far:

1. BacklogRequeueFront breaks the backlog head invariant — can lose a message permanently. It inserts into _backlogFront under _singleWriter but not under lock(_backlog), while CheckBacklogForTimeouts does peek-then-dequeue-and-compare under that lock. A requeue landing between the peek and the dequeue makes BacklogTryDequeue return the front message: message != message2 throws "Thread safety bug detected" from the backlog thread, and the dequeued live message is dropped uncompleted — it's in neither _writtenAwaitingResponse nor the backlog, so no timeout sweep ever fails it and an async caller awaits forever. The single-queue invariant was "only locked dequeuers change the head"; the front queue's enqueue has to take the same lock (or the peek/dequeue pair needs to tolerate a front insert).

2. OnFullyEstablished clears _recycledForHandoff too early, and outside the write lock. _singleWriter is a SemaphoreSlim (not fair), so a writer that snapshotted the old physical can be bypassed repeatedly by the backlog processor flushing held messages — long enough for the replacement to fully establish and OnFullyEstablished to null the marker. The writer then fails the TryRequeueForHandoff reference check, writes to the disposed old connection, and HandleWriteException → MarkNeedsReconnect tears down the healthy replacement; the message fails as WriteFailure despite never being written. Clear only in BeginQuiesce (or check against the current physical plus disposal); the field's "written and read under the write lock" comment is also false for this clear site.

3. MaintenanceRelaxedTimeout = 0 makes DrainCap zero. Zero is parse-accepted and is the documented way to disable timeout relaxation; with it, the first drain pass computes force=true and recycles with caller work still awaiting responses — relaxation-disabled users regress from "usually zero failed commands" to "guaranteed failures on every handoff". The cap needs a floor (even 1s).

4. EndQuiesce couples the two bridges. It only runs after both retire results resolve, but ShouldHoldForHandoff checks _quiescing and not _quiesceRecycled, and the write-loop gate is IsConnected && !_quiescing — so when interactive retires in one round trip and subscription takes the full cap+grace (~6s), all interactive traffic is held and its backlog refuses to flush to the fully established replacement for those seconds. Release each bridge as its result resolves (per-bridge EndQuiesce), or let _quiesceRecycled unlock the hold and the backlog gate.

5. Dispose mid-drain NREs. The drain loop re-reads the mutable interactive/subscription fields each pass and null-forgives them; ServerEndPoint.Dispose nulls those fields, so disposal during the 20ms poll loop throws NRE (surfaced as an InternalError event during shutdown), and the disposed bridge's _singleWriter can throw ObjectDisposedException where only OCE is caught. Snapshot both bridges into locals before BeginQuiesce.

6. A QUIT inside a held batch is failed against a live connection. TryEnqueue's hold branch routes every batch message through QueueOrFailMessage(holdForHandoff: true), whose never-queue-a-QUIT guard falls through to FailDueToNoConnection — the hold decision and the per-message action disagree. Exotic, but cheap to align.

7. EndQuiesce vs enqueuer is a store-load race. EndQuiesce writes _quiescing=false then reads the queues; an enqueuer writes the queue then reads _quiescing — both can see the old value (volatile here is release/acquire, not a fence), neither kicks, and the held message waits ~1s for the heartbeat. Same shape between ProcessBridgeBacklog's Reset() and its gate re-check. An unconditional kick when EndQuiesce transitions the flag (or an interlocked handshake) closes it.

8. Doc nit: the lockDeadline comment says "the drain decides how long, not the mutex timeout", but AwaitableMutex.TryTakeAsync still bounds the wait by _timeoutMilliseconds — whichever ends first. The retry loop hides it today; the comment shouldn't promise what the code doesn't do.

Trajectory check: round 1 was design, round 2 was the new mechanisms' blunt edges, round 3 is memory-ordering and lifecycle corners — converging, but (1) and (2) are merge-blockers; (3)–(5) are small deliberate fixes; (6)–(8) are alignment/comment work.

[mgravell]

Round 3 addressed in the latest commit:

1. Head invariant - BacklogRequeueFront inserts under lock(_backlog), the same lock the timeout sweep's peek/dequeue/compare holds. Lock order is _singleWriter then _backlog, which is the order ProcessBridgeBacklog already uses, so no new ordering is introduced.
2. Marker lifetime - _recycledForHandoff is no longer cleared in OnFullyEstablished; it is cleared only in BeginQuiesce, and the field comment now says exactly when it is set, read and cleared (set and read under the lock; cleared at the next hold).
3. Zero relaxed timeout - DrainCap has a one-second floor (DrainFloor); a new test runs the steady-load handoff with MaintenanceRelaxedTimeout = TimeSpan.Zero and asserts the same prompt, failure-free recycle.
4. Per-bridge release - each bridge's EndQuiesce runs the moment its own retirement resolves, inside the loop; the finally only covers what the loop did not release (lock-not-obtained exit, exceptions). EndQuiesce is idempotent.
5. Dispose mid-drain - the drain works on snapshots of interactive/subscription taken before BeginQuiesce; the mutex take catches ObjectDisposedException (nothing left to retire) and the release tolerates it too.
6. QUIT in a batch - a batch containing a QUIT is not held at all; the whole batch goes to the live connection, so the decision and the per-message action agree.
7. Store-load race - EndQuiesce kicks the processor unconditionally on the transition (one idle pass when nothing is queued), and BacklogRequeueFront kicks unconditionally as well.
8. Comment - rewritten: the deadline and the mutex timeout bound each wait, whichever ends first, and the drain re-waits until the deadline, so the deadline decides overall.

Verification: Moving (13), Maintenance (171), Backlog, Connect, Issue3167 and the Release analyzer build pass; full net10.0 suite 6878 total, 0 failed this time (the flaky failover test happened to pass). Live: rebind MOVING +5.8s, handoff +6.1s, 0 failed; maintenance mode this run drained a node the proxy was not on, so MIGRATING/MIGRATED only, 0 failed, as expected.

[mgravell]

Self-review notes, round 4 (head bd786f3). Round 3's items all resolved — the front queue now takes the dequeuer lock, the marker clear moved to BeginQuiesce, the floor/per-bridge/dispose-snapshot fixes are in. This round's findings cluster around a different question: what happens when the connection fails during the hold, or the hold outlives its budget. In severity order:

1. A natural socket death mid-hold abandons the held work under FailFast. The failureType != MaintenanceHandoff carve-out in OnConnectionFailed only covers the recycle the handoff performs itself. If the old connection dies during the hold — window expiry closes it by design, or any blip — AbandonPendingBacklog fails every held command: work that was only in the backlog because the hold diverted it off a live connection, and that pre-PR would have been written and answered. The hold needs its own marker (e.g. abandon skips while quiescing, letting the replacement flush), not just the failure-type exemption.

2. The 1s DrainFloor holds without relaxation cover. With maintRelaxedTimeout=0 (the documented way to disable relaxation) and a sub-second configured timeout, every MOVING holds all commands ≥1s while GetEffectiveTimeoutMilliseconds still returns the configured value — the sweep times out everything the hold is protecting. The new test passes only because the default 5s timeout exceeds the floor. Either the hold must impose a minimum effective timeout of (hold budget + margin) regardless of relaxation config, or the floor can't exceed the effective timeout.

3. BacklogRequeueFront skips Multiplexer.Root() on the empty→non-empty transition that BacklogEnqueue performs for exactly this reason: a requeued message into an otherwise-empty backlog doesn't pin the multiplexer, so an unrooted multiplexer can be collected with the caller's task permanently incomplete.

4. The drain budget is computed before DecideAsync spends the window. remaining = window − jitter is taken before the DNS poll, which production data says can eat 4–19s of a 15s window. The drain then starts a 5s-cap hold with ~1–2s of real window left — guaranteeing the collision in (1) when the server closes the socket. Budget should be decremented by DecideAsync's elapsed time (and the hold skipped entirely when what's left is below the floor).

5. Back-to-back handoffs clear _recycledForHandoff under a parked writer. BeginQuiesce nulls the previous handoff's marker; a writer starved across two handoffs by the unfair lock (the starvation the field's own doc-comment warns about) then passes the ReferenceEquals check and writes to the first handoff's disposed connection. Rolling maintenance is serial MOVINGs, so this isn't exotic — keep a small set, or clear only on successful establishment.

6. AbandonPendingBacklog dequeues without the dequeuers' lock while the front-queue comment now (correctly) asserts "only locked parties change the head". Pre-existing, but this PR makes it reachable: the hold deliberately builds a backlog that OnConnectionFailed can abandon concurrently with the sweep's locked peek/dequeue — "Thread safety bug detected", faulted backlog processor, dropped message. Take the lock in abandon.

7. Quiesced backlog passes contend the write lock for nothing. Each heartbeat kick blocks in TryTakeSync just to observe _quiescing and release — traffic the 20ms retire polls compete with on an unfair lock. An early return when _quiescing && IsConnected (after the timeout sweep) avoids it.

8. TryEnqueue re-implements ShouldHoldForHandoff inline and its batch variant holds internal messages inside a mixed batch — the predicates have already drifted once (NeedsReconnect) and will again; one shared predicate, one pass over the batch.

9. Dispose-race double-complete: StartBacklogProcessor's else-branch Set() has no ODE guard (the one in KickBacklogProcessor does), so a dispose between them propagates ODE into the write path's catch, which exception-completes a message that is also sitting in _backlogFront for the disposed-drain to complete again.

10. Docs overclaim on blocking commands. "A blocking command with a timeout up to [half the relaxed timeout] completes normally" — but a blocking command issued during the hold is backlogged, not in flight, and min(budget, half) reintroduces the window-derived cap for short windows, so a 3s BLPOP in a 2s window is failed at the cap even though the adjacent prose says exactly that shouldn't happen. The doc and the clamp need to agree.

Pattern this round: rounds 1–3 were the hold's internal mechanics; these are its interactions with the pre-existing failure paths (OnConnectionFailed/abandon/Root/timeout sweep) and with budget reality (DNS-delayed DecideAsync). 1–4 are merge-blockers to my eye; 5–6 are small locked/lifecycle fixes; 7–10 are tidy-ups and doc truth.

[mgravell]

Round 4 addressed in the latest commit:

1. Natural death mid-hold - AbandonPendingBacklog is skipped while _quiescing, not only for the MaintenanceHandoff failure type. The held backlog is work diverted off a live connection, so a window-expiry close or a blip during the hold leaves it for the replacement to flush. Once the hold is released (which the drain does as soon as the target is seen to be gone), the ordinary FailFast rules apply to new work.
2. Floor without relaxation cover - the hold now imposes its own floor on the effective timeout (SetHoldCover, read in GetEffectiveTimeoutMilliseconds as a third operand alongside the configured and relaxed values) for the cap plus the lock grace plus the connect timeout, independent of maintRelaxedTimeout. With relaxation disabled and a sub-second timeout, held work is no longer swept during the hold.
3. Root() - BacklogRequeueFront computes the empty-to-non-empty transition before the insert and roots the multiplexer on it, as BacklogEnqueue does.
4. Budget after DecideAsync - remaining is recomputed from the stopwatch after the decision (and after the half-window delay on the RecycleAtHalfWindow path). DrainThenRecycleAsync with less than the floor left skips the hold: cap zero, so the first pass retires at once, still under the lock, and the replacement gets whatever is left rather than colliding with the server's close.
5. Back-to-back handoffs - the marker is a ring of the last four retired connections, never cleared; BeginQuiesce no longer touches it.
6. Abandon under the lock - AbandonPendingBacklog dequeues under lock(_backlog) and completes outside it.
7. Quiesced passes - ProcessBridgeBacklog returns before contending for the write lock when _quiescing (after the timeout sweep); the release kicks it again.
8. One predicate - ShouldHoldForHandoff is HoldActive && IsHoldable(message); the batch gate is HoldActive && ShouldHoldBatch(messages), one pass that returns false on any QUIT and true on any holdable member. An internal member of a mixed batch does ride along, deliberately: splitting a batch would be worse than holding it, and the comment says so.
9. Dispose race - the Set() in StartBacklogProcessor's else-branch is guarded like the one in KickBacklogProcessor.
10. Docs - both the DrainCap remarks and docs/ServerMaintenanceEvent.md now state the clamp as it is: the window bounds the hold because the server closes the socket when the window ends, a blocking command already in flight whose timeout fits completes, and one issued during the hold is held and runs on the replacement. The contradictory 2s-window sentence is gone.

Verification: Moving (13), Maintenance (171), Backlog, Connect, Issue3167 and the Release analyzer build pass; full net10.0 suite 6878 with only the known-flaky failover test (it passed on the previous run and failed on this one, as it does on unmodified main). Live: rebind MOVING +5.9s, handoff +6.4s, restored +6.5s, 0 failed.