Repository navigation
Sync path must wait for completion, not merely for a fault - #3212
Merged
Merged
Conversation
ExecuteSyncImpl stopped waiting as soon as it observed IResultBox.IsFaulted. That is only safe when the message faulted inline on the calling thread during the write - in which case the completion's PulseAll happened under our own (reentrant) lock and is already gone. When another thread faults the box, the two steps are separate and unsynchronized: ResultProcessor.SetException publishes the exception with no lock, and the matching pulse only follows later, from Message.Complete -> ActivateContinuations. So a caller preempted between the write reaching the wire (inside TryPushMessageToBridgeSync) and the IsFaulted read can see the fault, skip the wait, and recycle the box into the [ThreadStatic] pool while the reader thread is still on its way to PulseAll on that very object. The next synchronous call on that thread borrows the same box, parks in Monitor.Wait, and is woken by that stale pulse: it returns with neither result nor exception. Its real reply then lands on the already-recycled box, so every later reply on the thread is shifted by one - and because the pool is thread-static, the damage survives new multiplexers and new servers. Track completion explicitly instead. ActivateContinuations sets _completed inside the lock, immediately before the pulse, and the waiter leaves only on _completed - so a box can never be recycled while a pulse is inbound, and a pulse the waiter did not cause no longer shortens the wait (the original deadline is preserved across such wakes). Recycling clears the flag. Reported against Garnet CI as microsoft/garnet#2110, where it surfaced as one test losing its exception and a later test in the same fixture receiving it.
mgravell
added a commit
that referenced
this pull request
Sep 10, 2026
…budget Two real conflicts, one of them semantic. ConnectionMultiplexer's sync wait had been changed on both sides for different reasons, and both are needed. #3212 made it loop until the box reports *completion*, because a pulse we did not cause must not shorten the wait. Ours re-reads the effective timeout on each pass, because a sync caller commits to a duration when it parks - so relaxation that begins mid-wait would otherwise time out at the strict timeout while an async neighbour is relaxed. The merged loop keeps #3212's structure exactly, with only IsCompleted ending the wait successfully, and takes its deadline from the effective timeout rather than the configured one, on entry and on every recompute. That preserves the shrink-back too: when a window closes, the budget drops to the configured value and the wait ends there. PublicAPI.Unshipped.txt was additive on both sides again - RespReaderExtensions and the ProductVariant members from main, the BitFieldOperation operators and TopologyRefreshInterval from here. The ProductVariant entries needed dropping from the incoming side: they were merged once already, and taking both sides verbatim would have listed them twice. Verified rather than assumed, since the sync path merge was by hand: #3212's own ResultBoxCompletionTests pass, as do all 53 maintenance-relaxation, sync-context and timeout tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ExecuteSyncImplstopped waiting as soon as it observedIResultBox.IsFaulted:That's only safe for the case the comment describes — the message faulting inline on this thread during the write, where the completion's
PulseAllran under our own (reentrant) lock with no waiter and is already gone.IsFaultedcan't distinguish that from a fault published by another thread, and those two are not the same:ResultProcessor.SetExceptionpublishes the exception with no lock (a plain volatile write);Message.Complete→ActivateContinuations, which needslock(box)— held by the caller.The message is on the wire from inside
TryPushMessageToBridgeSync(WriteMessageTakingWriteLockSyncflushes, then still hasSetIdle,UnmarkActiveMessageand_singleWriter.Release()to do). So a caller preempted anywhere between the flush and theIsFaultedread sees the fault, skips the wait, exits the lock, and recycles the box into the[ThreadStatic]pool — while the reader thread is still on its way toPulseAllon that very object.The next synchronous call on that thread then borrows the same box, parks in
Monitor.Wait, and is woken by that stale pulse. It returns with neither result nor exception. Its real reply lands on the already-recycled box, so every later reply on that thread is shifted by one — and because the pool is thread-static, the damage survives new multiplexers and new servers.Fix
Track completion explicitly rather than inferring it:
ActivateContinuationssets_completedinside the lock, immediately before the pulse.IsCompleted, notIsFaulted— the inline-fault case still short-circuits (it completed under our own lock), and a fault from another thread now correctly waits for the pulse it is owed.IsCompleted, so a pulse the waiter did not cause can no longer shorten it; the original deadline is preserved across such wakes so the timeout budget is unchanged.IsCompleted, no other thread can still be touching it.Every path that faults a sync box does subsequently reach
Message.Complete(SetExceptionAndComplete, orComputeResultreturning true, orFail(...)followed byComplete(...)inHandleWriteException), so there is no path that now waits where it previously returned.How it was found / verified
Reported against Garnet CI as microsoft/garnet#2110, where it surfaced as
LuaScriptTests.Issue1079losing its exception andIssue1235later receiving it, with counts matching 1:1 — the thread-static box is the only thing in the client that carries result state across multiplexers and servers, which is what pins the diagnosis.Reproduced end-to-end against the in-process test server. On this machine the window is too narrow to hit naturally (20k iterations, no hit, even pinned to 2 cores under load), but with a temporary 2ms sleep injected between the push and the
IsFaultedread — standing in for the preemption a loaded 2-core CI runner supplies for free — it reproduces in single-digit iterations, both symptoms:PINGreturningnullwith no exception, andPINGthrowing the previous command'sRedisServerException.With this change the same harness is clean over repeated runs. The probe was removed before commit; the tests left behind (
ResultBoxCompletionTests) pin the contract deterministically without it: a fault alone is not completion, a stray pulse does not end the wait, and recycling resets the state.Full suite green against the docker topology on net10.0 (5941 passed) and net8.0 (5875 passed),
Build.csproj -c Releaseclean with 0 warnings.