Skip to content

Sync path must wait for completion, not merely for a fault - #3212

Merged
mgravell merged 1 commit into
mainfrom
marc/investigate-resultbox-race
Sep 10, 2026
Merged

mgravell merged 1 commit into
mainfrom
marc/investigate-resultbox-race

Conversation

@mgravell

Copy link
Copy Markdown
Collaborator

ExecuteSyncImpl stopped waiting as soon as it observed IResultBox.IsFaulted:

result = TryPushMessageToBridgeSync(message, processor, source, ref server);
if (!source.IsFaulted) // if we faulted while writing, we don't need to wait
{
    ...
    if (Monitor.Wait(source, TimeoutMilliseconds)) ...
}
...
var val = source.GetResult(out var ex, canRecycle: true);

That's only safe for the case the comment describes — the message faulting inline on this thread during the write, where the completion's PulseAll ran under our own (reentrant) lock with no waiter and is already gone. IsFaulted can't distinguish that from a fault published by another thread, and those two are not the same:

  • ResultProcessor.SetException publishes the exception with no lock (a plain volatile write);
  • the matching pulse only follows later, from Message.Complete → ActivateContinuations, which needs lock(box) — held by the caller.

The message is on the wire from inside TryPushMessageToBridgeSync (WriteMessageTakingWriteLockSync flushes, then still has SetIdle, UnmarkActiveMessage and _singleWriter.Release() to do). So a caller preempted anywhere between the flush and the IsFaulted read sees the fault, skips the wait, exits the lock, and recycles the box into the [ThreadStatic] pool — while the reader thread is still on its way to PulseAll on that very object.

The next synchronous call on that thread then borrows the same box, parks in Monitor.Wait, and is woken by that stale pulse. It returns with neither result nor exception. Its real reply lands on the already-recycled box, so every later reply on that thread is shifted by one — and because the pool is thread-static, the damage survives new multiplexers and new servers.

Fix

Track completion explicitly rather than inferring it:

  • ActivateContinuations sets _completed inside the lock, immediately before the pulse.
  • The sync waiter's early-out tests IsCompleted, not IsFaulted — the inline-fault case still short-circuits (it completed under our own lock), and a fault from another thread now correctly waits for the pulse it is owed.
  • The wait is a loop on IsCompleted, so a pulse the waiter did not cause can no longer shorten it; the original deadline is preserved across such wakes so the timeout budget is unchanged.
  • Recycling clears the flag; since the box is only recycled once IsCompleted, no other thread can still be touching it.

Every path that faults a sync box does subsequently reach Message.Complete (SetExceptionAndComplete, or ComputeResult returning true, or Fail(...) followed by Complete(...) in HandleWriteException), so there is no path that now waits where it previously returned.

How it was found / verified

Reported against Garnet CI as microsoft/garnet#2110, where it surfaced as LuaScriptTests.Issue1079 losing its exception and Issue1235 later receiving it, with counts matching 1:1 — the thread-static box is the only thing in the client that carries result state across multiplexers and servers, which is what pins the diagnosis.

Reproduced end-to-end against the in-process test server. On this machine the window is too narrow to hit naturally (20k iterations, no hit, even pinned to 2 cores under load), but with a temporary 2ms sleep injected between the push and the IsFaulted read — standing in for the preemption a loaded 2-core CI runner supplies for free — it reproduces in single-digit iterations, both symptoms:

  • a PING returning null with no exception, and
  • a PING throwing the previous command's RedisServerException.

With this change the same harness is clean over repeated runs. The probe was removed before commit; the tests left behind (ResultBoxCompletionTests) pin the contract deterministically without it: a fault alone is not completion, a stray pulse does not end the wait, and recycling resets the state.

Full suite green against the docker topology on net10.0 (5941 passed) and net8.0 (5875 passed), Build.csproj -c Release clean with 0 warnings.

ExecuteSyncImpl stopped waiting as soon as it observed IResultBox.IsFaulted.
That is only safe when the message faulted inline on the calling thread during
the write - in which case the completion's PulseAll happened under our own
(reentrant) lock and is already gone. When another thread faults the box, the
two steps are separate and unsynchronized: ResultProcessor.SetException
publishes the exception with no lock, and the matching pulse only follows later,
from Message.Complete -> ActivateContinuations.

So a caller preempted between the write reaching the wire (inside
TryPushMessageToBridgeSync) and the IsFaulted read can see the fault, skip the
wait, and recycle the box into the [ThreadStatic] pool while the reader thread
is still on its way to PulseAll on that very object. The next synchronous call
on that thread borrows the same box, parks in Monitor.Wait, and is woken by that
stale pulse: it returns with neither result nor exception. Its real reply then
lands on the already-recycled box, so every later reply on the thread is shifted
by one - and because the pool is thread-static, the damage survives new
multiplexers and new servers.

Track completion explicitly instead. ActivateContinuations sets _completed
inside the lock, immediately before the pulse, and the waiter leaves only on
_completed - so a box can never be recycled while a pulse is inbound, and a
pulse the waiter did not cause no longer shortens the wait (the original
deadline is preserved across such wakes). Recycling clears the flag.

Reported against Garnet CI as microsoft/garnet#2110, where it surfaced as one
test losing its exception and a later test in the same fixture receiving it.
@mgravell
mgravell merged commit 0ee60b5 into main Sep 10, 2026
6 checks passed
@mgravell
mgravell deleted the marc/investigate-resultbox-race branch September 10, 2026 09:12
mgravell added a commit that referenced this pull request Sep 10, 2026
…budget

Two real conflicts, one of them semantic.

ConnectionMultiplexer's sync wait had been changed on both sides for
different reasons, and both are needed. #3212 made it loop until the box
reports *completion*, because a pulse we did not cause must not shorten the
wait. Ours re-reads the effective timeout on each pass, because a sync caller
commits to a duration when it parks - so relaxation that begins mid-wait would
otherwise time out at the strict timeout while an async neighbour is relaxed.
The merged loop keeps #3212's structure exactly, with only IsCompleted ending
the wait successfully, and takes its deadline from the effective timeout rather
than the configured one, on entry and on every recompute. That preserves the
shrink-back too: when a window closes, the budget drops to the configured
value and the wait ends there.

PublicAPI.Unshipped.txt was additive on both sides again - RespReaderExtensions
and the ProductVariant members from main, the BitFieldOperation operators and
TopologyRefreshInterval from here. The ProductVariant entries needed dropping
from the incoming side: they were merged once already, and taking both sides
verbatim would have listed them twice.

Verified rather than assumed, since the sync path merge was by hand: #3212's
own ResultBoxCompletionTests pass, as do all 53 maintenance-relaxation,
sync-context and timeout tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant