test(state): poll the published value, not the pump_cycles proxy - #326
Merged
Merged
Conversation
StateSessionStatusTest.ConflictAutoPublish fails roughly one Windows run in ten. It has been the only failing check on PRs #315 and #318, and it cost #324 a CI round too. It is a test defect, not a race and not scheduler starvation. The evidence that settles it is the duration of the same test in PASSING windows-2022 jobs: #315 bc5b9c3 FAIL 1.61s #318 eb3ec7b pass 1.71s #318 (old) FAIL 1.59s #324 branch pass 1.60s #319 branch pass 1.59s Pass and fail are indistinguishable in wall time, and the test is scheduled last in every run -- it fails alone and passes alone. So RUN_SERIAL (the #121/#325 remedy for the fork+IPC suites) cannot separate the populations; this session is transport_kind::inproc, with no fork and no socket. Nor is it a replication race: ingest_remote fills the conflict ring synchronously under mutex, and the EXPECT_GT(total_conflicts_detected(), 0u) above passes in both failing jobs. The observed "" is an auto-vivified, never-written node. The bug is the premise in the test's own comment: "At 1ms interval, 100 cycles ~= 100ms". pump_interval_ms is a sleep FLOOR, not a rate. Windows' ~15.6ms scheduler tick stretches sleep_for(1ms) to a full tick, so 100 pump cycles takes ~1.55s; meanwhile sleep_for(20ms) costs two ticks, capping the 50-iteration wait at ~1.56s. The two deadlines land one tick apart, so a single preemption exhausts the loop and the value is read one tick early. Linux clears it in ~130ms and macOS in ~1.0s, which is why only the windows-2022 lane ever saw this. Poll the published value itself instead. That is correct on any timer granularity -- it returns as soon as the publish lands -- and the assertion is now fatal and prints pump_cycles, so a future timeout reports "never published, pump_cycles=N" instead of the misleading `"" != "conflict.key"` that sent this one to the wrong suspect. Verified on linux: suite 8/8; ConflictAutoPublish alone 0.13s over three runs (unchanged -- the larger budget only costs time when the publish is genuinely slow); 0 failures in 8 concurrent runs and 0 in 5 runs pinned to a single core. Deliberately not folded in, both real but separate: * _pump_cycles is stored/loaded memory_order_relaxed. Hygiene only; the per-node mutex supplies the real edge, and this fix stops the counter being load-bearing. * The publish cadence is counted in cycles, so on Windows conflict metrics publish ~15x less often than pump_interval_ms implies. If that is meant to be a wall-clock guarantee, pump_loop should compare elapsed time -- a product change deserving its own PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
StateSessionStatusTest.ConflictAutoPublishfails roughly one Windows run in ten. It has been the only failing check on #315 and #318, and it cost #324 a CI round as well.It is a test defect — not a race, and not starvation
The evidence that settles it is the duration of the same test in passing
windows-2022jobs:bc5b9c3eb3ec7bPass and fail are indistinguishable in wall time, and the test is scheduled last in every run — it fails alone and passes alone.
RUN_SERIAL(the test(ci): RUN_SERIAL the fork()+IPC multiprocess tests to de-flake linux master #121/test(ci): RUN_SERIAL state_transport_ipc_test, the suite #121 missed #325 remedy) cannot separate two populations that are already running alone and take the same time. This session istransport_kind::inproc— no fork, no socket.ingest_remotefills the conflict ring synchronously on the test thread understd::mutex, and theEXPECT_GT(total_conflicts_detected(), 0u)above passes in both failing jobs. The write and the read both take the node's ownboost::mutex, so there is no visibility hole. The observed""is an auto-vivified, never-written node.The actual cause is the premise in the test's own comment
// At 1ms interval, 100 cycles ≈ 100ms. Wait generously for CI.pump_interval_msis a sleep floor, not a rate. Windows' ~15.6 ms scheduler tick stretchessleep_for(1ms)to a full tick, so 100 pump cycles takes ~1.55 s. Meanwhilesleep_for(20ms)costs two ticks, capping the 50-iteration wait at ~1.56 s.The two deadlines land one tick apart. A single preemption exhausts the loop, and the value is then read once, one tick early. Linux clears it in ~130 ms and macOS in ~1.0 s — which is exactly why only the
windows-2022lane ever saw this.The fix
Poll the published value itself rather than the
pump_cyclesproxy. Correct on any timer granularity: it returns as soon as the publish lands, so the common case is no slower.The assertion is now fatal and prints
pump_cycles, so a future timeout reportsnever published, pump_cycles=Ninstead of the misleading"" != "conflict.key"that sent this one to the wrong suspect in the first place.Verification (linux)
ConflictAutoPublishalone: 0.13 s across three runs — unchanged, since the larger budget only costs time when the publish is genuinely slowRejected alternatives
RUN_SERIAL TRUE— wrong here. No starvation, and it would leave a predicate that is unreachable on Windows even when the test passes; at best it masks the flake by removing the preemption that costs the tick.publish_conflictsdirectly — deterministic, butstate_conflict_ring_test.cppalready covers that path, and it would delete the only coverage of the auto-publish behaviour this test is named for.Deliberately not folded in
Both real, both separate:
_pump_cyclesis stored/loadedmemory_order_relaxed. Hygiene only — the per-node mutex supplies the real edge, and after this fix the counter is no longer load-bearing.pump_interval_msimplies. If that is meant to be a wall-clock guarantee,pump_loopshould compare elapsed time — a product change deserving its own PR.Sequencing
eb3ec7b; it is not blocked, it got lucky on the coin flip.