Skip to content

test(state): poll the published value, not the pump_cycles proxy - #326

Merged
transfix merged 1 commit into
masterfrom
fix/state-session-status-conflict-publish-poll
Sep 4, 2026
Merged

transfix merged 1 commit into
masterfrom
fix/state-session-status-conflict-publish-poll

Conversation

@transfix

@transfix transfix commented Sep 4, 2026

Copy link
Copy Markdown
Owner

StateSessionStatusTest.ConflictAutoPublish fails roughly one Windows run in ten. It has been the only failing check on #315 and #318, and it cost #324 a CI round as well.

It is a test defect — not a race, and not starvation

The evidence that settles it is the duration of the same test in passing windows-2022 jobs:

job branch result duration
100458098681 #315 bc5b9c3 FAIL 1.61 s
100488129413 #318 (old head) FAIL 1.59 s
100898090375 #318 eb3ec7b pass 1.71 s
100896505825 #324 branch pass 1.60 s
100877333733 #319 branch pass 1.59 s

Pass and fail are indistinguishable in wall time, and the test is scheduled last in every run — it fails alone and passes alone.

The actual cause is the premise in the test's own comment

// At 1ms interval, 100 cycles ≈ 100ms. Wait generously for CI.

pump_interval_ms is a sleep floor, not a rate. Windows' ~15.6 ms scheduler tick stretches sleep_for(1ms) to a full tick, so 100 pump cycles takes ~1.55 s. Meanwhile sleep_for(20ms) costs two ticks, capping the 50-iteration wait at ~1.56 s.

The two deadlines land one tick apart. A single preemption exhausts the loop, and the value is then read once, one tick early. Linux clears it in ~130 ms and macOS in ~1.0 s — which is exactly why only the windows-2022 lane ever saw this.

The fix

Poll the published value itself rather than the pump_cycles proxy. Correct on any timer granularity: it returns as soon as the publish lands, so the common case is no slower.

The assertion is now fatal and prints pump_cycles, so a future timeout reports never published, pump_cycles=N instead of the misleading "" != "conflict.key" that sent this one to the wrong suspect in the first place.

Verification (linux)

  • suite 8/8
  • ConflictAutoPublish alone: 0.13 s across three runs — unchanged, since the larger budget only costs time when the publish is genuinely slow
  • 0 failures in 8 concurrent runs, and 0 in 5 runs pinned to a single core (worst-case preemption)

Rejected alternatives

  • RUN_SERIAL TRUE — wrong here. No starvation, and it would leave a predicate that is unreachable on Windows even when the test passes; at best it masks the flake by removing the preemption that costs the tick.
  • Bump 50 → 60 iterations — works numerically but keeps the test coupled to the host tick rate, and breaks again on any slower runner.
  • Call publish_conflicts directly — deterministic, but state_conflict_ring_test.cpp already covers that path, and it would delete the only coverage of the auto-publish behaviour this test is named for.

Deliberately not folded in

Both real, both separate:

  • _pump_cycles is stored/loaded memory_order_relaxed. Hygiene only — the per-node mutex supplies the real edge, and after this fix the counter is no longer load-bearing.
  • The publish cadence is counted in cycles, so on Windows conflict metrics publish ~15× less often than pump_interval_ms implies. If that is meant to be a wall-clock guarantee, pump_loop should compare elapsed time — a product change deserving its own PR.

Sequencing

StateSessionStatusTest.ConflictAutoPublish fails roughly one Windows run in
ten. It has been the only failing check on PRs #315 and #318, and it cost
#324 a CI round too.

It is a test defect, not a race and not scheduler starvation. The evidence
that settles it is the duration of the same test in PASSING windows-2022
jobs:

    #315 bc5b9c3   FAIL   1.61s        #318 eb3ec7b   pass   1.71s
    #318 (old)     FAIL   1.59s        #324 branch    pass   1.60s
                                       #319 branch    pass   1.59s

Pass and fail are indistinguishable in wall time, and the test is scheduled
last in every run -- it fails alone and passes alone. So RUN_SERIAL (the
#121/#325 remedy for the fork+IPC suites) cannot separate the populations;
this session is transport_kind::inproc, with no fork and no socket. Nor is it
a replication race: ingest_remote fills the conflict ring synchronously under
mutex, and the EXPECT_GT(total_conflicts_detected(), 0u) above passes in both
failing jobs. The observed "" is an auto-vivified, never-written node.

The bug is the premise in the test's own comment: "At 1ms interval, 100
cycles ~= 100ms". pump_interval_ms is a sleep FLOOR, not a rate. Windows'
~15.6ms scheduler tick stretches sleep_for(1ms) to a full tick, so 100 pump
cycles takes ~1.55s; meanwhile sleep_for(20ms) costs two ticks, capping the
50-iteration wait at ~1.56s. The two deadlines land one tick apart, so a
single preemption exhausts the loop and the value is read one tick early.
Linux clears it in ~130ms and macOS in ~1.0s, which is why only the
windows-2022 lane ever saw this.

Poll the published value itself instead. That is correct on any timer
granularity -- it returns as soon as the publish lands -- and the assertion
is now fatal and prints pump_cycles, so a future timeout reports "never
published, pump_cycles=N" instead of the misleading `"" != "conflict.key"`
that sent this one to the wrong suspect.

Verified on linux: suite 8/8; ConflictAutoPublish alone 0.13s over three
runs (unchanged -- the larger budget only costs time when the publish is
genuinely slow); 0 failures in 8 concurrent runs and 0 in 5 runs pinned to a
single core.

Deliberately not folded in, both real but separate:
  * _pump_cycles is stored/loaded memory_order_relaxed. Hygiene only; the
    per-node mutex supplies the real edge, and this fix stops the counter
    being load-bearing.
  * The publish cadence is counted in cycles, so on Windows conflict metrics
    publish ~15x less often than pump_interval_ms implies. If that is meant
    to be a wall-clock guarantee, pump_loop should compare elapsed time --
    a product change deserving its own PR.
@transfix
transfix merged commit 6bd54cf into master Sep 4, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant