Repository navigation
[Subscription] Fix WAL replay progress after realtime gaps - #18895
Conversation
| subscriptionWALIterator.advanceTo( | ||
| startSearchIndex, this::isWriterProgressCoveredForWalFastForward); | ||
| } | ||
| } |
There was a problem hiding this comment.
If the created subscriptionWALIterator does not advance to startSearchIndex, then what is the point of passing it in line 2518?
There was a problem hiding this comment.
Moved the coverage-aware advanceTo() into createSubscriptionWALIterator(startSearchIndex) and removed the separate initialization step from resetSubscriptionWALPosition() in 60e3ae3.
The constructor argument only sets the local search-index filtering lower bound. Skipping whole WAL files also requires the queue's writer-progress coverage check, since those files may contain uncovered follower requests. The factory now completes both steps before returning the iterator.
Extended the regression to call the factory directly and reach the target within a two-entry WAL scan budget; it fails before this change and passes afterward. Validation: 87 targeted tests, 0 failures/errors, 1 opt-in performance test skipped; full-reactor English and Chinese test-compile builds passed.
Description
A consensus subscription queue recovering from a realtime admission gap can have pending requests and free subscription memory while its replay cursor remains unchanged. A freshly rebuilt WAL iterator installs a search-index lower bound but starts at the first retained file, and
advanceTo()previously returned immediately for that same bound. Bounded replay rounds can therefore spend many iterations scanning already covered history before reaching the gap.Allow a fresh iterator to realign at its existing bound, and perform that realignment on initialization and cursor resets. Historical files are skipped only after checking coverage of every writer; uncovered follower requests remain replayable. Repeated equal-bound probes after scanning starts keep their reader position without rechecking the same file footers.
Start the cooperative 50 ms scan budget at the first WAL probe so queue housekeeping cannot exhaust it before replay begins. Preserve the shared physical-entry budget and incomplete request fragments across yields.
Add the last completed prefetch duration/result and a WAL reader state snapshot (yield, temporary unreadability, buffered request, file version, entry offset, body-read attempts) to queue diagnostics. Warn after 60 seconds without cursor progress while lag remains positive, throttled to once per 30 seconds; reset the observation on progress, inactivity, or an empty backlog.
Validation
mvn -o test-compile -DskipTests: passed (53 modules).mvn -o test-compile -DskipTests -Pwith-zh-locale: passed (53 modules).git diff --checkpass.The original hour-long multi-region workload has not been rerun; end-to-end recovery latency still needs workload validation.
This PR has: