Skip to content

[Subscription] Fix WAL replay progress after realtime gaps - #18895

Merged
jt2594838 merged 3 commits into
apache:masterfrom
Caideyipi:fix/subscription-region-replay-stall
Oct 16, 2026
Merged

jt2594838 merged 3 commits into
apache:masterfrom
Caideyipi:fix/subscription-region-replay-stall

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Description

A consensus subscription queue recovering from a realtime admission gap can have pending requests and free subscription memory while its replay cursor remains unchanged. A freshly rebuilt WAL iterator installs a search-index lower bound but starts at the first retained file, and advanceTo() previously returned immediately for that same bound. Bounded replay rounds can therefore spend many iterations scanning already covered history before reaching the gap.

Allow a fresh iterator to realign at its existing bound, and perform that realignment on initialization and cursor resets. Historical files are skipped only after checking coverage of every writer; uncovered follower requests remain replayable. Repeated equal-bound probes after scanning starts keep their reader position without rechecking the same file footers.

Start the cooperative 50 ms scan budget at the first WAL probe so queue housekeeping cannot exhaust it before replay begins. Preserve the shared physical-entry budget and incomplete request fragments across yields.

Add the last completed prefetch duration/result and a WAL reader state snapshot (yield, temporary unreadability, buffered request, file version, entry offset, body-read attempts) to queue diagnostics. Warn after 60 seconds without cursor progress while lag remains positive, throttled to once per 30 seconds; reset the observation on progress, inactivity, or an empty backlog.

Validation

  • The delayed-first-probe and fresh-iterator realignment regressions fail before the fix and pass afterward.
  • Targeted WAL iterator, pending-gap/backpressure, queue, initialization, and seek tests: 87 run, zero failures/errors, one opt-in performance test skipped.
  • Full-reactor English mvn -o test-compile -DskipTests: passed (53 modules).
  • Full-reactor Chinese mvn -o test-compile -DskipTests -Pwith-zh-locale: passed (53 modules).
  • English/Chinese message key parity and git diff --check pass.

The original hour-long multi-region workload has not been rerun; end-to-end recovery latency still needs workload validation.

This PR has:

  • been self-reviewed, including concurrent lifecycle and consumer access.
  • added comments explaining the progress and coverage constraints.
  • added or extended unit tests for the changed paths.

subscriptionWALIterator.advanceTo(
startSearchIndex, this::isWriterProgressCoveredForWalFastForward);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the created subscriptionWALIterator does not advance to startSearchIndex, then what is the point of passing it in line 2518?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the coverage-aware advanceTo() into createSubscriptionWALIterator(startSearchIndex) and removed the separate initialization step from resetSubscriptionWALPosition() in 60e3ae3.

The constructor argument only sets the local search-index filtering lower bound. Skipping whole WAL files also requires the queue's writer-progress coverage check, since those files may contain uncovered follower requests. The factory now completes both steps before returning the iterator.

Extended the regression to call the factory directly and reach the target within a two-entry WAL scan budget; it fails before this change and passes afterward. Validation: 87 targeted tests, 0 failures/errors, 1 opt-in performance test skipped; full-reactor English and Chinese test-compile builds passed.

@jt2594838
jt2594838 merged commit 9a078cc into apache:master Oct 16, 2026
39 checks passed
@jt2594838
jt2594838 deleted the fix/subscription-region-replay-stall branch October 16, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants