Skip to content

[Subscription] Report retained WAL file gaps and retry transient reads - #18818

Merged
jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/subscription-unreadable-wal-replay
Oct 10, 2026
Merged

jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/subscription-unreadable-wal-replay

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Continuous subscription WAL replay could permanently skip unreadable retained files while subscription_consensus_wal_gap remained zero. The detailed warning only ran when the iterator was exhausted, and clients received a noncritical error containing only a file count.

This change makes file skips observable and diagnosable:

  • Add queue/region metrics subscription_consensus_wal_file_gap and subscription_consensus_wal_file_gap_entries, plus queue diagnostics. Counters accumulate across iterator resets. File counts measure skip occurrences; entry counts include only known skipped WAL entries/fragments, including discarded incomplete request fragments. An unreadable footer leaves its entry count explicitly unknown.
  • Log each permanent skip immediately with its absolute filename, version ID, file-level searchIndex bounds, entry offsets and exception. Send a critical poll error with queue/region context and up to eight recent file details; the server log contains every skipped file.
  • Retry EOF at the original file/entry offset while preserving incomplete requests. Only a repeated EOF at the same offset in an unchanged sealed retained file confirms truncation. Near-live EOF and ordinary I/O failures wait for another replay attempt. Preserve the existing escape path for entries above the supported WAL size limit, with an explicit gap signal.
  • Preserve a critical error from parallel consumer poll tasks even when sibling topics return data. Healthy messages are returned once, and the next poll raises the saved critical exception instead of permanently reducing it to a warning.
  • Explicitly declare both Logback artifacts used by the warning assertions with runtime scope. Account for their role as the server logging backend and in LogbackMetrics in the dependency analyzer's test-only-scope check.

subscription_consensus_wal_gap retains its local searchIndex unit. Monitoring and regression checks must require both it and subscription_consensus_wal_file_gap to be zero; a zero known-entry count alone does not exclude a file gap. Filename searchIndex bounds describe the whole file and are not an inferred row-loss count.

Validation

  • Full English reactor build through test: 88 tests in eight targeted subscription/WAL classes, zero failures/errors, one optional performance test skipped.
  • Full Chinese reactor test-compile -DskipTests -P with-zh-locale: passed (53 modules).
  • Spotless applied to DataNode, Node Commons and Subscription Client; new English/Chinese message keys and format specifiers match.
  • DataNode and its upstream modules (28-module reactor): test dependency:analyze-only@check-dependencies -Dtest=ProgressWALIteratorTest passed; 20 tests passed and one optional performance test skipped. Both Logback artifacts remain present in the runtime dependency tree.

This PR has:

  • been self-reviewed.
  • added comments explaining the retry, counting and error-delivery behavior.
  • added or updated unit tests for transient/sealed/live EOF, preservation of incomplete fragments, immediate file warnings, bounded diagnostics, cumulative/region-scoped gauges, and delivery of a critical error after healthy parallel poll results.

The supplied regression logs confirm six noncritical skip errors, zero detailed iterator warnings and 5,400 zero-valued WAL-gap samples. Those logs cannot identify the skipped files or establish a row-loss count; the original high-load benchmark has not been rerun.

@jt2594838
jt2594838 merged commit 054f5de into apache:master Oct 10, 2026
36 of 39 checks passed
@jt2594838
jt2594838 deleted the fix/subscription-unreadable-wal-replay branch October 10, 2026 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants