Skip to content

[Flink] Restore source split logs - #40354

Merged
Abacn merged 1 commit into
apache:masterfrom
pkuzmickas:pkuzmickas/flink-restore-source-split-logs
Sep 30, 2026
Merged

Abacn merged 1 commit into
apache:masterfrom
pkuzmickas:pkuzmickas/flink-restore-source-split-logs

Conversation

@pkuzmickas

Copy link
Copy Markdown
Contributor

Summary

Restore the source split logs in the Flink DataStream runner that #39874 removed when it moved splitting into FlinkSourceSplitUtils.

Before #39874, the static and lazy enumerators logged:

  • Split bounded source {} in {} splits
  • Split source {} to {} splits (unbounded)

After the refactor, only the opt-in size-based enumerator logged split counts. The default lazy and static paths logged nothing about splitting.

Why

These logs are the main way to diagnose source skew in batch jobs. For example, comparing the split count with the source parallelism shows whether a source produces too few splits to distribute work evenly.

Changes

  • Log in FlinkSourceSplitUtils.splitBoundedSource / splitUnboundedSource, so every enumerator emits the log.
  • Keep the original message prefixes so existing log searches still match. The bounded log also includes the estimated source size and desired split size.
  • Remove the size-based enumerator's post-split log, which would now be a duplicate. Its existing assignment-mode log still reports the selected mode.

  • Logging-only change; no CHANGES.md entry needed.

The move of source splitting into FlinkSourceSplitUtils dropped the
split-count logs from the default and lazy enumerators. Log in the
shared helpers so every enumerator reports how many splits a source
produced, and remove the now-duplicate size-based enumerator log.
@pkuzmickas
pkuzmickas marked this pull request as ready for review September 30, 2026 12:56
@pkuzmickas

Copy link
Copy Markdown
Contributor Author

Hey @Abacn could you please take a look? 🙏 I accidentally removed this in the previous PR we merged.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @chamikaramj added as fallback since no labels match configuration

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@Abacn
Abacn merged commit b462381 into apache:master Sep 30, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants