Repository navigation
Conversation
There was a problem hiding this comment.
📋 Review Summary
This PR introduces combiners.ToIterable and refactors ToList and ToDict to inherit from a shared ToSingleElementCollection base class that uses AsIter (for GlobalWindows) or GroupByKey (for non-global windows) instead of CombineGlobally, avoiding unnecessary partial-combine overhead. The approach is clean and well-tested across empty, non-empty, and windowed cases.
🔍 General Feedback
- In
ToSingleElementCollection._expand, checkingpcoll.windowing.windowfn == window.GlobalWindows()rather thanpcoll.windowing.is_default()causesGlobalWindowsPCollections configured with non-default triggers (e.g.,Repeatedly(AfterCount(...))with.without_defaults()) to take the single-impulseDoOnce+AsIterpath instead ofGroupByKey, dropping the trigger/accumulation strategy and breaking per-pane firing.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
| raise NotImplementedError | ||
|
|
||
| def _expand(self, pcoll, post_process): | ||
| if pcoll.windowing.windowfn == window.GlobalWindows(): |
There was a problem hiding this comment.
🟠 High: Checking pcoll.windowing.windowfn == window.GlobalWindows() also matches PCollections in GlobalWindows that have non-default triggers or accumulation modes (for example, WindowInto(GlobalWindows(), trigger=Repeatedly(AfterCount(N)), accumulation_mode=AccumulationMode.DISCARDING) used with .without_defaults()). Taking the DoOnce >> Create([None]) side-input branch in that case replaces the output windowing strategy with default GlobalWindows and only fires once rather than emitting one collection per trigger pane via GroupByKey (which previously worked via CombineGlobally(...).without_defaults()).
Using pcoll.windowing.is_default() ensures that GlobalWindows with custom triggers fall through to the GroupByKey path when without_defaults() is used.
| if pcoll.windowing.windowfn == window.GlobalWindows(): | |
| if pcoll.windowing.is_default(): |
…ines. A tree-based shuffle with combining tables is waseful in this case.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #41054 +/- ##
============================================
- Coverage 57.53% 57.53% -0.01%
Complexity 2492 2492
============================================
Files 1205 1205
Lines 171468 171485 +17
Branches 2103 2103
============================================
+ Hits 98652 98656 +4
- Misses 70164 70177 +13
Partials 2652 2652
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @claudevdm for label python. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
A tree-based shuffle with combining tables is waseful in this case.