Repository navigation
Conversation
…Fn, and ConcatListCombineFn
|
/gemini review |
There was a problem hiding this comment.
📋 Review Summary
This PR introduces well-targeted, idiomatic performance optimizations across five built-in combiners (CountCombineFn, TopCombineFn, SampleCombineFn, ToListCombineFn, and ConcatListCombineFn). The changes eliminate quadratic list concatenations, redundant materializations of iterables, and unnecessary object allocations/method dispatches in hot loops while preserving exact semantics.
🔍 General Feedback
- All five optimizations are clean, safe, and adhere to
CombineFnaccumulator mutability contracts (notably copying the first accumulator's inner heap inTopCombineFn.merge_accumulatorsand creating fresh lists viaitertools.chain.from_iterable). - One minor edge case in
TopCombineFnwhenn <= 0is worth noting for consistency betweenadd_inputandmerge_accumulators, though it is non-blocking.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
| result_heap = heap | ||
| else: | ||
| result_heap = list(heap) | ||
| elif n > 0: |
There was a problem hiding this comment.
🟢 Nit: If n <= 0 (e.g. TopCombineFn(0)), add_input will still push up to 1 element into heap because len(heap) < self._n is False and else: heapq.heappushpop(heap, comparable) is executed on an empty list (which pushes the element and pops it, leaving heap empty—wait: heappushpop([], x) actually returns x and leaves the list empty!). However, if the first accumulator in merge_accumulators somehow had elements or if n <= 0 is guarded in add_input as well (like in SampleCombineFn.add_input), guarding if self._n <= 0: return accumulator in TopCombineFn.add_input would make TopCombineFn(0) consistent and avoid unnecessary ComparableValue allocations.
Description
Optimizes five core combiners in
sdks/python/apache_beam/transforms/combiners.py:CountCombineFn.add_inputs: Checksisinstance(elements, Sized)to returnaccumulator + len(elements)insum(1 for _ in elements)instead of allocatinglist(elements)(79.42x faster on sized batches).ToListCombineFn.merge_accumulatorsandConcatListCombineFn.merge_accumulators: Replaces quadraticsum(accumulators, [])with linearlist(itertools.chain.from_iterable(accumulators)), and usesaccumulator.extend(element)inConcatListCombineFn.add_input(20.41x faster merging 500 100-item lists).SampleCombineFn.add_input: Inlines reservoir min-heap threshold check (r > inner_heap[0][0]) to avoid tuple allocation and method dispatch when an element is not sampled (3.45x faster).TopCombineFn.merge_accumulators: Pushes directly intoresult_heapviaheappush/heappushpopinstead of re-wrapping elements throughself.add_input(3.44x faster).Testing
pytest sdks/python/apache_beam/transforms/combiners_test.py.