fix: duplicate groups after spilling in legacy hash aggregation with a nested group key - #24889
Draft
rluvaton wants to merge 1 commit into
Draft
fix: duplicate groups after spilling in legacy hash aggregation with a nested group key#24889rluvaton wants to merge 1 commit into
rluvaton wants to merge 1 commit into
Conversation
rluvaton
force-pushed
the
fix-legacy-agg-nested-key-duplicate-groups
branch
from
September 2, 2026 20:37
32b789c to
8b49968
Compare
rluvaton
marked this pull request as draft
September 2, 2026 20:44
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24889 +/- ##
==========================================
+ Coverage 81.63% 81.65% +0.02%
==========================================
Files 1123 1123
Lines 409963 409961 -2
Branches 409963 409961 -2
==========================================
+ Hits 334684 334765 +81
+ Misses 55602 55505 -97
- Partials 19677 19691 +14 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
rluvaton
force-pushed
the
fix-legacy-agg-nested-key-duplicate-groups
branch
from
September 3, 2026 07:24
8b49968 to
c281838
Compare
…a nested group key When GroupedHashAggregateStream spills and switches to merging the sorted spill files it relies on GroupOrderingFull, which requires group ids to be assigned in first-seen order. It recreated its group values collector to guarantee that only when there was more than one group column, assuming a single column always uses a sequential single-column collector. A single nested column (Struct, Map) has no specialized single-column collector and is handled by GroupValuesColumn through a row-backed column, whose vectorized interning assigns ids out of input order. The ordering then emitted groups that were still in progress and the next batch reopened them as new groups, so the same key came out more than once with its aggregates split between the rows. Always recreate the collector for the merge phase.
rluvaton
force-pushed
the
fix-legacy-agg-nested-key-duplicate-groups
branch
from
September 3, 2026 10:29
c281838 to
977bdd1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
GROUP BYon a single nested column (Struct,Map) returns duplicate groups when the legacyGroupedHashAggregateStreamspills: the same key comes out as several rows, with the aggregate values split between them. It needs the legacy stream (datafusion.execution.enable_migration_aggregate = false, or aPartialReducestage on ordered input), a single nested group key, and enough memory pressure to spill in aFinalorSinglestage. Results are silently wrong rather than an error.After spilling, the stream re-aggregates the merged spill files with
GroupOrderingFull, which requires group ids in first-seen order along the sorted input. The stream recreates its group values collector for that phase to guarantee the order, but only when there is more than one group column, assuming a single column always uses a sequential single-column collector. A single nested column has no specialized single-column collector and is served byGroupValuesColumnthrough a row-backed column, whose vectorized interning assigns new ids out of input order under hash collisions. In a merged batch of 28 sorted rows the ids came out as 0 to 4, then 9 to 13, then 5 to 8.GroupOrderingFullthen treated a group that was still arriving as complete and emitted it, and the next batch reopened it as a new group.What changes are included in this PR?
GroupedHashAggregateStreamnow always recreates the group values collector when it switches to merging spill files, instead of only for multi-column keys.What is the testing strategy for this PR?
New integration test
memory_limit::legacy_stream_nested_key_spill_keeps_groups_unique: a 200k-row table grouped by a struct of a list and an integer, with null and empty lists, null numbers and null structs mixed in, aggregated with six aggregates includingcount(distinct)on the legacy stream under a 4 MBFairSpillPoolwith a 64-row batch size, compared against the same query with unlimited memory. Without the fix it fails deterministically with 70 rows instead of 67, three keys split into two rows whose counts add up to the reference. With the fix it passes.Are there any user-facing changes?
No API changes. Queries that hit this path now return correct results.
🤖 Generated with Claude Code