Skip to content

fix: duplicate groups after spilling in legacy hash aggregation with a nested group key - #24889

Draft
rluvaton wants to merge 1 commit into
apache:mainfrom
rluvaton:fix-legacy-agg-nested-key-duplicate-groups
Draft

fix: duplicate groups after spilling in legacy hash aggregation with a nested group key#24889
rluvaton wants to merge 1 commit into
apache:mainfrom
rluvaton:fix-legacy-agg-nested-key-duplicate-groups

Conversation

@rluvaton

@rluvaton rluvaton commented Sep 2, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

  • No issue filed; found by a fuzz test that runs every aggregate execution chain under a memory limit.

Rationale for this change

GROUP BY on a single nested column (Struct, Map) returns duplicate groups when the legacy GroupedHashAggregateStream spills: the same key comes out as several rows, with the aggregate values split between them. It needs the legacy stream (datafusion.execution.enable_migration_aggregate = false, or a PartialReduce stage on ordered input), a single nested group key, and enough memory pressure to spill in a Final or Single stage. Results are silently wrong rather than an error.

After spilling, the stream re-aggregates the merged spill files with GroupOrderingFull, which requires group ids in first-seen order along the sorted input. The stream recreates its group values collector for that phase to guarantee the order, but only when there is more than one group column, assuming a single column always uses a sequential single-column collector. A single nested column has no specialized single-column collector and is served by GroupValuesColumn through a row-backed column, whose vectorized interning assigns new ids out of input order under hash collisions. In a merged batch of 28 sorted rows the ids came out as 0 to 4, then 9 to 13, then 5 to 8. GroupOrderingFull then treated a group that was still arriving as complete and emitted it, and the next batch reopened it as a new group.

What changes are included in this PR?

GroupedHashAggregateStream now always recreates the group values collector when it switches to merging spill files, instead of only for multi-column keys.

What is the testing strategy for this PR?

New integration test memory_limit::legacy_stream_nested_key_spill_keeps_groups_unique: a 200k-row table grouped by a struct of a list and an integer, with null and empty lists, null numbers and null structs mixed in, aggregated with six aggregates including count(distinct) on the legacy stream under a 4 MB FairSpillPool with a 64-row batch size, compared against the same query with unlimited memory. Without the fix it fails deterministically with 70 rows instead of 67, three keys split into two rows whose counts add up to the reference. With the fix it passes.

Are there any user-facing changes?

No API changes. Queries that hit this path now return correct results.

🤖 Generated with Claude Code

@github-actions github-actions Bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Sep 2, 2026
@rluvaton
rluvaton force-pushed the fix-legacy-agg-nested-key-duplicate-groups branch from 32b789c to 8b49968 Compare September 2, 2026 20:37
@github-actions github-actions Bot added the functions Changes to functions implementation label Sep 2, 2026
@rluvaton
rluvaton marked this pull request as draft September 2, 2026 20:44
@codecov-commenter

codecov-commenter commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.65%. Comparing base (4a93ade) to head (977bdd1).

Files with missing lines Patch % Lines
...hysical-plan/src/aggregates/grouped_hash_stream.rs 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24889      +/-   ##
==========================================
+ Coverage   81.63%   81.65%   +0.02%     
==========================================
  Files        1123     1123              
  Lines      409963   409961       -2     
  Branches   409963   409961       -2     
==========================================
+ Hits       334684   334765      +81     
+ Misses      55602    55505      -97     
- Partials    19677    19691      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rluvaton
rluvaton force-pushed the fix-legacy-agg-nested-key-duplicate-groups branch from 8b49968 to c281838 Compare September 3, 2026 07:24
…a nested group key

When GroupedHashAggregateStream spills and switches to merging the sorted
spill files it relies on GroupOrderingFull, which requires group ids to be
assigned in first-seen order. It recreated its group values collector to
guarantee that only when there was more than one group column, assuming a
single column always uses a sequential single-column collector.

A single nested column (Struct, Map) has no specialized single-column
collector and is handled by GroupValuesColumn through a row-backed column,
whose vectorized interning assigns ids out of input order. The ordering then
emitted groups that were still in progress and the next batch reopened them
as new groups, so the same key came out more than once with its aggregates
split between the rows.

Always recreate the collector for the merge phase.
@rluvaton
rluvaton force-pushed the fix-legacy-agg-nested-key-duplicate-groups branch from c281838 to 977bdd1 Compare September 3, 2026 10:29
@github-actions github-actions Bot removed the functions Changes to functions implementation label Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants