Skip to content

fix: SortPreservingMergeExec drops input orderings when merging partitions - #26654

Merged
jayzhan211 merged 1 commit into
apache:mainfrom
HarshilMital:fix-spm-orderings
Oct 23, 2026
Merged

jayzhan211 merged 1 commit into
apache:mainfrom
HarshilMital:fix-spm-orderings

Conversation

@HarshilMital

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

When a sorted input with more than one partition goes through SortPreservingMergeExec, the planner treats every ordering of the input as still holding after the merge. Only the merge key ordering actually holds. The other orderings are true within each partition, but not across them.

Because of this, the planner can remove a sort that is still needed, and the query returns rows out of order. With two files that are each sorted by a and b and target_partitions = 2:

SELECT * FROM (SELECT a, b FROM t ORDER BY a LIMIT 10) ORDER BY b;

the plan has no sort for the outer ORDER BY b, and the rows come back as (1, 10), (1, 1), (2, 20), (3, 2).

What changes are included in this PR?

  • SortPreservingMergeExec::compute_properties clears the input orderings when there is more than one input partition, before adding the merge key ordering. This matches what CoalescePartitionsExec and RepartitionExec already do. With a single input partition nothing is merged, so the input orderings are kept as before.

The preserve-order RepartitionExec case mentioned in the issue is not covered here.

Draft #26643 moves this logic into EquivalenceProperties::merged. If that lands first, I'll update this PR to use .merged(input_partitions, false) followed by add_ordering.

What is the testing strategy for this PR?

  • New sqllogictest case at the end of order.slt, using the reproduction from the issue. It checks both the query result and the physical plan, which must keep the SortExec above the SortPreservingMergeExec. It fails on main with the wrong row order from the issue, and passes with this change.
  • The existing sqllogictests and the datafusion-physical-plan / datafusion-physical-optimizer unit tests pass with no changes to expected plans.

Are there any user-facing changes?

Queries like the one above now return correctly ordered results. No public API changes.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Oct 23, 2026
…tions

Merging multiple partitions only preserves the merge key ordering. Other
input orderings hold within each partition only, so keeping them let the
planner remove a sort that was still needed and return wrong results.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @HarshilMital 👍🏻

@jayzhan211
jayzhan211 enabled auto-merge October 23, 2026 11:03
@jayzhan211
jayzhan211 added this pull request to the merge queue Oct 23, 2026
Merged via the queue into apache:main with commit ebf4858 Oct 23, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt) v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wrong results: SortPreservingMergeExec keeps orderings of its input that do not hold after the merge

2 participants