Repository navigation
fix: SortPreservingMergeExec drops input orderings when merging partitions - #26654
Merged
Merged
Conversation
…tions Merging multiple partitions only preserves the merge key ordering. Other input orderings hold within each partition only, so keeping them let the planner remove a sort that was still needed and return wrong results.
HarshilMital
force-pushed
the
fix-spm-orderings
branch
from
October 23, 2026 09:18
88265fe to
5ba0ed9
Compare
jayzhan211
approved these changes
Oct 23, 2026
jayzhan211
left a comment
Contributor
There was a problem hiding this comment.
Thanks @HarshilMital 👍🏻
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
SortPreservingMergeExeckeeps orderings of its input that do not hold after the merge #26646.Rationale for this change
When a sorted input with more than one partition goes through
SortPreservingMergeExec, the planner treats every ordering of the input as still holding after the merge. Only the merge key ordering actually holds. The other orderings are true within each partition, but not across them.Because of this, the planner can remove a sort that is still needed, and the query returns rows out of order. With two files that are each sorted by
aandbandtarget_partitions = 2:the plan has no sort for the outer
ORDER BY b, and the rows come back as(1, 10), (1, 1), (2, 20), (3, 2).What changes are included in this PR?
SortPreservingMergeExec::compute_propertiesclears the input orderings when there is more than one input partition, before adding the merge key ordering. This matches whatCoalescePartitionsExecandRepartitionExecalready do. With a single input partition nothing is merged, so the input orderings are kept as before.The preserve-order
RepartitionExeccase mentioned in the issue is not covered here.Draft #26643 moves this logic into
EquivalenceProperties::merged. If that lands first, I'll update this PR to use.merged(input_partitions, false)followed byadd_ordering.What is the testing strategy for this PR?
order.slt, using the reproduction from the issue. It checks both the query result and the physical plan, which must keep theSortExecabove theSortPreservingMergeExec. It fails onmainwith the wrong row order from the issue, and passes with this change.datafusion-physical-plan/datafusion-physical-optimizerunit tests pass with no changes to expected plans.Are there any user-facing changes?
Queries like the one above now return correctly ordered results. No public API changes.