Skip to content

Run SortAggregateExec as a native hash aggregate without its input sort #7146

Open

Description

@comphead

What is the problem the feature request solves?

Spark plans SortAggregateExec when an aggregate buffer is not UnsafeRow-mutable (for example string min/max), and EnsureRequirements puts a SortExec on the grouping keys under it. Comet keeps that sort, so DataFusion runs its ordered aggregate streams on sorted input. The sort exists only because of Spark's hash map limitation, and in Comet it is the expensive part:

On a job grouping by 87 columns with string min/max, 39 of 40 map tasks failed with Additional allocation failed for OrderedPartialAggregateStream[0], and retries failed in ExternalSorterMerge[0].

Describe the potential solution

When a SortAggregateExec sits on the SortExec for its grouping keys, run the native aggregate over the sort's child:

  1. Partial aggregate under a shuffle exchange: drop the sort and report no output ordering.
  2. Final aggregate: drop the sort, and keep ordered_by_grouping_keys so the native planner sorts the aggregated output, unless the parent is an exchange.
  3. Leave first, last and collect_list on the current path, and gate the change behind a config.

Additional context

Apache Auron does this in AuronConverters.convertSortAggregateExec, unless a parent needs the order. #6936 (hash-mode string min/max under-reports memory) should land first or together. Related: #1994, #4565, #6880.

Activity

  1. comphead commented on Oct 21, 2026

    @comphead
    ContributorAuthor

    Potential workaround

    spark.comet.exec.respectDataFusionConfigs=true
    spark.comet.datafusion.execution.enable_migration_aggregate=false
    
  2. comphead commented on Oct 28, 2026

    @comphead
    ContributorAuthor

    depends on DF56

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions