Repository navigation
feat: reuse range partitioning for date_trunc aggregation - #3
Draft
benbellick wants to merge 58 commits into
Draft
benbellick wants to merge 58 commits into
benbellick wants to merge 58 commits into
Conversation
This was referenced Sep 18, 2026
benbellick
force-pushed
the
feat/issue-25344-range-transform-locality
branch
from
September 18, 2026 19:50
a5d4922 to
a60803e
Compare
benbellick
changed the base branch from
fix/issue-25425-date-trunc-scalar-rounding
to
main
September 18, 2026 19:50
benbellick
force-pushed
the
feat/issue-25344-range-transform-locality
branch
from
September 20, 2026 17:38
381dd98 to
0731094
Compare
benbellick
changed the base branch from
main
to
fix-date-trunc-scalar-range
September 20, 2026 17:38
benbellick
force-pushed
the
feat/issue-25344-range-transform-locality
branch
from
October 15, 2026 15:13
6d25405 to
534e02a
Compare
benbellick
force-pushed
the
feat/issue-25344-range-transform-locality
branch
2 times, most recently
from
October 17, 2026 15:38
5c30856 to
50ee850
Compare
Mark intentional expression hash keys with narrowly scoped lint expectations. Interior-mutable memory accounting does not affect expression hashing or equality.
Reject malformed effective split points before proving transformed-key locality. Cover uncombined aggregation, repeated optimization, incompatible join layouts, and grouping-set subtotals with executable regressions.
benbellick
force-pushed
the
feat/issue-25344-range-transform-locality
branch
from
October 17, 2026 16:32
2a0b4f5 to
a6482e5
Compare
…6582) ## Which issue does this PR close? - Related to apache#25088, apache#25148 ## Rationale for this change Normal `cargo` commands reject circular dependencies between packages, but they allow cycles through dev-dependencies. This has bitten us in the past, which is why the `dev/depcheck` tool and its associated CI job were added to detect dev-dependency cycles. Running `dev/depcheck` in CI is relatively expensive, in part because it depends on the `cargo` crate, which has a large dependency tree. Building and running `depcheck` takes ~130 seconds on a 4 vCPU container, and also requires an `ubuntu-latest` runner. Cargo 1.90 added support for multiple packages in a single `cargo publish`, which does check for dev-dependency cycles. It follows that we can use a wrapper script over `cargo publish --workspace --dry-run` to replace `depcheck`. This is more efficient and can run on `ubuntu-slim`. Moreover, it avoids the need to maintain a separate tool and ensures that the CI check is consistent with how packages are actually published. (We should probably also switch to using multi-package `cargo publish` to make releases, but I'll defer that for now.) One downside is that error reporting when a cycle is found is less helpful, but I don't think that is a major concern. ## What changes are included in this PR? * Delete `dev/depcheck` * Change `ci/scripts/check_circular_dependencies.sh` to call `cargo publish --dry-run --workspace` * The "Circular Dependency Check" now runs on `ubuntu-slim` ## What is the testing strategy for this PR? CI / tooling only. Manual local testing: - Small test workspace with a dev-dependency cycle: `cargo test` passes, and `cargo package` / `cargo publish --workspace --dry-run` fail. - Same workspace without the cycle: packaging passes. - Same workspace with a dev-dependency that has a path but no version: packaging passes, because Cargo drops that dependency. - DataFusion main: the dry run passes in 15 seconds and skips the 11 `publish = false` crates. - DataFusion with a dev-dependency cycle added: `cargo check --tests` passes, and the dry run fails. - DataFusion with a normal-dependency cycle added: `cargo check` fails by itself. - DataFusion bumped to unpublished versions: the dry run passes. - Bumped versions plus the dev-dependency cycle: the dry run fails. ## Are there any user-facing changes? No. Co-authored-by: Kumar Ujjawal <ujjawalpathak6@gmail.com>
## Which issue does this PR close? - Closes apache#26347. ## Rationale for this change `array_agg(DISTINCT x)` had no `GroupsAccumulator`. Grouped queries fell back to `GroupsAccumulatorAdapter`, which keeps one `DistinctArrayAggAccumulator` per group, each with its own hash table and encoded rows. With many groups this dominates memory and CPU. ## What changes are included in this PR? New `DistinctArrayAggGroupsAccumulator`: * Each distinct value is encoded once with a `RowConverter` into a `Rows` buffer shared by all groups, so a value that appears in many groups is stored once. * Groups record `(group_index, value_index)` pairs, deduplicated in one hash set. * `evaluate` builds the `ListArray` with a counting sort by group. * It also covers `array_agg(DISTINCT x ORDER BY x)`, sorting each group by the encoded rows. * `EmitTo::First` drops values only referenced by emitted groups. Small refactors: the DISTINCT / ORDER BY check is shared by the row and groups paths (`distinct_sort_options`), and the one-row-per-list `convert_to_state` is shared with `ArrayAggGroupsAccumulator`. `state_compat` now compares list results as multisets for DISTINCT without ORDER BY, since the element order is not defined there. The row accumulator returns hash table order. ## Are these changes tested? * **Unit tests** cover dedup across batches, nulls with and without IGNORE NULLS, FILTER, ORDER BY DESC, `EmitTo::First`, state/merge, `convert_to_state`, dictionary types, and a differential check against `DistinctArrayAggAccumulator`. * **`state_compat`** passes with the new accumulator included. * **`array_agg.slt`** gains grouped cases mixing DISTINCT with `count(*)`, FILTER, ORDER BY and IGNORE NULLS. The existing aggregate slt files pass. ## Benchmarks Measured locally with a criterion benchmark (not included here, happy to add it): one 8192-row Utf8 batch, `update_batch` plus `evaluate`, the new accumulator against `GroupsAccumulatorAdapter` over `DistinctArrayAggAccumulator`. Apple M-series: | groups | input | groups accumulator | adapter (before) | speedup | | -- | -- | -- | -- | -- | | 100 | ~25 distinct | 154 µs | 251 µs | 1.6x | | 100 | ~7800 distinct | 384 µs | 969 µs | 2.5x | | 1000 | ~25 distinct | 173 µs | 1.42 ms | 8.2x | | 1000 | ~7800 distinct | 382 µs | 2.26 ms | 5.9x | | 8192 | ~25 distinct | 279 µs | 10.2 ms | 36x | | 8192 | ~7800 distinct | 414 µs | 10.4 ms | 25x | Live heap after `update_batch`, measured with a counting global allocator: 65 536 Utf8 rows in 8 batches. `size()` tracks these within ~10% for both. | groups | input | groups accumulator | adapter (before) | | -- | -- | -- | -- | | 100 | ~25 distinct | 11 KB | 720 KB | | 1000 | ~7800 distinct | 2.2 MB | 9.9 MB | | 8192 | ~25 distinct | 1.7 MB | 18.3 MB | | 65536 | ~7800 distinct | 2.2 MB | 100.7 MB | Memory now grows with the number of distinct values instead of the number of groups. ## Are there any user-facing changes? No API changes. `array_agg(DISTINCT x)` without ORDER BY may return elements in a different (still unspecified) order. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Which issue does this PR close? - Addresses an issue introduced in apache#25820 - Related to apache#25088, apache#25148 ## Rationale for this change apache#25820 added a test, `sort_after_group_by_reserves_shared_output_once`, which takes ~17 minutes when run under `force_hash_collisions`. This increased the runtime of the extended "hash collisions" job from 3-5 minutes to ~20 minutes, which slows down the merge queue and consumes CI resources. This PR disables the test under `force_collisions` mode. ## What changes are included in this PR? See above. ## What is the testing strategy for this PR? Test config change only. ## Are there any user-facing changes? No.
…#26013) ## Which issue does this PR close? - Closes apache#25897. ## Rationale for this change Since apache#25838, a correlated filter below a window function keeps the subquery correlated unless the filter reads only `PARTITION BY` columns. Such queries fail with "Physical plan does not support logical expression ...". For example (apache#25792): ```sql SELECT t1.id, sub.id, sub.rn FROM t1, LATERAL ( SELECT t2.id, row_number() OVER (ORDER BY t2.id) AS rn FROM (SELECT * FROM t2 WHERE t2.t1_id = t1.id) AS t2 ) AS sub; ``` ## What changes are included in this PR? When every correlated conjunct has the form `col = <outer expr>`, the filter keeps the rows of one `col` value for each outer row. `PullUpCorrelatedExpr` then adds `col` to the `PARTITION BY` of every window function and pulls the filter above the window, as Spark (`DecorrelateInnerQuery`) and DuckDB (`FlattenDependentJoins`) do. Each extended window expression is aliased to its old name, so the output schema does not change. The subquery stays correlated when: - the outer expression is volatile. - `col` has a float leaf. `-0.0 = 0.0` holds, but a partition keeps the two apart. - a conjunct casts `col` with a lossy cast. A lossy cast lets several `col` values match one outer row, which is apache#26014. Widening casts that keep distinct values distinct are accepted through `is_injective_cast`, which was a private Map key helper in `datafusion-common`. - an aggregate without `GROUP BY` sits below the window. It returns a row for an outer row without a match, which the pulled up filter would drop. A Projection over a count bug aggregate now records the expressions that read no column, such as `1 AS x`, in `collected_count_expr_map`. Before, such a Projection dropped the entry. A scalar subquery then returned NULL instead of `1` for an outer row without a match, and the window guard above did not see the aggregate. The core of the change is @adriangb's commit 1a2ad03, which apache#25838 reverted to stay a bug fix. ## What is the testing strategy for this PR? - `subquery.slt` and `lateral_join.slt`: the window cases from apache#25838 that expected an error now check results, plus an `EXPLAIN` that shows `PARTITION BY [b.y]`. - New cases that stay correlated: a lossy `CAST`, a float column with signed zeros, a volatile outer expression, and a window over an aggregate without `GROUP BY` behind a Projection, `DISTINCT` or a Filter, or in an `EXISTS`, `NOT EXISTS`, `IN` or `LATERAL` subquery. - New cases with results: a widening cast, a lower window that already outputs the extended name, a constant projected over a count bug aggregate, and a window over a grouped aggregate. ## Are there any user-facing changes? Queries with a correlated equality below a window function now run instead of failing with a not-implemented error. A scalar subquery that projects a constant over a correlated aggregate now returns the constant for outer rows without a match, instead of NULL. `datafusion_common::nested_struct::is_injective_cast` is now public. --------- Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
…lists (apache#25187) ## Which issue does this PR close? - Part of apache#19241. ## Rationale for this change DataFusion currently rewrites short `IN` lists like `x IN (1, 2)` as `x = 1 OR x = 2`. Evaluating `IN` directly can be faster. This PR keeps the `IN` form for the types below, so short lists can use DataFusion's existing optimized `IN` filters. ## What changes are included in this PR? - Keep two- and three-value `IN` and `NOT IN` lists for `Int32`, `Int64`, `UInt64`, `Float32`, `Float64`, and `FixedSizeBinary` columns with widths of 1, 2, 4, 8, or 16 bytes. The list must contain constant values of the column's type. - Add benchmarks that use normal optimization to compare performance before and after this change. - Add optimizer and SQL logic tests covering list sizes, types, NULLs, and non-constant list values. ## What is the testing strategy for this PR? All 935 optimizer unit tests and all 533 SQL logic files passed, including TPC-H. Formatting, Clippy, and lint checks passed. All 336 benchmark cases passed on both builds. The benchmark bot's ARM [two-value](apache#25187 (comment)) and [three-value](apache#25187 (comment)) results show: - Integer lists take about 32–55% less time when half the rows match or none do. In the extreme case where every row matches the first value, they can take up to about twice as long. - Floating-point and fixed-size binary lists are faster in every tested case, including NULLs and small batches. <details> <summary>Benchmark setup</summary> Both builds use the same 336 cases. The [baseline](apache@31a333d) contains `main` at `06aa131cda0` plus the same benchmark code as the [PR](apache@8aea24b). Queries use normal optimization. Timings cover running the chosen expressions; query planning and table setup happen before timing. </details> ## Are there any user-facing changes? `EXPLAIN` shows `IN` or `NOT IN` for these short lists in place of the expanded `OR` or `AND` comparisons. This performance PR aligns constant two- and three-value `Float32`/`Float64` lists with existing behavior for 4+ values, including the signed-zero bug (`+0.0` versus `-0.0`). That bug [predates this optimization series](apache#7378) and already affects longer lists with constant or column values; fixing it is left to apache#25186. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
… left (apache#26476) I've got a few fixes to upstream so just starting with small ones. ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#26475 ## Rationale for this change If row-group statistics or Bloom filters prune every row group in a file range, the opener still builds the decoder read plans, the decoder and the row filter, then hands back a stream morsel that yields nothing. `PruneFile` already ends early when file statistics prune the range; this does the same once row-group pruning has. ## What changes are included in this PR? `PruneWithStatistics` and `PruneWithBloomFilters` return `ParquetOpenState::Done` when no row group survives. ## What is the testing strategy for this PR? Testing to ensure that we do actually skip them! ## Are there any user-facing changes? Nope! --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#23436. ## Rationale for this change - Hive grouped scans declare Partitioning::Hash, but no hash ever ran. - The optimizer treats any two Hash partitionings with equal counts as co-partitioned, so it skips the repartition. - Partitioned hash joins then drop rows. Fact {A, B, C} joined to dimension {A, C, D} returns only A, because fact group 1 holds B while dimension group 1 holds C. - Joining a Hive scan to a VALUES list also loses rows, since the planner hash repartitions the other side. ## What changes are included in this PR? - Add FileGroup::group_by_partition_values_with_split_points, returning groups plus the SplitPoints between them. - Merge values into contiguous chunks rather than round robin. Only contiguous chunks can be described by split points. Group counts are unchanged. - Give ListFilesResult a partition_split_points field. - In ListingTable::scan build a logical Range partitioning from the partition columns and the split points and convert it with create_physical_partitioning, the same path a user declared output partitioning takes. No new public helper. - Declare nothing when split_file_groups_by_statistics re-cuts the groups. Comparing counts misses this, since the re-cut can return the same number. - Drop the dynamic filter guard in HashJoinExec since it existed only because Hive groups reported Hash. ## What is the testing strategy for this PR? - TEST 14 and 15 in preserve_file_partitioning.slt cover the two failures above. - TEST 16 self joins the merged high cardinality table. TEST 17 asserts the statistics re-cut declares nothing. - Reverting only table.rs makes TEST 14 return one row instead of two and breaks TEST 12. - Six unit tests route every file through its split points and assert it lands in its own group, covering NULLs, compound keys, a NULL inside a split point, and unordered split points. - Two datafusion-proto round trips cover Dictionary(UInt16, Utf8) and Utf8View split points. ## Are there any user-facing changes? - EXPLAIN prints Range([col ASC], [(v1), (v2)], N) instead of Hash([col], N). - Joins between Hive tables with different values gain a RepartitionExec(Range) rather than returning wrong results. - Partitioned dynamic filter pushdown works again under preserve_file_partitions. - File group membership changes when distinct values exceed target_partitions. Byte balance can shift on date partitions with larger recent days. Set preserve_file_partitions = 0 to restore the old behaviour. - API break: ListFilesResult gains a public field. Add partition_split_points: vec![] or use ... - Providers declaring Hash over value grouped files should declare Range from the split points instead. They already return missing rows today when the other join input is hash repartitioned. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…apache#26085) ## Which issue does this PR close? - Part of apache#25459. This is a follow-up to apache#25455 (apache#14540). ## Rationale for this change `PushDownLeafProjections` and `OptimizeProjections` undo each other in every optimizer pass for a common plan shape: a projection of a struct field (or another `MoveTowardsLeafNodes` expression) directly over a node that the extraction cannot go through, such as a `TableScan` or an `Aggregate`. ```text Projection: leaf_udf(test.a) TableScan: test ``` In each pass, `push_down_leaf_projections` splits this into a recovery projection over an in-place extraction projection: ```text Projection: __datafusion_extracted_1 AS leaf_udf(test.a) Projection: leaf_udf(test.a) AS __datafusion_extracted_1, test.a TableScan: test ``` Then `optimize_projections` merges the two back into the original plan. The loop stops only because the plan at the end of a pass is the same as at the end of the previous pass. The final plan is correct, but each pass does the work of both rules again. The test that apache#25455 added found this shape. That test records which rules change the plan in each pass. On `main` it prints, for the plan above: ```text pass 0: push_down_leaf_projections, optimize_projections pass 1: push_down_leaf_projections, optimize_projections ``` With this PR: ```text pass 0: optimize_projections ``` ### Which rule yields `PushDownLeafProjections` yields. The split has no use when the extraction projection cannot move: a plain `Projection(get_field(...))` directly over a Parquet scan already reaches `DataSourceExec projection=[..., get_field(s@1, value) ...]`. The existing `EXPLAIN` checks in `projection_pushdown.slt` show this and do not change. If `OptimizeProjections` yielded instead, the plan would keep a projection node that does no work. ## What changes are included in this PR? In `split_and_push_projection` (`extract_leaf_expressions.rs`), the rule still builds the in-place extraction projection, because a mixed projection over a `Filter` must become a pure extraction projection before it can go through the filter. The rule now immediately tries to push that extraction projection into its input. If it cannot move, the rule returns the original projection unchanged. The early return for a projection without new extractions stays: it also ends the recursion of that push attempt. - The decision is recorded in the module documentation of `extract_leaf_expressions.rs` and `optimize_projections`, and as a second entry in the "Rule Precedence" section of `docs/source/library-user-guide/query-optimizer.md`. ## What is the testing strategy for this PR? - `filter_and_extraction_projection_reach_fixed_point` in `push_down_filter.rs` now checks that no rule at all changes the plan in the last optimizer pass (before, it checked only `PushDownFilter` and `PushDownLeafProjections`). It also covers a projection over a scan with no filter. Without the fix it fails with `[["push_down_leaf_projections","optimize_projections"],["push_down_leaf_projections","optimize_projections"]]`. - Six stage snapshots in `extract_leaf_expressions.rs` change. In each one, only the "After Pushdown" stage changes, to "(same as after extraction)". The "Optimized" plan is the same in all six. - One line in `unnest.slt` changes (logical plan line 03). A nested alias is now kept: `get_field(__unnest_placeholder(...,depth=1) AS UNNEST(recursive_unnest_table.column3), Utf8("c1"))`. The split and merge used to remove that alias as a side effect. The physical plan is the same. This is the same display-only nested alias that apache#25455 showed for `length(...)`. Commands run: - `cargo test --profile ci -p datafusion-optimizer --lib`: 932 passed. - `cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests`: 526 of 526 files pass. - `cargo fmt --all` and `cargo clippy --profile ci -p datafusion-optimizer --all-targets -- -D warnings`: clean. ## Are there any user-facing changes? No. Final plans and results do not change, except for the alias display in the `unnest` plan above. Planning does less work for queries that read struct fields. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes apache#26448.
Related to apache#23569, apache#25344, and apache#25345.
Rationale for this change
This PR implements a focused subset of the optimization in Nga's PR #24501 for a known-safe case. The plan is to iteratively expand support to more safe cases, following Gene's review feedback.
Suppose
eventsis range-partitioned ontimestamp, with partitions split at hourly boundaries.Each
(key, hour)group belongs entirely to one input partition. The simplified physical plans below omit projections and usehourfor the grouping expression.Without the optimization
With the optimization
The new locality check is specific to aggregates. General distribution satisfaction and join-layout checks are unchanged.
What changes are included in this PR?
Add a default-deny capability that identifies functions and argument types safe for range-key analysis. Check expression ordering and arguments, then evaluate both sides of each split to prove groups stay within partitions.
The optimizer calls it through this path.
flowchart TD A["EnsureRequirements checks aggregate input"] --> B["AggregateExec::input_keeps_groups_local"] B --> C["Partitioning::keeps_keys_local"] C --> D{"Function admitted, expression supported, and split boundaries safe?"} D -->|Yes| E["Can skip the aggregate hash shuffle"] D -->|No or unknown| F["Retain normal repartitioning"]Support is intentionally limited to timezone-less
date_truncover single-key timestamp ranges, with literal non-range arguments.date_binremains excluded pending its safety audit, including null behavior. Compound range keys remain follow-up work in apache#24644.Keep the shuffle when buckets cross partitions or the checks fail. Grouping sets still use the existing distribution checks.
What is the testing strategy for this PR?
Added multiple unit and integration tests.
I'm happy to add a benchmark if maintainers would find it useful.
Are there any user-facing changes?
Eligible
date_truncaggregations can skip a hash shuffle. Results are unchanged.Note: This PR was developed with AI assistance. All changes have been reviewed, and I take full responsibility for this contribution.