diff --git a/content/blog/2026-08-25-datafusion-55.0.0.md b/content/blog/2026-08-25-datafusion-55.0.0.md new file mode 100644 index 00000000..123fdb00 --- /dev/null +++ b/content/blog/2026-08-25-datafusion-55.0.0.md @@ -0,0 +1,721 @@ +--- +layout: post +title: Apache DataFusion 55.0.0 Released +date: 2026-08-25 +author: pmc +categories: [release] +--- + + + +[TOC] + +We are proud to announce the release of [DataFusion 55.0.0]. This post +highlights some of the many improvements since [DataFusion 54.0.0], such as +significant performance increases, range partitioning, `MERGE INTO` support, and +runtime row-group pruning. The complete list of changes is available in the +[changelog]. This release represents roughly 10 weeks of development and 877 +commits. Thanks to the [175 contributors] (a new record!) for making it +possible. + +[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0 +[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/ +[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md +[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits + + + +**Figure 1**: Development activity over the last three DataFusion releases: +total commits, commits per day, and unique contributors, computed from each +release's [changelog] and release dates. + +## Performance Improvements 🚀 + +In this release, we focused our optimizations on making DataFusion faster across +the board rather than further optimizing our very satisfying ClickBench numbers +(DataFusion is already the fastest in some cases — see the +[appendix]), as ClickBench represents only a tiny fraction of what our actual users +do (e.g. its files have no page index and contain only integer and string columns). + +Here is a representative sample of the performance improvements in this release; +see the [full list in the appendix][perf appendix]. + +| Improvement | Representative Result | Area | +|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------| +| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK | +| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions | +| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO | +| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO | +| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting | +| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation | + + +### Sort Pushdown + TopK Pruning + +The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and +`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter +threshold tightens, the Parquet reader re-evaluates the threshold against the +remaining row groups and drops those that can no longer contribute ([#22450]), +and compound `ORDER BY` queries are now supported. Together these reduce the +total `topk_tpch` suite runtime by ~43%; see our +[Optimizing for Almost Sorted Data] blog post for more details. Thanks to +[@zhuqi-lucas] for driving this work, with reviews from [@adriangb]. + +[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036 +[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/ + +### Aggregation Improvements + +**Complete Multi-Column `GROUP BY` Type Coverage**: +DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has +type-specific fast paths, but previously any unsupported column type forced +the entire grouping onto a slower row-encoded fallback. For example, this +query to deduplicate a table of UUIDs used to hit the slow path: + +```sql +SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id); +``` + +DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5% +faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to +[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work. + +### Faster Functions + +DataFusion ships hundreds of built-in functions, so speeding them up improves performance +for many workloads. This release optimizes dozens of functions — up to 24x faster +for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct] +with low-cardinality inputs and many groups ([#22768]). It also includes +dictionary-encoding preservation for many string functions ([#23743], +[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the +[full list in the appendix][perf appendix]. +Thanks to the many contributors who drove this work, especially +[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and +[@haohuaijin]. + +[perf appendix]: #appendix-full-list-of-performance-improvements + +### Planner Improvements + +**Unified Distribution and Sorting Enforcement**: +The [`EnforceDistribution`][EnforceDistribution] and +[`EnforceSorting`][EnforceSorting] physical optimizer passes are +now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort +pushdown ([#21976]), fixing longstanding ordering issues between the two passes +and enabling the sort pushdown work described above. +Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and +[@alamb]. + +[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html +[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html +[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html + +**Smarter Join Planning**: +DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]), +eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles +intermediate projections in outer join elimination ([#22534]), and reorders +predicates in conjunctions using a cost heuristic ([#22343]). +Thanks to [@neilconway] and [@simonvandel] for driving this work. + +**Better Scalar UDF Metadata APIs**: +Scalar UDFs can now declare that they are *strict* (they return `NULL` when any +input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for +queries that filter on a function result, and *strictly order preserving* +(sorted input yields identically sorted output) ([#23807]), letting the +optimizer eliminate redundant sorts on expressions such as custom casts. +Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from +[@alamb], [@kosiew], and [@getChan]. + +**Faster Optimizer**: +The optimizer continues to get faster, with improvements such as selective +subquery traversal and in-place rewrites ([#22298]), collapsing chained +projections ([#22389]), avoiding re-inlining expensive common subexpressions +([#23459]), and a faster `PushDownFilter` rule that modifies plans in place +rather than copying them ([#20002], [#21668]). +Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work. + +### Scan Improvements + +**Pruning Unread Parquet Leaves for Nested Columns**: + +Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and +Iceberg integrations — often hand DataFusion a table schema that includes only the nested +subfields the query needs. For example, given a file whose `events` column +physically holds four subfields, a table might declare only two of them: + +```sql +-- events column is ARRAY> +-- Table definition only refers to the first two subfields, id and name +CREATE EXTERNAL TABLE events ( + events ARRAY> +) +STORED AS PARQUET LOCATION 'events.parquet'; +``` + +DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all +four leaves were read from the file and decoded, including the large `payload` +and `trace` subfields, which were then thrown away. The Comet project reported +a production query where this extra decoding caused 1.35 TB of reads, whereas +plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes +that gap by not reading the undeclared `payload` and `trace` leaves from the +file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from +[@adriangb]. + +[DataFusion Comet]: https://datafusion.apache.org/comet/ +[delta-rs]: https://github.com/delta-io/delta-rs + +**Other Scan Improvements**: +DataFusion 55 also skips loading the page index (and an expensive +[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports +file-level Parquet row selections ([#22940]), and lowers the default +`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on +small files ([#22439]). +Thanks to [@alamb], [@haohuaijin], and [@adriangb]. + +## Stability Improvements 🛡️ + +The community also improved DataFusion's handling of larger-than-memory +aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a +refactoring of the aggregation path into dedicated streams (epic [#22710]). +Sorts under memory pressure are more resilient: when a spill +merge cannot reserve enough memory, DataFusion now re-spills the largest stream +in smaller batches rather than failing ([#22945]), and caps the merge fan-in to +bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt], +[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup +bug in the spill pool, [#23522]) for this work. + +## New Features ✨ + +### `file_row_index()` and `input_file_name()` + +DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions +to expose Parquet virtual columns: + +```sql +> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet'; ++---------+-------------------+------------------+ +| column1 | input_file_name() | file_row_index() | ++---------+-------------------+------------------+ +| 100 | tmp/foo.parquet | 0 | +| 200 | tmp/foo.parquet | 1 | ++---------+-------------------+------------------+ +``` + +Such functions are useful for change data capture, debugging, and +Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work +(reviving earlier work from [@jkylling]), with reviews from [@adriangb], +[@comphead], and [@niebayes]. + +### Range Partitioning + +DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash +values). Query inputs are often range partitioned in real-world scenarios, such as time-series data +written as one file per day or hour. DataFusion uses range partitioning information to +avoid expensive repartitioning operations and push more specific dynamic filters +to scans. + +Data that is range partitioned declares an ordering and a list of split +points. Partition `i` holds the keys that fall between split point `i-1` and +split point `i`: + +```text +ordering = [date ASC NULLS LAST] +split_points = [(2022-01-01), (2023-01-01)] + +partition 0: date < 2022-01-01 +partition 1: 2022-01-01 <= date < 2023-01-01 +partition 2: date >= 2023-01-01 +``` + +For more details, please see the documentation for +[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and +the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood], +[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry], +[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community +effort. + +### `MERGE INTO` Planner Support + +`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and +conditional update workloads, and a key building block for table formats such +as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types +([#20763]) along with SQL planner and physical planner support, and a new +[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can +execute merge operations: + +```sql +MERGE INTO target t +USING source s +ON t.id = s.id +WHEN MATCHED AND s.deleted THEN DELETE +WHEN MATCHED THEN UPDATE SET name = s.name +WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name); +``` + +Built-in table providers do not yet implement the hook, but custom +[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug +in their own merge execution. More `MERGE INTO` improvements are planned for +upcoming releases; see [#20746] for details. +Thanks to [@wirybeaver] for implementing this feature, with reviews from +[@alamb] and [@kosiew]. + + +### Pluggable Spill Backends + +DataFusion spills to disk when a query exceeds its memory budget, but the spill +infrastructure was previously hardwired to OS-level temporary files. DataFusion +55 introduces a pluggable [`SpillFile`][SpillFile] trait and +[`TempFileFactory`][TempFileFactory] ([#21882], +[#22230]) so hosts can route spill data through their own storage layers — for +example, extensions like ParadeDB can now integrate spilling into the +Postgres buffer pool. Implement [`TempFileFactory`][TempFileFactory] and install +it on the [`RuntimeEnv`][RuntimeEnv]: + +```rust +let runtime = RuntimeEnvBuilder::new() + .with_disk_manager_builder( + // register a custom TempFileFactory + DiskManagerBuilder::default() + .with_temp_file_factory(Arc::new(MyTempFileFactory::new())), + ) + .build_arc()?; +let ctx = SessionContext::new_with_config_rt(SessionConfig::new(), runtime); +``` + +See the [`object_store_spill.rs`][object_store_spill.rs] example for a complete implementation that +spills to an [`ObjectStore`][ObjectStore] such as S3. Thanks to [@pantShrey] for this work, +with reviews from [@alamb]. + +### Extensibility for Distributed Engines + +Several new APIs make it easier to build distributed systems such as +[datafusion-distributed], [DataFusion Ballista], and [DataFusion Python] on +top of DataFusion: + +- **Dynamic filter propagation across network boundaries**: new + [`ExecutionPlan::apply_expressions`][ExecutionPlan::apply_expressions] and + [`ExecutionPlan::dynamic_expressions_produced`][ExecutionPlan::dynamic_expressions_produced] methods let engines discover + which plan nodes produce dynamic filters and re-wire them across stage + boundaries ([#24018], [#24068]). Thanks to [@jayshrivastava]. +- **`FFI_QueryPlanner`**: foreign libraries can now provide a custom query + planner over the FFI boundary — for example, connecting a distributed + planner to a [`SessionContext`][SessionContext] in Python ([#24028]). Thanks to [@timsaucer]. +- **Self-serializing execution plans**: built-in [`ExecutionPlan`][ExecutionPlan]s were ported + to per-type [`try_to_proto`][try_to_proto] / [`try_from_proto`][try_from_proto] hooks ([#23494]), putting built-in and + third-party plans on the same code path. Thanks to [@adriangb]. +- **Window accumulator state access**: [`BoundedWindowAggExec`][BoundedWindowAggExec] can now expose + finalized accumulator state to an observer callback, enabling incremental / + prefix-scan use cases ([#24035]; see [how it is used in Ballista]). Thanks to + [@avantgardnerio], with reviews from [@alamb] and [@timsaucer]. + +[datafusion-distributed]: https://github.com/datafusion-contrib/datafusion-distributed +[DataFusion Ballista]: https://datafusion.apache.org/ballista/ +[DataFusion Python]: https://datafusion.apache.org/python/ +[ExecutionPlan::apply_expressions]: https://docs.rs/datafusion/55.0.0/datafusion/physical_plan/trait.ExecutionPlan.html#tymethod.apply_expressions +[ExecutionPlan::dynamic_expressions_produced]: https://docs.rs/datafusion/55.0.0/datafusion/physical_plan/trait.ExecutionPlan.html#method.dynamic_expressions_produced +[SessionContext]: https://docs.rs/datafusion/55.0.0/datafusion/execution/context/struct.SessionContext.html +[ExecutionPlan]: https://docs.rs/datafusion/55.0.0/datafusion/physical_plan/trait.ExecutionPlan.html +[try_to_proto]: https://docs.rs/datafusion-physical-plan/55.0.0/datafusion_physical_plan/filter/struct.FilterExec.html#method.try_to_proto +[try_from_proto]: https://docs.rs/datafusion-physical-plan/55.0.0/datafusion_physical_plan/filter/struct.FilterExec.html#method.try_from_proto +[BoundedWindowAggExec]: https://docs.rs/datafusion/55.0.0/datafusion/physical_plan/windows/struct.BoundedWindowAggExec.html +[how it is used in Ballista]: https://github.com/apache/datafusion-ballista/pull/2211 + +### `EXPLAIN` Improvements + +DataFusion 55 adds a Postgres-style `EXPLAIN (...)` option list ([#21768]) and +a `pgjson` output format for [`EXPLAIN ANALYZE`][EXPLAIN ANALYZE] ([#21767]), making plan output +easier to consume with existing Postgres tooling such as plan visualizers. + +[EXPLAIN ANALYZE]: https://datafusion.apache.org/user-guide/explain-usage.html#execution-counters-explain-analyze + +`EXPLAIN ANALYZE` can produce many metrics, and you can narrow them down +explicitly with the `METRICS` option. For example, to see only row counts for +each plan node, use this: + +```sql +> EXPLAIN (ANALYZE, METRICS 'rows', LEVEL summary) + SELECT region_id, sum(amount) FROM orders GROUP BY region_id; + +AggregateExec: mode=Single, gby=[region_id@0 as region_id], aggr=[sum(orders.amount)], metrics=[output_rows=5] + DataSourceExec: file_groups={1 group: [[orders.parquet]]}, projection=[region_id, amount], file_type=parquet, metrics=[output_rows=100.0 K, row_groups_pruned_statistics=1 total → 1 matched, scan_efficiency_ratio=7.83% (28.30 K/361.5 K), ...] +``` + +Adding `FORMAT pgjson` renders the same physical plan and its live metrics as +Postgres-compatible JSON, which can be pasted straight into plan visualizers +such as [Dalibo]: + +```sql +> EXPLAIN (ANALYZE, FORMAT pgjson, METRICS 'rows', LEVEL summary) + SELECT region_id, sum(amount) FROM orders GROUP BY region_id; +``` + +```json +[ + { + "Plan": { + "Node Type": "AggregateExec", + "Actual Rows": 5, + "Plans": [ + { "Node Type": "DataSourceExec", "Actual Rows": 100000, ... } + ] + } + } +] +``` + +See the [EXPLAIN usage guide] for the full option list. Thanks to [@adriangb] +for this work. + +[Dalibo]: https://explain.dalibo.com/ +[EXPLAIN usage guide]: https://datafusion.apache.org/user-guide/explain-usage.html + +### New Functions + +**SQL and Scalar Functions**: +DataFusion 55 adds new array math functions [`array_scale`][array_scale], [`array_add`][array_add], +[`array_subtract`][array_subtract], [`array_sum`][array_sum], and [`array_avg`][array_avg], plus the higher-order +[`array_first`][array_first] function. It also includes the new +[`unnest_outer`][unnest_outer] function ([#22100]), which preserves +empty inputs as `NULL`s rather than dropping them, and +[`approx_distinct`][approx_distinct] now supports more types. +Thanks to [@crm26], [@SubhamSinghal], [@EdsonPetry], [@athlcode], and +[@mkleen] for these contributions. + +**Spark-Compatible Functions**: +The [datafusion-spark crate] gains new or improved Spark-compatible functions, +including [`hypot`][hypot], [`atan2`][atan2], [`weekday`][weekday], [`monthname`][monthname], and [`concat_ws`][concat_ws] with array +support, plus a new Spark SQL parser dialect config ([#22529]). +Thanks to the contributors who drove this work, especially +[@KarpagamKarthikeyan], [@sjhddh], [@JeelRajodiya], [@davidlghellin], and +[@kumarUjjawal]. + +## Upgrade Guide and Changelog 📖 + +Upgrading to 55.0.0 should be straightforward for most users, though there are +some breaking changes. See the [Upgrade Guide] for details and +migration snippets, and the [changelog] for the full list of changes. + +## About DataFusion + +[Apache DataFusion] is an extensible query engine, written in [Rust], that uses +[Apache Arrow] as its in-memory format. DataFusion is used by developers to +create new, fast, data-centric systems such as databases, dataframe libraries, +and machine learning and streaming applications. While [DataFusion's primary +design goal] is to accelerate the creation of other data-centric systems, it +provides a reasonable experience directly out of the box as a [dataframe +library], [Python library], and [command-line SQL tool]. + +DataFusion's core thesis is that, as a community, together we can build much +more advanced technology than any of us as individuals or companies could build +alone. Without DataFusion, highly performant vectorized query engines would +remain the domain of a few large companies and world-class research +institutions. With DataFusion, we can all build on top of a shared foundation +and focus on what makes our projects unique. + +## How to Get Involved + +DataFusion is not a project built or driven by a single person, company, or +foundation. Rather, our community of users and contributors works together to +build a shared technology that none of us could have built alone. + +If you are interested in joining us, we would love to have you. You can try out +DataFusion on some of your own data and projects and let us know how it goes, +contribute suggestions, documentation, bug reports, or a PR with documentation, +tests, or code. A list of open issues suitable for beginners is [here], and you +can find out how to reach us on the [communication doc]. + +## Appendix: ClickBench Results + +We try not to get too excited by benchmarks, though it is hard not to get caught up +in Benchmaxxing. As noted +above, ClickBench covers only a tiny fraction of what our users actually do, and +reads local files rather than object storage. Even so, DataFusion now sits at +the top of the ClickBench leaderboard for processing partitioned Parquet files, +as measured by ClickBench's combined metric. Results vary +slightly by engine and instance type, but DataFusion matches other +state-of-the-art engines on local files and often significantly exceeds them on +object storage. Plenty left to optimize, of course. + + + +**Figure 2**: [ClickBench results for c7a.metal-48xlarge as of 2026-08-24]; DataFusion is the fastest engine for +processing partitioned Parquet files, by ClickBench's combined metric. See the [ClickBench results page] for the latest results. + +[ClickBench results for c7a.metal-48xlarge as of 2026-08-24]: https://benchmark.clickhouse.com/#system=+hqa|curp|ti%20rud|Distt|kBP%20t|rsoP%20t|Sili|feeid|traan|Tutt&type=-&machine=+ae-&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=- +[ClickBench results page]: https://benchmark.clickhouse.com/ + + +**Figure 3**: [ClickBench results for c6a.4xlarge as of 2026-08-24]; DataFusion is the second +fastest engine for processing partitioned Parquet files (combined metric) on this VM type. See the [ClickBench results page] for the latest results. + +[ClickBench results for c6a.4xlarge as of 2026-08-24]: https://benchmark.clickhouse.com/#system=+hqa|curp|ti%20rud|Distt|kBP%20t|rsoP%20t|Sili|feeid|traan|Tutt&type=-&machine=+ca4e&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=- + + + +**Figure 4**: Average and median normalized execution times for DataFusion 55.0.0 on ClickBench queries, compared to previous releases. +Query times are normalized using the ClickBench definition. See the +[DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/) +for more details. + +## Appendix: Full List of Performance Improvements + +The tables below list the performance improvements in this release along with a +representative measurement for each. Results marked *(micro)* come from +Criterion microbenchmarks and are not expected to translate directly into +end-to-end query speedups. Speedups are reported as ratios of old to new +runtime: "n% faster" means the old runtime was (100+n)% of the new, and +speedups of 2x or more are reported as a multiple. + +### Sort / TopK + +| Improvement | Issue / PR | Representative Result | +|------------------------------------------|--------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| +| Runtime row-group pruning for TopK | [#23036](https://github.com/apache/datafusion/issues/23036), [#22450](https://github.com/apache/datafusion/pull/22450) | [4.2x faster on `topk_tpch` Q8; 5 of 11 queries 3.6-4.2x faster](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | +| Faster `SortPreservingMerge` tie-breaker | [#23107](https://github.com/apache/datafusion/pull/23107) | [8% faster on `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | + +### Window Functions + +| Improvement | Issue / PR | Representative Result | +|-----------------------------------------------|-------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------| +| `LEAD` / `LAG` with `IGNORE NULLS` | [#23711](https://github.com/apache/datafusion/pull/23711) | [21.7x faster for `List`, 10.5x for `Utf8View`](https://github.com/apache/datafusion/pull/23711) *(micro)* | +| Sliding-window `MIN` / `MAX` monotonic deques | [#23827](https://github.com/apache/datafusion/pull/23827) | [up to 3.5x faster (strings; ~2x for numerics)](https://github.com/apache/datafusion/pull/23827#issuecomment-5067789139) *(micro)* | +| Skip fully calculated window partitions | [#24127](https://github.com/apache/datafusion/pull/24127) | [49% faster with 32,768 sparse partitions: 161ms → 108ms](https://github.com/apache/datafusion/pull/24127) | +| Skip re-slicing quiet window partitions | [#24047](https://github.com/apache/datafusion/pull/24047) | [26% faster with 32,768 sparse partitions: 209ms → 166ms](https://github.com/apache/datafusion/pull/24047) | + +### Aggregation + +| Improvement | Issue / PR | Representative Result | +|-----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `approx_distinct` with many groups | [#22768](https://github.com/apache/datafusion/pull/22768) | [101x faster: 1723ms → 17ms (Int64, 50K groups)](https://github.com/apache/datafusion/pull/22768#issuecomment-4627539820) | +| `array_agg(DISTINCT ...)` | [#23716](https://github.com/apache/datafusion/pull/23716) | [4.0x faster at high cardinality](https://github.com/apache/datafusion/pull/23716) *(micro)* | +| `percentile_cont` / `median` | [#23954](https://github.com/apache/datafusion/pull/23954) | [2.7x faster: 247µs → 92µs (`median`, window=256)](https://github.com/apache/datafusion/pull/23954) *(micro)* | +| Multi-column `GROUP BY` type coverage | [#22715](https://github.com/apache/datafusion/issues/22715), [#23523](https://github.com/apache/datafusion/pull/23523) | [46% less memory for mixed-schema keys: 1096KB → 594KB](https://github.com/apache/datafusion/pull/23523) | +| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [#23645](https://github.com/apache/datafusion/issues/23645), [#23646](https://github.com/apache/datafusion/pull/23646) | [~5% faster: 1.13s → 1.07s grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), addressing a [reported out-of-memory crash](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | +| Semi / anti join index alignment | [#22794](https://github.com/apache/datafusion/pull/22794) | [TPC-DS Q15 16% faster](https://github.com/apache/datafusion/pull/22794#issuecomment-4640131159) | + +### Expressions and Functions + +| Improvement | Issue / PR | Representative Result | +|--------------------------------------------------------------------------| --- |----------------------------------------------------------------------------------------------------------------------------| +| Faster `IN` list evaluation | [#19241](https://github.com/apache/datafusion/issues/19241), [#23014](https://github.com/apache/datafusion/pull/23014) | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014) *(micro)* | +| Faster `IN` list (`UInt8` bitmap) | [#23011](https://github.com/apache/datafusion/pull/23011) | [9.7x faster: 38.4µs → 4.0µs](https://github.com/apache/datafusion/pull/23011) *(micro)* | +| Faster `IN` list (`Int8` / `Int16`) | [#23299](https://github.com/apache/datafusion/pull/23299) | [4.4x faster for `i16`, 4-element list](https://github.com/apache/datafusion/pull/23299#issuecomment-4875954295) *(micro)* | +| Preserve dictionary encoding for string functions | [#23930](https://github.com/apache/datafusion/pull/23930) | [360x faster `initcap` on cardinality-10 dictionaries](https://github.com/apache/datafusion/pull/23930) *(micro)* | +| Preserve dictionary encoding for trim family | [#24100](https://github.com/apache/datafusion/pull/24100) | [151x faster `ltrim` on cardinality-10 dictionaries](https://github.com/apache/datafusion/pull/24100) *(micro)* | +| Preserve dictionary encoding for (`ascii`, `bit_length`, `octet_length`) | [#23743](https://github.com/apache/datafusion/pull/23743) | [28.7x faster `ascii`](https://github.com/apache/datafusion/pull/23743) *(micro)* | +| `find_in_set` | [#23460](https://github.com/apache/datafusion/pull/23460) | [24x faster: 1.25ms → 52µs](https://github.com/apache/datafusion/pull/23460) *(micro)* | +| `array_has` with array needle | [#23337](https://github.com/apache/datafusion/pull/23337) | [15.1x faster; join query 0.95s → 0.059s](https://github.com/apache/datafusion/pull/23337) | +| `trunc` with scalar precision | [#23593](https://github.com/apache/datafusion/pull/23593) | [12.9x faster: 9.5µs → 735ns](https://github.com/apache/datafusion/pull/23593) *(micro)* | +| `encode` / hex | [#23456](https://github.com/apache/datafusion/pull/23456) | [5.1x faster: 200µs → 39.5µs](https://github.com/apache/datafusion/pull/23456) *(micro)* | +| `arrays_zip` perfect-list fast path | [#22285](https://github.com/apache/datafusion/pull/22285) | [4.2x faster](https://github.com/apache/datafusion/pull/22285#issuecomment-4537748133) *(micro)* | +| `overlay` | [#22182](https://github.com/apache/datafusion/pull/22182) | [5.3x faster on high-null input: 401µs → 75µs](https://github.com/apache/datafusion/pull/22182) *(micro)* | +| `translate` | [#22171](https://github.com/apache/datafusion/pull/22171) | [4.4x faster: 265µs → 60µs](https://github.com/apache/datafusion/pull/22171) *(micro)* | +| `date_trunc` | [#23542](https://github.com/apache/datafusion/pull/23542) | [4.2x faster for `week`: 23.3µs → 5.6µs](https://github.com/apache/datafusion/pull/23542) *(micro)* | +| `replace` | [#23589](https://github.com/apache/datafusion/pull/23589) | [2.5x faster: 228µs → 91µs](https://github.com/apache/datafusion/pull/23589) *(micro)* | +| `regexp_instr` | [#23540](https://github.com/apache/datafusion/pull/23540) | [90% faster: 49.2µs → 26.0µs](https://github.com/apache/datafusion/pull/23540) *(micro)* | +| `left` / `right` | [#23762](https://github.com/apache/datafusion/pull/23762) | [81% faster on `string_view` long results: 94µs → 52µs](https://github.com/apache/datafusion/pull/23762) *(micro)* | +| `round` | [#23471](https://github.com/apache/datafusion/pull/23471) | [79% faster: 1486ns → 831ns](https://github.com/apache/datafusion/pull/23471) *(micro)* | +| `regexp_match` with literal pattern | [#23547](https://github.com/apache/datafusion/pull/23547) | [59% faster with literal pattern and flags: 280µs → 176µs](https://github.com/apache/datafusion/pull/23547) *(micro)* | +| `date_part` `isodow` | [#23491](https://github.com/apache/datafusion/pull/23491) | [reported 38% improvement](https://github.com/apache/datafusion/pull/23491#issuecomment-4962484527) *(micro)* | + +### Planning + +| Improvement | Issue / PR | Representative Result | +| --- | --- | --- | +| Collapse chained projections | [#22389](https://github.com/apache/datafusion/pull/22389) | [4.0x faster planning: 623ms → 155ms](https://github.com/apache/datafusion/pull/22389) | +| Skip subquery traversal, rewrite in place | [#22298](https://github.com/apache/datafusion/pull/22298) | [29% faster TPC-DS optimization: 220ms → 170ms](https://github.com/apache/datafusion/pull/22298) | +| Don't re-inline CSE'd expensive expressions | [#23459](https://github.com/apache/datafusion/pull/23459) | [67% faster on repeated `power(a, 2)`](https://github.com/apache/datafusion/pull/23459) | +| Skip `ensure_distribution` rebuild for unchanged children | [#22521](https://github.com/apache/datafusion/pull/22521) | [2.9x faster per call: 171µs → 59µs](https://github.com/apache/datafusion/pull/22521) | +| Unified `EnsureRequirements` pass | [#21976](https://github.com/apache/datafusion/pull/21976) | [TPC-H: 8 queries faster, 0 slower](https://github.com/apache/datafusion/pull/21976#issuecomment-4521276189) | +| Predicate reordering heuristic | [#22343](https://github.com/apache/datafusion/pull/22343) | [ClickBench Q21 10-13% faster](https://github.com/apache/datafusion/pull/22343#issuecomment-4483681627) | + +### Scan / IO + +| Improvement | Issue / PR | Representative Result | +| --- | --- | --- | +| Prune unread Parquet leaves for nested columns | [#24090](https://github.com/apache/datafusion/pull/24090) | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | +| Fewer object store requests for CSV | [#22962](https://github.com/apache/datafusion/pull/22962) | [70% faster on TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | +| Lower `repartition_file_min_size` to 1 MiB | [#22439](https://github.com/apache/datafusion/pull/22439) | [TPC-H Q22 68% faster](https://github.com/apache/datafusion/pull/22439#issuecomment-4511995613) | +| Skip page index load when the file has none | [#24149](https://github.com/apache/datafusion/issues/24149), [#24150](https://github.com/apache/datafusion/pull/24150) | [ClickBench (single file) Q1 38% faster](https://github.com/apache/datafusion/pull/24150#issuecomment-5226213508) | + + +[Apache DataFusion]: https://datafusion.apache.org/ +[Rust]: https://www.rust-lang.org/ +[Apache Arrow]: https://arrow.apache.org +[DataFusion's primary design goal]: https://datafusion.apache.org/user-guide/introduction.html#project-goals +[dataframe library]: https://datafusion.apache.org/user-guide/dataframe.html +[Python library]: https://datafusion.apache.org/python/ +[command-line SQL tool]: https://datafusion.apache.org/user-guide/cli/ +[Upgrade Guide]: https://datafusion.apache.org/library-user-guide/upgrading/55.0.0.html +[here]: https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22 +[communication doc]: https://datafusion.apache.org/contributor-guide/communication.html + +[@2010YOUY01]: https://github.com/2010YOUY01 +[@AdamGS]: https://github.com/AdamGS +[@Dandandan]: https://github.com/Dandandan +[@EmilyMatt]: https://github.com/EmilyMatt +[@EdsonPetry]: https://github.com/EdsonPetry +[@JSOD11]: https://github.com/JSOD11 +[@JeelRajodiya]: https://github.com/JeelRajodiya +[@KarpagamKarthikeyan]: https://github.com/KarpagamKarthikeyan +[@Rachelint]: https://github.com/Rachelint +[@Rich-T-kid]: https://github.com/Rich-T-kid +[@SubhamSinghal]: https://github.com/SubhamSinghal +[@adriangb]: https://github.com/adriangb +[@alamb]: https://github.com/alamb +[@andygrove]: https://github.com/andygrove +[@athlcode]: https://github.com/athlcode +[@avantgardnerio]: https://github.com/avantgardnerio +[@blinding-pixels]: https://github.com/blinding-pixels +[@comphead]: https://github.com/comphead +[@crm26]: https://github.com/crm26 +[@davidlghellin]: https://github.com/davidlghellin +[@fordN]: https://github.com/fordN +[@getChan]: https://github.com/getChan +[@gene-bordegaray]: https://github.com/gene-bordegaray +[@geoffreyclaude]: https://github.com/geoffreyclaude +[@gmhelmold]: https://github.com/gmhelmold +[@haohuaijin]: https://github.com/haohuaijin +[@jayshrivastava]: https://github.com/jayshrivastava +[@jkylling]: https://github.com/jkylling +[@joroKr21]: https://github.com/joroKr21 +[@kosiew]: https://github.com/kosiew +[@kumarUjjawal]: https://github.com/kumarUjjawal +[@lyne7-sc]: https://github.com/lyne7-sc +[@mattp5657]: https://github.com/mattp5657 +[@maxburke]: https://github.com/maxburke +[@mbutrovich]: https://github.com/mbutrovich +[@mithuncy]: https://github.com/mithuncy +[@mkleen]: https://github.com/mkleen +[@neilconway]: https://github.com/neilconway +[@niebayes]: https://github.com/niebayes +[@pantShrey]: https://github.com/pantShrey +[@pepijnve]: https://github.com/pepijnve +[@peterxcli]: https://github.com/peterxcli +[@rluvaton]: https://github.com/rluvaton +[@saadtajwar]: https://github.com/saadtajwar +[@simonvandel]: https://github.com/simonvandel +[@sjhddh]: https://github.com/sjhddh +[@stuhood]: https://github.com/stuhood +[@theirix]: https://github.com/theirix +[@timsaucer]: https://github.com/timsaucer +[@tohuya6]: https://github.com/tohuya6 +[@wirybeaver]: https://github.com/wirybeaver +[@yinli-systems]: https://github.com/yinli-systems +[@zhuqi-lucas]: https://github.com/zhuqi-lucas + +[find_in_set]: https://github.com/apache/datafusion/pull/23460 +[approx_distinct]: https://github.com/apache/datafusion/pull/22768 +[file_row_index]: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#file-row-index +[input_file_name]: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#input-file-name +[unnest_outer]: https://github.com/apache/datafusion/pull/22100 +[array_scale]: https://github.com/apache/datafusion/pull/22466 +[array_add]: https://github.com/apache/datafusion/pull/22459 +[array_subtract]: https://github.com/apache/datafusion/pull/22556 +[array_sum]: https://github.com/apache/datafusion/pull/22542 +[array_avg]: https://github.com/apache/datafusion/pull/23168 +[array_first]: https://github.com/apache/datafusion/pull/23267 +[hypot]: https://github.com/apache/datafusion/pull/23774 +[atan2]: https://github.com/apache/datafusion/pull/23962 +[weekday]: https://github.com/apache/datafusion/pull/22740 +[monthname]: https://github.com/apache/datafusion/pull/21639 +[concat_ws]: https://github.com/apache/datafusion/pull/20928 + +[appendix]: #appendix-clickbench-results +[Partitioning::Range]: https://docs.rs/datafusion/55.0.0/datafusion/physical_expr/struct.RangePartitioning.html +[TableProvider]: https://docs.rs/datafusion/55.0.0/datafusion/catalog/trait.TableProvider.html +[TableProvider::merge_into]: https://docs.rs/datafusion/55.0.0/datafusion/catalog/trait.TableProvider.html#method.merge_into +[SpillFile]: https://docs.rs/datafusion/55.0.0/datafusion/execution/spill_file/trait.SpillFile.html +[TempFileFactory]: https://docs.rs/datafusion/55.0.0/datafusion/execution/spill_file/trait.TempFileFactory.html +[RuntimeEnv]: https://docs.rs/datafusion/55.0.0/datafusion/execution/runtime_env/struct.RuntimeEnv.html +[object_store_spill.rs]: https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/data_io/object_store_spill.rs +[ObjectStore]: https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html + +[#19241]: https://github.com/apache/datafusion/issues/19241 +[#20002]: https://github.com/apache/datafusion/issues/20002 +[#20746]: https://github.com/apache/datafusion/issues/20746 +[#20763]: https://github.com/apache/datafusion/pull/20763 +[#21668]: https://github.com/apache/datafusion/pull/21668 +[#21768]: https://github.com/apache/datafusion/pull/21768 +[#21767]: https://github.com/apache/datafusion/pull/21767 +[#21882]: https://github.com/apache/datafusion/pull/21882 +[#21976]: https://github.com/apache/datafusion/pull/21976 +[#21992]: https://github.com/apache/datafusion/issues/21992 +[#22100]: https://github.com/apache/datafusion/pull/22100 +[#22230]: https://github.com/apache/datafusion/pull/22230 +[#22298]: https://github.com/apache/datafusion/pull/22298 +[#22343]: https://github.com/apache/datafusion/pull/22343 +[#22389]: https://github.com/apache/datafusion/pull/22389 +[#22439]: https://github.com/apache/datafusion/pull/22439 +[#22450]: https://github.com/apache/datafusion/pull/22450 +[#22529]: https://github.com/apache/datafusion/pull/22529 +[#22534]: https://github.com/apache/datafusion/pull/22534 +[#22604]: https://github.com/apache/datafusion/pull/22604 +[#22652]: https://github.com/apache/datafusion/pull/22652 +[#22715]: https://github.com/apache/datafusion/issues/22715 +[#22710]: https://github.com/apache/datafusion/issues/22710 +[#22768]: https://github.com/apache/datafusion/pull/22768 +[#22940]: https://github.com/apache/datafusion/pull/22940 +[#22945]: https://github.com/apache/datafusion/pull/22945 +[#22978]: https://github.com/apache/datafusion/pull/22978 +[#22988]: https://github.com/apache/datafusion/pull/22988 +[#23066]: https://github.com/apache/datafusion/pull/23066 +[#23148]: https://github.com/apache/datafusion/pull/23148 +[#23459]: https://github.com/apache/datafusion/pull/23459 +[#23494]: https://github.com/apache/datafusion/issues/23494 +[#23522]: https://github.com/apache/datafusion/pull/23522 +[#23566]: https://github.com/apache/datafusion/pull/23566 +[#23645]: https://github.com/apache/datafusion/issues/23645 +[#23657]: https://github.com/apache/datafusion/pull/23657 +[#23743]: https://github.com/apache/datafusion/pull/23743 +[#23807]: https://github.com/apache/datafusion/pull/23807 +[#23930]: https://github.com/apache/datafusion/pull/23930 +[#23965]: https://github.com/apache/datafusion/pull/23965 +[#24018]: https://github.com/apache/datafusion/pull/24018 +[#24028]: https://github.com/apache/datafusion/pull/24028 +[#24035]: https://github.com/apache/datafusion/pull/24035 +[#24061]: https://github.com/apache/datafusion/pull/24061 +[#24068]: https://github.com/apache/datafusion/pull/24068 +[#24090]: https://github.com/apache/datafusion/pull/24090 +[#24100]: https://github.com/apache/datafusion/pull/24100 +[#24150]: https://github.com/apache/datafusion/pull/24150 +[#22395]: https://github.com/apache/datafusion/issues/22395 + +[datafusion-spark crate]: https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html diff --git a/content/images/datafusion-55.0.0/clickbench.c6a.4xlarge.2026_08_24.png b/content/images/datafusion-55.0.0/clickbench.c6a.4xlarge.2026_08_24.png new file mode 100644 index 00000000..e22e3972 Binary files /dev/null and b/content/images/datafusion-55.0.0/clickbench.c6a.4xlarge.2026_08_24.png differ diff --git a/content/images/datafusion-55.0.0/clickbench.c7a.metal-48xlarge.2026_08_24.png b/content/images/datafusion-55.0.0/clickbench.c7a.metal-48xlarge.2026_08_24.png new file mode 100644 index 00000000..8ac29942 Binary files /dev/null and b/content/images/datafusion-55.0.0/clickbench.c7a.metal-48xlarge.2026_08_24.png differ diff --git a/content/images/datafusion-55.0.0/commits_contributors.svg b/content/images/datafusion-55.0.0/commits_contributors.svg new file mode 100644 index 00000000..48bd3269 --- /dev/null +++ b/content/images/datafusion-55.0.0/commits_contributors.svg @@ -0,0 +1,49 @@ + + DataFusion development activity: releases 53.0.0 to 55.0.0 + + + + + Commits + + + + + 475 + 740 + 877 + 53.0.0 + 54.0.0 + 55.0.0 + + + Commits / day + + + + + 6.8 + 9.6 + 12.4 + 53.0.0 + 54.0.0 + 55.0.0 + + + Contributors + + + + + 114 + 139 + 175 + 53.0.0 + 54.0.0 + 55.0.0 + diff --git a/content/images/datafusion-55.0.0/performance_over_time_clickbench.png b/content/images/datafusion-55.0.0/performance_over_time_clickbench.png new file mode 100644 index 00000000..7ede450b Binary files /dev/null and b/content/images/datafusion-55.0.0/performance_over_time_clickbench.png differ