[EPIC] DataFusion 56 upgrade: regressions found by tracking DataFusion main #6410

What is the problem the feature request solves?

Draft PR #6404 builds Comet against DataFusion main instead of a release. That way, upstream changes reach Comet's CI weeks before DataFusion 56.0.0 is released (targeted for Oct/Nov, apache/datafusion#24461). The PR is long-lived and bumped weekly. It is not meant to merge before that release. It currently pins these versions:

This issue tracks the regressions that PR finds. The goal is to fix them upstream, or work around them in Comet, before the release rather than after it.

Describe the potential solution

Regressions

Check an item off once #6404 pins a revision that has the fix.

• The native Iceberg writer puts NaN bounds in manifests.
  • CometIcebergWriteActionSuite "native acceleration: NaN float/double manifest metrics match the JVM writer" fails (run). The native writer produces [1,0.0,NaN,2,0.25,NaN,3,NaN,NaN] where the JVM writer produces [1,0.0,2.5,2,0.25,1.5,3,null,null].
  • Cause: Parquet 60 now writes NaN min/max statistics for a column chunk whose values are all NaN, where 59 wrote none (get_min_max in parquet/src/column/writer/encoder.rs). iceberg-rust's MinMaxColAggregator::update copies the Parquet statistics into lower and upper bounds without filtering out NaN. So a data file with an all-NaN float or double column gets NaN bounds, where iceberg-java writes none.
  • Impact: iceberg-java readers treat a NaN bound as unreliable, so this should not cause wrong results there. But the manifests no longer match the JVM writer's.
  • The fix belongs in iceberg-rust, ideally as part of its Arrow 60 bump (Bump to Arrow 60 iceberg-rust#3257). Not reported upstream yet.

Behavior changes reviewed and accepted

• Arrow 60 accepts empty entries in unsorted Variant dictionaries (Add proptest fuzzing to parquet-variant and implement fixes for findings arrow-rs#10352). Comet's empty-key workaround used to reject two malformed dictionaries. Both now pass through unchanged, as they do in Spark.
• A failed read from a JVM input stream now reports Cannot get next batch from input stream. Error code: N. Producer error: .... This is because Arrow 60 made the C Stream callbacks private, so Comet now uses the stock ArrowArrayStreamReader.
• An invalid shuffle IPC schema message now returns a decode error instead of panicking (try_fb_to_schema).
• DataFusion's new floor/ceil return types and map_extract absent-key results don't reach Comet, because Comet registers its own implementations.
• DataFusion now treats a missing Parquet null count as unknown rather than zero. That is a correctness fix. It can reduce pruning on files written by parquet-rs versions before 53.1.

Upstream follow-ups

• Move the iceberg-rust pin to a main revision once Bump to Arrow 60 iceberg-rust#3257 merges.
• DataFusion 56 deprecates HashJoinExec::with_dynamic_filter_expr with no replacement, and DynamicFilterJoinExec still needs it. File a DataFusion issue that describes the use case.
• Remove the Variant workarounds that Arrow 60 makes unnecessary ([Variant] Track and remove native Variant compatibility workarounds after upstream fixes #5477).

Open Comet issues that depend on DataFusion main

Each issue below waits on a DataFusion change that is on main but not in branch-55, so it can only be finished or rechecked once Comet moves to DataFusion 56. Check an item off once Comet has adopted the change and the issue is resolved.

• Remove the spill replay workaround from the memory pools once DataFusion includes apache/datafusion#25383 #6583: remove the spill replay workaround from the memory pools. Needs fix: leave memory for aggregate spill replay datafusion#25383 (merged to main on 2026-09-17, the branch-55 backport [branch-55] fix: leave memory for aggregate spill replay (#25383) datafusion#25814 was closed).
• Native PartialMerge over input sorted on a prefix of its keys fails to spill on DataFusion 55.2 #6880: on 56, drop the switch in planner/aggregate.rs that plans a PartialMerge over input sorted on a key prefix as Final, since Single runs ClusteredSingleAggregateStream there. Also update ORDERED_FINAL_AGGREGATE in spill_replay.rs (used by the fair pool exemption) to ClusteredFinalAggregateStream[, which refactor: use group clustering terminology in aggregate execution datafusion#24697 renamed the stream and consumer to.
• Run SortAggregateExec as a native hash aggregate without its input sort #7146: run SortAggregateExec as a native hash aggregate. The issue relies on the ExternalSorter spill workspace fix, fix: preserve external sort workspace across spilling datafusion#24740 for External sort can lose reserved spill workspace when memory availability decreases datafusion#24739 (merged to main on 2026-09-04, not in branch-55).
• [Variant] Track and remove native Variant compatibility workarounds after upstream fixes #5477: remove the native Variant compatibility workarounds. The footer reconstruction step needs the lazy per-file schema hook from feat: Support lazy per-file Parquet Arrow schema derivation datafusion#25343 (merged to main on 2026-09-16, not in branch-55).
• Native existence join enumerates every duplicate build match (N*M candidates for M markers) #6484: native existence join enumerates every duplicate build match. Related upstream join work is feat(pwmj): support LeftSemi/LeftAnti existence joins via classic scan datafusion#23870 (merged to main on 2026-08-14, not in branch-55). It is not confirmed to fix the N*M enumeration, so recheck the issue on 56.
• Native Parquet scan assigns row groups to splits differently from Spark #6512
• Wire stack depends on feat: support Spark-compatible stack function datafusion#26266
  Tested revisions

Additional context

Steps for triaging a red run on #6404:

1. Re-run it once and compare with main's latest nightly, since some failures are known flakes.
2. Run the failing test on the merge commit with the previous upstream revision. This tells an upstream change apart from a Comet change on main.
3. Bisect the upstream commits between the two revisions, using a local [patch] override. DataFusion main gets 100 to 120 commits a week, so this takes about 7 steps.

Spark's SQL tests, Iceberg's Spark tests and the non-default Spark profiles have not run on #6404 yet. They will run once the run-spark-4.1-tests, run-iceberg-tests and run-all-spark-profiles labels are applied.