[JingsongLi]

Purpose

Upgrade DataFusion 54 to 55.2.0 on the current Rust 1.95 main branch, unify the workspace on Arrow 59, and integrate DataFusion 55 scan pruning, UDF optimization, and EXPLAIN metrics.

Brief change log

• Rebase onto main (aec0ce70). Upgrade DataFusion/FFI/Spark to 55.2.0, Arrow/Parquet to 59.3.2, ORC to 0.9, and PyO3 to 0.29; use Python DataFusion 55.0.0.
• Upgrade Mosaic to 0.3.0 and Vortex to 0.90.0. Remove the Arrow 58 aliases and the Arrow 59/58 C Data bridge completely. Refresh dependency reports and remove obsolete PyO3/Thrift advisory exceptions.
• Migrate Vortex Arrow conversion to session APIs and layout traversal to logical slots. Preserve nested ROW validity, projection, and row selections; add a real Vortex 0.75 golden file to verify backward reads.
• Adapt main's Spark and selection aggregates to DataFusion 55's grouped-state API, including partial-aggregation bypass and nullable FILTER inputs. Preserve Paimon's any_value(value[, ignoreNulls]) semantics and subsequent user function overrides.
• Refresh dynamic predicates before each unread Parquet row group, retaining exact decoder filtering, row selections, memory admission, and conservative fallback. Floating-point predicates retain DataFusion evaluation because Parquet bounds exclude NaN.
• Declare strict metadata for NULL-propagating variant/blob functions while preserving is_variant_null; expose actual output rows, batches, bytes, and compute time for regular and audit-log scans, including PostgreSQL-style EXPLAIN JSON.
• Keep MAP and MULTISET on Arrow Map with non-nullable keys. Remove the multiset List representation, metadata, and format conversions; Arrow runtime validation rejects NULL keys. Persist the C vector-test fixture schema so main's automatic compaction can reload it.
• Fix cross-thread Python writer collection while retaining exclusive method access. Expose URI factory/reader/stream references to GC, bound retired-reference retention, and preserve single-owner stream cleanup. Make cancellation cleanup callbacks safe to reenter Native APIs.

Tests

Validated locally on macOS with the final source and rebuilt Python extension:

• Rust library tests: 5,550 passed, 8 ignored, including 151 Python-binding Rust tests with the configured Python home/site-packages. DataFusion tests use CI's 16 MiB test-thread stack.
• Related core, DataFusion, and Java-golden integration tests: 242 passed across 13 targets, including non-null map/multiset values, nested native writes, schema evolution, incremental diff, and Mosaic/Vortex.
• Python: 730 passed with zero warnings, using PyArrow 23.0.1 and Python DataFusion 55.0.0 with Rust Arrow 59.3.2. An explicit Python query also verifies any_value(..., false/true) after rebuilding the final extension.
• Rust 1.95.0: locked workspace check with all targets and all features.
• Rust 1.98.0: locked workspace Clippy with all targets and Vortex, denying warnings; formatting and diff whitespace checks.
• Dependency report verification, cargo-deny advisories/licenses, release-license verification, and 13 release-script tests passed.

Regression coverage includes Vortex 0.75 disk compatibility, nested independent NULLs, reordered/sparse projection, Mosaic reads/writes, aggregate state conversion, custom function overrides, MERGE, strict-UDF join elimination, live TopK I/O pruning, signed NaN/zero handling, actual pgjson scan metrics, and Java Row/Avro/Parquet compatibility.

The final Python suite treats PytestUnraisableExceptionWarning and PytestUnhandledThreadExceptionWarning as errors and completed with zero warnings. New regressions cover cross-thread GC for both writer types, captured/replaced callbacks, shared and independent URI readers, explicit thread handoff and concurrent-call rejection, reference retirement after rejected replacements/prepares, single-owner stream cleanup, and cancellation-time Native callback reentry. Each reproduced GC/resource/runtime failure was verified before its fix. Multiple review passes found no remaining actionable issues. Linux/Windows execution of the new Python fixes remains for CI; no benchmark speedup is claimed.

API and Format

DataFusion consumers must use 55.x and Arrow consumers must use 59.x. Mosaic and Vortex now use Arrow 59 directly. RowFilterFactory gains optional pruning methods with conservative defaults.

MAP<K, V> and MULTISET<T> both use Arrow Map with non-nullable entry structs and keys. Multiset counts remain non-null Int32. Column and ordinary map value nullability retain their existing rules. Actual NULL keys are rejected by Arrow runtime validation. The Paimon logical table schema and Java-compatible Row/Avro/Parquet file encodings are unchanged; no multiset List metadata or representation bridge is introduced.

Documentation

Update SQL dependency examples, explain DataFusion 55 integration boundaries, and document the non-null Arrow Map key policy. Paimon MERGE continues through SQLContext; generic provider MERGE, file virtual columns, range declarations, and query-planner FFI are not advertised by this upgrade.

[leaves12138]

LGTM