[JingsongLi]

Purpose

Follow-up to #1236. Paimon scans report UnknownPartitioning even when table partition keys prove disjoint ranges, preventing DataFusion 55 from reusing that distribution for compatible aggregations and joins.

Brief change log

• Group whole splits using typed Arrow comparisons with ASC NULLS LAST, keeping equal keys together and combining adjacent key groups within target_partitions. Support Boolean, integer, string, date, millisecond time, Decimal128, and composite partition keys.

• Native primary-key scans require every partition column to belong to the primary key. Non-key partition columns can be changed by current or historical value aggregation, so these layouts retain the original unknown distribution regardless of the current merge engine.

• Use the shared scan construction path for regular, Variant, and audit-log scans, and expose range metadata in EXPLAIN. This declares distribution without claiming rows within each range are sorted.

• Preserve original split assignments and UnknownPartitioning for unsupported types, unavailable partition keys, malformed metadata, authorization rules, and empty/single-partition/single-key scans.

• Stabilize the shared-cache batch lock regression by retaining duplicate descriptors and explicitly unlocking before non-blocking reacquisition, covering descriptor inheritance during concurrent process tests.

Tests

Added 5 unit tests and 12 integration tests covering boundaries, duplicate keys and NULLs across files, composite keys and projection, unsupported/authorization fallbacks, aggregation exchange elimination, Hash/Sort-Merge Join alignment after PK updates, Variant pushdown, and audit-log scans. Native PK regressions verify aggregation, partial-update sequence groups, and persisted aggregate values after a merge-engine change; full-PK partition columns retain range metadata and exchange elimination. Compatible-range Hash Join and Sort-Merge Join cases assert correct many-to-many results without exchanges. A format-table fixture with partition columns absent from its Parquet files verifies restored row membership and correct exchange-free GROUP BY results.

Passed:

• cargo test --locked -p paimon --lib --features vortex io::cache::disk::shared::tests (24 tests)
• cargo test --locked -p paimon --lib --features vortex -- --test-threads=4 (4823 passed, 6 ignored)
• RUST_MIN_STACK=16777216 cargo test --locked -p paimon-datafusion --test range_partitioning
• RUST_MIN_STACK=16777216 cargo test --locked -p paimon-datafusion --test format_table_partition_filters --test format_table_statistics
• cargo clippy --locked -p paimon-datafusion --test range_partitioning --no-deps -- -D warnings
• cargo fmt --all -- --check

The preceding ec02b94f revision also passed the full DataFusion suite with the two baseline failures below excluded, and all GitHub CI jobs. This follow-up changes tests only.

The unfiltered test run failed on test_create_tag_from_timestamp and test_rollback_to_timestamp; both failures also reproduce in a clean worktree at the unchanged base 5bb42a11. Full dependency linting encounters six existing clippy::nonminimal_bool diagnostics in the core paimon crate.

API and Format

No public API or storage format changes. Eligible scans expose Range distribution metadata; scan parallelism can be lower than the target when there are fewer distinct partition keys.

Documentation

Updated docs/src/sql.md with supported types, grouping behavior, ordering guarantees, and fallback conditions.

[SteNicholas]

@JingsongLi, I left some comments for extending regression coverage. PTAL.

[SteNicholas]

LGTM.