[JingsongLi]

Purpose

Use DataFusion 55's native MERGE planner and Paimon's table provider for execution. Planning and EXPLAIN must not mutate tables; executing the DML plan applies the merge and returns its affected-row count.

Brief change log

• Implement TableProvider::merge_into with typed DataFusion expressions and a DataSinkExec, replacing the custom MERGE AST-to-SQL execution path. Keep execution in merge_into.rs and unit tests in merge_into/tests.rs.
• Preserve data-evolution partial-column updates, copy-on-write rewrites, atomic publication, and five-attempt conflict retries. Capture the source once and rebuild the target scan and writers on retry, including structured row-ID existence conflicts.
• Preserve positional inserts excluding the virtual _ROW_ID and source-scoped NOT MATCHED expressions, including nested fields and lambda bindings. Retain session variable providers, reject target dependencies in NOT MATCHED, and honor SQLOptions.
• Prune unused source and target columns, preserving physical filter dependencies, aliases, and zero-column row counts. Avoid collisions with internal tracking-column names and collapse duplicate matches for a sole unconditional DELETE while rejecting conflicting target actions.
• Scope conditional predicates with CASE so inapplicable rows are never evaluated. Abort known-unsubmitted staged files when a later action fails, without deleting files after a commit with an uncertain outcome.
• Fix the two timestamp procedure tests: $snapshots.commit_time is a local TIMESTAMP, while procedure arguments are epoch milliseconds. Use deterministic snapshot metadata times and assert exact/in-between timestamp boundaries and the selected snapshots.

Tests

• RUST_MIN_STACK=16777216 cargo test --locked -p paimon-datafusion --all-targets (no skip filters).
• Timestamp procedure tests with TZ=UTC and TZ=Asia/Shanghai.
• Sixteen native MERGE integration tests cover side-effect-free planning/EXPLAIN, direct native-session execution, conflict retries, staged-file cleanup, duplicate DELETEs, scoped predicates, variables/lambdas/nested fields, BLOB projection, filter dependencies, aliases, zero-column source cardinality, and internal tracking-column collisions. Review regressions exercise DuckDB/Spark lambdas and session variables in ON, CoW matched predicates (true and false), UPDATE assignments, and NOT MATCHED inserts/predicates.
• cargo fmt --all -- --check
• cargo clippy --locked -p paimon-datafusion --all-targets --no-deps -- -D warnings
• git diff --check

API and Format

MERGE through SQLContext now returns a native lazy DML DataFrame; collect/execute the plan to perform the write. Storage formats and public Rust signatures are unchanged. The executor materializes the captured source and CoW target batches; large-input memory/performance behavior has not been benchmarked. Alias resolution follows the native planner; existing data-evolution action limitations remain.

Documentation

Update SQL support, MERGE execution and syntax limits, SQLOptions behavior, and the epoch-millisecond contract for timestamp procedures.

[SteNicholas]

@JingsongLi, I left two inline comments on MERGE expression handling. PTAL.

[SteNicholas]

@JingsongLi, one non-blocking test-helper cleanup suggestion.

[SteNicholas]

LGTM.