[EPIC] Iceberg work for 1.3.0: native writes on by default and scan correctness #7293

What is the problem the feature request solves?

Comet 1.2.0 is branched and 1.3.0 is due 2026-11-20. This epic collects the Iceberg work that should land in that window. It has two goals:

1. Turn native Iceberg writes (spark.comet.write.iceberg.enabled) back on by default on main, so they run as the default for the whole 1.3.0 cycle. The criteria are in Enable the split-operator plan and native Iceberg writes by default #5644 and the plan to restore the default after the branch cut is in feat: make native Iceberg writes opt-in for 1.2.0 #6962.
2. Fix the native Iceberg scan bugs that return wrong rows or values, because a faster write path does not help if reads are wrong.

Describe the potential solution

Enable native Iceberg writes by default (#5644, #5649)

• Open a PR that reverts the default from feat: make native Iceberg writes opt-in for 1.2.0 #6962 back to true, with docs and the upgrade guide updated
• Enable the split-operator plan and native Iceberg writes by default #5644: classify the remaining fallback reasons in the Iceberg CI write report (input not in Arrow format, writes with no recorded reason, "IcebergWrite is not supported", "could not determine the table format-version", Spark-planned AppendData and streaming writes). Check whether INSERT ... VALUES falling back through LocalTableScanExec accounts for most of the first bucket
• Enable the split-operator plan and native Iceberg writes by default #5644: assert manifest metrics parity per data type on the pinned Iceberg versions (1.5.2, 1.8.1, 1.10.0, 1.11.0)
• Decide keep-or-lift for each remaining native Iceberg write eligibility restriction #5643: a keep-or-lift decision for each remaining eligibility restriction
• fix: fall back for brotli Iceberg writes when Hadoop's BrotliCodec cannot be loaded #6996: fall back for brotli writes that no JVM reader can read
• Native Iceberg writer records NaN as a manifest lower/upper bound for all-NaN float and double columns on Parquet 60 #7242: NaN manifest bounds on Parquet 60. This must be fixed before the DataFusion 56 upgrade ([EPIC] DataFusion 56 upgrade: regressions found by tracking DataFusion main #6410) lands
• Native Iceberg writes accept write.parquet.compression-codec=none, which iceberg-java rejects #7008: write.parquet.compression-codec=none is accepted natively but rejected by iceberg-java
• feat: bump iceberg-rust past apache/iceberg-rust#3255 and narrow the nested schema evolution fallback #7028: bump iceberg-rust and narrow the nested schema evolution fallback

Writer coverage and performance, if time allows:

• Native Iceberg writes into NOT NULL columns fall back on Spark 3.5+ because Comet has no native AssertNotNull #7049: native AssertNotNull so writes into NOT NULL columns do not fall back
• Native Iceberg writes: stop falling back on settings iceberg-java's writer ignores #6955, Native Iceberg writes: translate the Parquet settings that have a parquet-rs equivalent #6954, Native Iceberg writes: support uuid and unknown columns #6959: lift fallbacks on settings iceberg-java ignores, translate the Parquet settings that have a parquet-rs equivalent, and support uuid and unknown columns
• Hand natively written Iceberg files' Parquet footers to the JVM instead of reading them back #6772: hand written Parquet footers to the JVM instead of reading them back, which matters on object stores

Native Iceberg scan correctness

Wrong results:

• Native Iceberg scan drops -0.0 rows for floating-point predicates #7136: drops -0.0 rows for floating-point predicates
• Native Iceberg scan drops NULL rows for NOT (col <=> value) #7137: drops NULL rows for NOT (col <=> value)
• Native Iceberg scan drops rows from data files that store a narrower type than the column #7138: drops rows from data files that store a narrower type than the column
• Native Iceberg scan applies equality deletes to the wrong column when column names differ only in case #7129: applies equality deletes to the wrong column when column names differ only in case
• Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot #6822: wrong values for a column renamed after a VERSION AS OF snapshot
• Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping #6790, Native Iceberg scan returns nested values that Spark reads as NULL in files without field ids or a name mapping #6794, Native Iceberg scan returns NULL for an INT96 column read through a name mapping when other columns are selected #7212: NULLs where Spark reads values, for files read through a name mapping or without field ids

Additional context

This epic lists the work and does not decide it. Items may move to a later milestone, and the writer coverage items are optional. Related epics: #5649 (native Iceberg writes), #7040 and #7051 (nested types in the native scans and writers), and #6911 (end-to-end Iceberg performance).