Write blog post for 1.2.0 release #6889

What is the problem the feature request solves?

The Comet 1.2.0 release needs a blog post. The release process is to draft it in a shared document, agree on the content, and then open a PR against datafusion-site.

Release tracking: #6550. The 1.1.0 post is tracked in #6120.

Describe the potential solution

Draft the post from the summary below, then open the datafusion-site PR so it can go out with the 1.2.0 announcement.

The summary is a snapshot of main at 333bae8 (2026-10-23), refreshed from the 2026-10-12 snapshot at 019f3be. Counts, defaults and the list of open PRs will move before the release branch is cut, so refresh them then. Performance figures are quoted from the PR descriptions and were not re-measured, so re-verify any number before it goes in the post.

Additional context

What is new on main since 1.1.0

branch-1.1 was cut from main at 36ab57c (2026-09-25), so git log 1.1.0..main overcounts. After removing the commits that are also on branch-1.1 as backports (matched by PR number and by title), main has 336 new commits from 2026-09-26 to 2026-10-23, from 34 authors plus Dependabot: 123 fixes, 50 features, 42 performance changes, 32 docs changes, 37 test changes and 52 chore, refactor, CI and build changes. For scale, 1.1.0 contains 418 commits that 1.0.0 does not.

branch-1.1 has 18 commits after the 1.1.0 tag (backports for a 1.1.1 release). They are excluded from the count above, so anything listed below that is also among those backports is not new relative to the branch head.

1. Spark-exact floats and timezones. A systematic sweep instead of per-expression fixes.

• The timezone EPIC [EPIC] Timezone handling bugs #6335 is closed (8 of 8). It covers correct timezone labels on native timestamp results (fix: label native timestamp_seconds results as UTC timestamps #6341, fix: keep the input's timezone label on native date_trunc results #6345, fix: relabel CASE and COALESCE timestamp branches instead of panicking #6347), session timezone ID normalization (fix: normalize session timezone IDs before passing them to native code #6351), a warning when the native and JVM timezone databases differ (fix: warn when native and JVM timezone databases differ #6353), per-row date_trunc formats (fix: truncate per-row date_trunc formats with the literal-format kernel #6354), and a test check that fails when a native plan mislabels a timestamp (test: fail Comet tests when a native plan mislabels a timestamp's timezone #6868, test: verify native timestamp output timezone labels #6930).
• The float EPIC [EPIC] Match Spark's -0.0 and NaN semantics systematically instead of per expression #6385 has 14 of 18 checklist items done (open: Share one Spark-equality match kernel between array_contains, array_position and array_remove #6614, test: move CometFloatSemanticsSuite's matching cases to Comet SQL tests #6636 with PR test: move the float semantics sweep to SQL file tests #7114, Native mode treats all NaNs as one value on Spark 3.4, where Spark counts each NaN separately #6738 with PR fix: fall back to Spark for mode on FLOAT and DOUBLE before Spark 3.5.2 #7115, and native map construction on Spark 4.0+, Native map construction doesn't match Spark 4.0+ float key normalization (-0.0 keys, missing DUPLICATED_MAP_KEY) #6549, where fix: stop building maps natively when the key type holds a FLOAT or DOUBLE #6992 now falls back to Spark for float keys). Signed zero and NaN now follow Spark in hashing (fix: hash NaN the way Spark does in hash, xxhash64 and native shuffle #6413), min/max/greatest/least (fix: match Spark's float ordering in min, max, greatest and least #6457, fix: compare greatest and least struct arguments by position #6760, fix: order nulls inside nested values first in least #6761), nested sort and window keys (fix: match Spark's float ordering for sort and window keys nested in arrays and structs #6475, fix: preserve Spark nested null ordering in native sort #6968), array_remove, sort_array, array_distinct and array_union (fix: match Spark's float semantics in array_remove and sort_array #6518, feat: normalize floats in native array_distinct and array_union #6563), signum (fix: return -0.0 for signum(-0.0), as Spark does #6556), percentile (fix: match Spark percentile ordering for negative NaNs #6829), arrays_overlap (fix: treat -0.0 and 0.0 as one value in arrays_overlap on Spark 4.0.5, 4.1.4 and 4.2.1 #6989) and Remainder (fix: compare a Remainder divisor with zero in Spark's ordering without ANSI #6896).
• Collation: unsupported collated expressions now fall back to Spark or route through the codegen dispatcher (fix: reject casts involving non-default collated strings #5302, fix: reject non-default collations in all sort keys #6206, fix: Route collated array element comparisons through the codegen dispatcher #6471, fix: Route collated instr, substring_index, trim, greatest and least through the codegen dispatcher #6480).

2. Comet's in-memory cache stays opt-in. spark.comet.exec.inMemoryCache.enabled is off by default in 1.1.0 and in 1.2.0. During the 1.2 cycle #5634 enabled it by default on Spark 3.5+ and #6963 turned that off again before the release, because of three open wrong-results bugs (#6787, #6898, #6922) that default settings could reach. No released version had it on by default, so the post should describe the decision (we held it back for correctness), not a revert. The hardening that landed on main is still worth a mention: serializer and Kryo gating (#6360, #6537), EXPLAIN and SQL-tab fidelity (#6453, #6574, #6577), planner statistics (#6412), fused vector reads (#5859), the reader class generated once per executor (#6860) and an opt-in lz4 codec (#6859). Audit follow-ups are tracked in #6834.

3. Iceberg native writes and row-level operations.

• Native Iceberg writes (spark.comet.write.iceberg.enabled) are off by default in 1.2.0, as in 1.1.0, where they were experimental. During the 1.2 cycle feat: enable native Iceberg writes by default #6664 enabled them by default and feat: make native Iceberg writes opt-in for 1.2.0 #6962 turned that off the next day, so no released version had them on by default. 1.2.0 is their first release as a supported opt-in feature, and the default is revisited after the branch is cut. Plan for each fallback: docs: record the plan for each native Iceberg write fallback #6961 (open).
• Writes to format-version 3 tables (feat: support native Iceberg writes to format-version 3 tables #6672), per-location credentials for reads and writes (feat: per-location credentials for native Iceberg reads and writes #6478), dictionary encoding that matches iceberg-java (perf: choose dictionary encoding like iceberg-java in the native Iceberg writer #6241), writer buffers charged to the task's memory pool (feat: charge native write buffers to the task's memory pool #6247), fanout partitions closed early when the memory pool refuses the writer (feat: close fanout partitions early when the memory pool refuses the native Iceberg writer #6773), float and double identity partitions written natively (feat: bump iceberg-rust and write float and double identity partitions natively #6851), and eligibility gates so unsupported writes fall back to iceberg-java (fix: gate unsupported Iceberg S3 write settings #6441, fix: read the Iceberg write gate's storage scheme the same way the native factory does #6502, fix: fall back for Iceberg writes with float or double identity partitions #6531, fix: decline native Iceberg writes through a void partition field whose source column was dropped #6677, fix: fall back to Spark when Iceberg reports an output ordering #7043, fix: fall back when an Iceberg equality delete is keyed on a nested field #7122).
• The writer's dictionary pre-pass is 1.8x to 3.9x faster (perf: make the native Iceberg writer's dictionary pre-pass 1.8x to 3.9x faster #6937).
• Reads: only the nested fields Spark keeps are read by the native scan (perf: read only the nested fields Spark keeps in the native Iceberg scan #6725), and the native scan opens two data files concurrently by default (perf: default the native Iceberg scan to two concurrent data files #7132).
• Row-level operations: native MergeRowsExec for MERGE INTO (feat: add native support for MergeRowsExec (row-level MERGE INTO) #5318, spark.comet.exec.mergeRows.enabled, off by default), MergeSummary on Spark 4.1+ (feat: support native MergeSummary on Spark 4.1+ #6640) and Spark 4.2 write transactions (fix(iceberg): preserve Spark 4.2 write transactions #6587).
• The write settings were renamed (chore: move native write configs under spark.comet.write #6494): spark.comet.iceberg.write.enabled is now spark.comet.write.iceberg.enabled, and spark.comet.parquet.write.enabled is now spark.comet.write.parquet.enabled.

4. Memory and spill.

• Native plan memory is returned before the Spark task ends (fix: return a native plan's memory before its Spark task ends #6261), acquires are retried after Spark drops the task's entry (fix: retry a native acquire that Spark failed after dropping the task's entry #6310, fix: retry a shuffle page allocation that Spark failed after dropping the task's entry #6403, fix: share the missing task entry retry and stop retrying it after the task has ended #6813), a spilled final aggregate reads its spill files back past its memory share (fix: let a spilled final aggregate read its spill files back past its memory share #6544), grouped integer SUM state is counted in the reservation (fix: count grouped integer SUM state in the aggregate's memory reservation #6370), spill directories are created on the first spill (perf: create a native plan's spill directories on its first spill #7066), and the native memory limit follows YARN and Kubernetes container sizing (fix: derive the native memory limit the way YARN and Kubernetes size executor containers #6375).
• The bypass-merge JVM shuffle writer starts each partition's page small and lets it grow (perf: start bypass shuffle partition writers on small pages that grow #7059). The PR reports REPARTITION(64) on TPC-H orders going from 266 s and 2.48 GB to 1.1 s and 156 MB, and on lineitem from 15.5 minutes to 3.3 s.
• Spill files are compressed with lz4 by default (feat: add a spill compression codec config, defaulting to lz4 #6698, spark.comet.exec.spill.compression.codec). The PR reports TPC-H q21 going from 380 s to 250 s, against 367 s for Spark.
• Executor memory usage is written to the Spark event log (feat: write the executor memory usage log to the Spark event log #6401, feat: report JVM Arrow memory in the executor memory usage log #6250, fix: count pool overcommit as untracked memory in the native memory usage log #6271).

5. More of the plan runs natively.

• On by default: existence joins (feat: wire native existence join support #4587, spark.comet.exec.existenceJoin.enabled), conversion of OneRowRelation (feat: convert OneRowRelation to Arrow by default #6666), global aggregates with no grouping keys (feat: run global aggregates with no grouping keys or functions natively #6899), min and max over string and binary columns (feat: run min and max over string and binary columns natively #6927), and broadcast joins whose build side holds an unsupported scan (feat: run broadcast joins natively when an unsupported scan is on the build side  #6786).
• Supported: SortAggregateExec (feat: support SortAggregateExec #4565), distinct-combined collect_list (feat: add native distinct-combined collect_list aggregate support #4727), HLL sketch functions (feat: support Spark HyperLogLog sketch functions (hll_sketch_agg, hll_union_agg, hll_sketch_estimate, hll_union) #4802), PivotFirst (feat: support PivotFirst aggregate for the optimized PIVOT fast path #4798), AtLeastNNonNulls (feat: support AtLeastNNonNulls natively #6180), replace for safe inputs (feat: enable native replace for safe inputs #6680), native hashing of wide decimals (feat: support native wide-decimal hashing #6548), RANGE window frames with an offset on a DATE key (feat: native RANGE window frames with explicit offset on DATE ORDER BY #4974), and interval multiply and divide through codegen dispatch (feat: support multiply_ym_interval with YearMonth interval codegen dispatch #4715, feat: support divide_dt_interval with codegen dispatch #4901).
• Opt-in: native RangeExec (feat: add a native Comet version of RangeExec #6553, spark.comet.exec.range.enabled), conversion of typed Dataset operations (feat: run the operators above a typed Dataset operation natively #6564), native shuffle over row input (feat: use native shuffle for row input by converting it to Arrow #6607), RDD, range, cache and Data Source V1 conversions, each with its own spark.comet.convert.* setting (feat: give each default Spark-to-Arrow conversion its own spark.comet.convert config #6602), and join filter pushdown through deterministic residuals (feat: opt in to join filter pushdown through deterministic residuals #6431).
• Experimental: custom Rust UDFs through Arrow FFI (feat: custom Rust UDFs via arrow-ffi [experimental] #4459, with a setting to disable or restrict library loading, feat: add config to disable or restrict native UDF library loading, and load libraries without a global lock #6775), and scalar Arrow Python UDFs run natively (feat: execute scalar Arrow Python UDFs in native Comet #6130, spark.comet.exec.nativeArrowPythonUDF.enabled, needs a native library built with the python-udf Cargo feature). The PR reports 3.58x faster than Spark on a pyarrow.compute.negate query and 3.44x slower on a Python-loop UDF, so the post should not oversell it.
• New contrib module: a native Delta Lake scan with deletion vectors and page and row-group pruning (feat: add native Delta Lake scan contrib module (page/row-group pruning) #5365). It is opt-in: a separate -Pdelta jar and spark.comet.scan.delta.enabled, default false. Part of Explore integration with Delta Lake #174.

6. Performance.

• Native CASE WHEN and IF up to 12x faster (perf: optimize native CASE WHEN and IF (up to 12x faster) #6350), string-to-timestamp up to 5.6x faster (perf: classify string-to-timestamp shapes in one byte scan (up to 5.6x faster) #5130), and integer, decimal, float and boolean casts to timestamp vectorized (perf: vectorize int, decimal, float and boolean casts to timestamp #6969, for example i64 20.2 to 2.3 µs per 8192 rows).
• Spark-to-Arrow conversion: bulk column writes (perf: write Spark columns to Arrow in bulk in CometSparkToColumnarExec #6566), unsafe rows converted straight from their memory (perf: convert Spark's unsafe rows to Arrow straight from their memory #6730), JVM columnar shuffle for strings, maps and arrays (perf: speed up string, map and array conversion in JVM columnar shuffle #6604), and a direct columnar-to-row path (perf: optimize JVM columnar-to-row conversion #5496, up to 1.93x faster on decimal-heavy schemas, but 29% slower on a plain long, int, double, string schema).
• Shuffle: offsets stored as lengths, so zstd shuffle files with string, binary, array and map columns are 9% to 40% smaller (perf: store shuffle offsets as lengths with zstd #6943); a whole partition's spill written in one native call, 21% to 28% faster for 1,000 partitions and 45% to 49% faster when spilling (perf: write every partition of a JVM shuffle spill in one native call #7065); spill pointers sorted in scratch space (perf: sort JVM shuffle spill pointers in the sorter's scratch space #7125); blocks read in 64 KiB pieces (perf: read shuffle blocks in 64 KiB pieces instead of through Channels.newChannel #6805).
• Broadcast: one decoded copy shared by the tasks of an executor (perf: share one decoded copy of a broadcast among the tasks of an executor #7016). The PR reports a 1M-row build with 128 tasks on local[8] going from 1,321 ms to 684 ms in its first column.
• Joins and scans: filtering decoded probe batches before intermediate joins (feat: filter decoded probe batches before intermediate joins #6433), the next Parquet file's footer and page index loaded while the current file is read (perf: load the next Parquet file's footer and page index while the current file is read #7094, up to 1.9x on high-latency storage in a synthetic test), and opt-in TopK threshold pushdown into Parquet readers (feat: add opt-in local TopK fusion for native Parquet scans #6181, feat: push local TopK thresholds into Parquet readers #6263).
• Benchmarks for the remaining native kernels (chore: add benches remaining scalars #6541, chore: add benches for remaining agg functions #6647, chore: add benches for remaining comet native kernels #6648), window operators (test: add window operator benchmarks #7139) and a nested TPC-H variant (feat: add nested benchmark derived from TPC-H #7044).

7. Platform and project.

• Spark 4.2.0 is a supported version, and the Spark SQL tests run against the released 4.2.0 (build: enable Spark SQL tests for released Spark 4.2.0 #4950, docs: promote Spark 4.2 from experimental to supported #6417).
• DataFusion 55.2.0 (deps: bump to datafusion 55.2.0 #6766). iceberg-rust is still a pinned git revision, as in 1.1.0, but a newer one (feat: bump iceberg-rust and write float and double identity partitions natively #6851).
• The native code is split into focused crates (refactor: separate JNI-free core logic and error classification at the native boundary #6435, refactor: move the remaining PhysicalExpr impls from core into spark-expr #6450, refactor: move the self-contained operators from core into a new operators crate #6503, refactor: move the remaining JNI-free entry point logic into core functions #6534), and the planner into dedicated modules (refactor: move MergeRows, Explode, aggregate, join and window planning into dedicated modules #7014).
• Packaging: comet-common, the parent pom and comet-spark-integration are no longer published (build: stop publishing comet-common, the parent pom and comet-spark-integration #6888), release jars are built from the same commit as the native libraries (fix: build release jars from the same commit as the native libraries #6377), bundled native libraries are extracted into private directories (fix: extract bundled native libraries into private directories #6987), and native panics are logged with a backtrace instead of attached to the exception (fix: log native panic backtraces instead of attaching them to the exception #7042).
• Release hygiene: a license check on the release source tarball (ci: check licenses on release source tarball #6639) and NOTICE shipped in the jars (fix: ship NOTICE.txt as META-INF/NOTICE in the comet-spark jar and sources jar #6657).
• Project: the proposal to establish Apache Comet as a top-level project (docs: add proposal to establish Apache Comet as a top-level project #6817, merged; the vote is the next step).
• Docs: an FAQ and a refreshed Gluten comparison (docs: add an FAQ page with an answer on vectorized Java/Scala UDFs #6778, docs: expand the FAQ and refresh the Gluten comparison #6783), a Celeborn guide (docs: add a Celeborn integration guide #6373), a timezone page (docs: add contributor guide page on timezone handling #6337, docs: update the timezone page for the fixes from the timezone EPIC #6686), Parquet and Iceberg documentation consolidated into one page each with known issues listed (docs: consolidate Parquet and Iceberg documentation into one page each #7090, docs: list known Parquet and Iceberg issues and add a data source audit log #7052), and TPC-DS results for 1.1.0 on Spark 4.2 (docs: update TPC-DS benchmark results for Comet 1.1.0 on Spark 4.2 #6308).

Not worth a headline: Spark 4.1 TIME type support through codegen dispatch (#4821). Spark keeps the type behind an internal flag, spark.sql.timeType.enabled, that defaults to off outside tests in 4.1.1, 4.1.3 and 4.2.0.

Changes since the 2026-10-12 snapshot

• Native Iceberg writes: feat: enable native Iceberg writes by default #6664 merged and feat: make native Iceberg writes opt-in for 1.2.0 #6962 made them opt-in again before any release. Relative to 1.1.0 nothing changed: they are opt-in in both, so they are not a "now on by default" item for 1.2.0.
• In-memory cache: opt-in in 1.1.0 and 1.2.0 (fix: make Comet's in-memory cache opt-in again #6963 undid the default from feat: enable Comet's in-memory cache by default #5634 within the 1.2 cycle). The earlier "on by default" headline is gone.
• Native Delta Lake scan (feat: add native Delta Lake scan contrib module (page/row-group pruning) #5365) merged, as an opt-in contrib module.
• Arrow Python UDFs (feat: execute scalar Arrow Python UDFs in native Comet #6130), global aggregates with no keys (feat: run global aggregates with no grouping keys or functions natively #6899), min/max on strings (feat: run min and max over string and binary columns natively #6927) and the shared broadcast decode (perf: share one decoded copy of a broadcast among the tasks of an executor #7016) are new.

Open PRs worth watching before the cut (as of 2026-10-23)

• fix: native windows reserve the rows they buffer and buffer one window partition at a time #6279: native windows reserve the rows they buffer and buffer one window partition at a time.
• fix: make fair_unified's per-consumer limit bind when tasks run concurrently #7096: make fair_unified's per-consumer limit bind when tasks run concurrently.
• fix: keep Text build sides on Spark by default and add Text conversion configs #7123: keep Text build sides on Spark by default and add Text conversion configs.
• test: move the float semantics sweep to SQL file tests #7114 and fix: fall back to Spark for mode on FLOAT and DOUBLE before Spark 3.5.2 #7115: the remaining float EPIC follow-ups (test: move CometFloatSemanticsSuite's matching cases to Comet SQL tests #6636, Native mode treats all NaNs as one value on Spark 3.4, where Spark counts each NaN separately #6738).