diff --git a/docs/source/changelog/1.2.0.md b/docs/source/changelog/1.2.0.md new file mode 100644 index 00000000000..e15e7a6ae7d --- /dev/null +++ b/docs/source/changelog/1.2.0.md @@ -0,0 +1,482 @@ + + +# DataFusion Comet 1.2.0 Changelog + +This release consists of 396 commits from 40 contributors. See credits at the end of this changelog for more information. + +**Fixed bugs:** + +- fix: reject a file without field ids at any depth whether or not id matching is on [#6116](https://github.com/apache/datafusion-comet/pull/6116) (dwsmith1983) +- fix: decline native Iceberg writes with a custom location provider [#6216](https://github.com/apache/datafusion-comet/pull/6216) (liupoyi-1031) +- fix: report task input metrics after the native iterator closes and add to Spark's counters [#5880](https://github.com/apache/datafusion-comet/pull/5880) (dwsmith1983) +- fix: Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm [#6025](https://github.com/apache/datafusion-comet/pull/6025) (parthchandra) +- fix: log partial memory grants at DEBUG and drop the memory usage dump [#6269](https://github.com/apache/datafusion-comet/pull/6269) (andygrove) +- fix: prevent libhdfs thread destructor use-after-free [#5036](https://github.com/apache/datafusion-comet/pull/5036) (peterxcli) +- fix: return a native plan's memory before its Spark task ends [#6261](https://github.com/apache/datafusion-comet/pull/6261) (andygrove) +- fix: native ansi-errors raised as spark error [#5169](https://github.com/apache/datafusion-comet/pull/5169) (peterxcli) +- fix: credit co-authors of a PR in the generated changelog [#6358](https://github.com/apache/datafusion-comet/pull/6358) (andygrove) +- fix: validate map constructor row lengths and null short-circuiting [#5846](https://github.com/apache/datafusion-comet/pull/5846) (sunchao) +- fix: match Spark ordering in native array extrema [#5403](https://github.com/apache/datafusion-comet/pull/5403) (sunchao) +- fix: label native timestamp_seconds results as UTC timestamps [#6341](https://github.com/apache/datafusion-comet/pull/6341) (andygrove) +- fix: keep the input's timezone label on native date_trunc results [#6345](https://github.com/apache/datafusion-comet/pull/6345) (andygrove) +- fix: validate spark.comet.shuffle.jvm.batchSize and spark.comet.exec.memoryPool [#6287](https://github.com/apache/datafusion-comet/pull/6287) (andygrove) +- fix: reject non-default collations in all sort keys [#6206](https://github.com/apache/datafusion-comet/pull/6206) (LinSimon-901101) +- fix: zero sliced boolean offsets at every level before exporting to the JVM [#6339](https://github.com/apache/datafusion-comet/pull/6339) (andygrove) +- fix: count pool overcommit as untracked memory in the native memory usage log [#6271](https://github.com/apache/datafusion-comet/pull/6271) (andygrove) +- fix: install Comet's cache serializer only when Comet and native execution are enabled [#6360](https://github.com/apache/datafusion-comet/pull/6360) (andygrove) +- fix: fall back from the native CSV scan for timestamps outside UTC [#6349](https://github.com/apache/datafusion-comet/pull/6349) (andygrove) +- fix: key each scan's planning data by its plan node in a native block [#6297](https://github.com/apache/datafusion-comet/pull/6297) (dwsmith1983) +- fix: derive the native memory limit the way YARN and Kubernetes size executor containers [#6375](https://github.com/apache/datafusion-comet/pull/6375) (andygrove) +- fix: route nested DATE to numeric casts in structs and maps through the codegen dispatcher [#6319](https://github.com/apache/datafusion-comet/pull/6319) (andygrove) +- fix: match iceberg-java for Iceberg partitions past year 262142 in the native writer [#6239](https://github.com/apache/datafusion-comet/pull/6239) (andygrove) +- fix: count each reused exchange once in extended explain [#6379](https://github.com/apache/datafusion-comet/pull/6379) (andygrove) +- fix: normalize session timezone IDs before passing them to native code [#6351](https://github.com/apache/datafusion-comet/pull/6351) (andygrove) +- fix: count every hash-based JVM shuffle batch as shuffle bytes written [#6369](https://github.com/apache/datafusion-comet/pull/6369) (andygrove) +- fix: release input Arrow streams that native never takes [#6372](https://github.com/apache/datafusion-comet/pull/6372) (andygrove) +- fix: make benchmark cases over ShortType columns exercise Comet [#6376](https://github.com/apache/datafusion-comet/pull/6376) (andygrove) +- fix: fail a native plan whose producer task is cancelled before its stream ends [#6362](https://github.com/apache/datafusion-comet/pull/6362) (andygrove) +- fix: relabel CASE and COALESCE timestamp branches instead of panicking [#6347](https://github.com/apache/datafusion-comet/pull/6347) (andygrove) +- fix: warn about unreleased native memory only after a task's last native plan closes [#6374](https://github.com/apache/datafusion-comet/pull/6374) (andygrove) +- fix: cap spark.comet.shuffle.jvm.batchSize at spark.comet.batchSize where it is read [#6366](https://github.com/apache/datafusion-comet/pull/6366) (andygrove) +- fix: warn when native and JVM timezone databases differ [#6353](https://github.com/apache/datafusion-comet/pull/6353) (andygrove) +- fix: describe CometInMemoryTableScan by the scan it replaces in EXPLAIN [#6453](https://github.com/apache/datafusion-comet/pull/6453) (andygrove) +- fix: remove nested planner placeholders after recursive conversion [#6439](https://github.com/apache/datafusion-comet/pull/6439) (sunchao) +- fix: fall back to Spark for RANK and DENSE_RANK limits over nested floating-point keys [#6468](https://github.com/apache/datafusion-comet/pull/6468) (andygrove) +- fix: make adaptive partial aggregation opt-in with spark.comet.exec.aggregate.skipPartial.enabled [#6474](https://github.com/apache/datafusion-comet/pull/6474) (andygrove) +- fix: fall back when struct field names collide case-insensitively [#6326](https://github.com/apache/datafusion-comet/pull/6326) (comphead) +- fix: reject casts involving non-default collated strings [#5302](https://github.com/apache/datafusion-comet/pull/5302) (stantheman0128) +- fix: `get_json_object` returns first value for duplicate keys to match Spark [#4971](https://github.com/apache/datafusion-comet/pull/4971) (u70b3) +- fix: build with Rust 1.99, which deprecates the legacy f64 constants and fetch_update [#6507](https://github.com/apache/datafusion-comet/pull/6507) (andygrove) +- fix: fall back to Spark for \_metadata.file_block_start and file_block_length [#6510](https://github.com/apache/datafusion-comet/pull/6510) (andygrove) +- fix: report a cached relation's decoded size to the planner, not its compressed size [#6412](https://github.com/apache/datafusion-comet/pull/6412) (andygrove) +- fix: Route collated instr, substring_index, trim, greatest and least through the codegen dispatcher [#6480](https://github.com/apache/datafusion-comet/pull/6480) (viirya) +- fix: keep Spark's cache scan for a relation whose cached plan records observed metrics [#6421](https://github.com/apache/datafusion-comet/pull/6421) (andygrove) +- fix: load the bundled native library only once per class loader [#6493](https://github.com/apache/datafusion-comet/pull/6493) (zhangfengcdt) +- fix: match Iceberg's rounding for pre-1970 timestamps in native years/months/days/hours [#6456](https://github.com/apache/datafusion-comet/pull/6456) (andygrove) +- fix: cast native IF branches to a common type like CASE WHEN [#6458](https://github.com/apache/datafusion-comet/pull/6458) (andygrove) +- fix: fall back to Spark for regr\_\* aggregates until their merge matches Spark [#6451](https://github.com/apache/datafusion-comet/pull/6451) (andygrove) +- fix: defer Parquet conversion errors until a row group is decoded, as Spark does [#6515](https://github.com/apache/datafusion-comet/pull/6515) (andygrove) +- fix: publish metrics on the update interval for native blocks without a JVM input [#6314](https://github.com/apache/datafusion-comet/pull/6314) (andygrove) +- fix: match Spark's whitespace trimming in casts from string to timestamp and timestamp_ntz [#6445](https://github.com/apache/datafusion-comet/pull/6445) (andygrove) +- fix: run reverse of a binary value through the codegen dispatcher [#6461](https://github.com/apache/datafusion-comet/pull/6461) (andygrove) +- fix: match Spark's float semantics in array_remove and sort_array [#6518](https://github.com/apache/datafusion-comet/pull/6518) (andygrove) +- fix: fall back from the native Iceberg scan when a nested field was added or renamed [#6543](https://github.com/apache/datafusion-comet/pull/6543) (andygrove) +- fix: dispatch StaticInvoke and Invoke only into Spark's own classes [#6542](https://github.com/apache/datafusion-comet/pull/6542) (andygrove) +- fix: fall back for Iceberg writes with float or double identity partitions [#6531](https://github.com/apache/datafusion-comet/pull/6531) (zhangfengcdt) +- fix: match Spark's float ordering for sort and window keys nested in arrays and structs [#6475](https://github.com/apache/datafusion-comet/pull/6475) (andygrove) +- fix: check accelerated mapInArrow output against the declared schema [#6365](https://github.com/apache/datafusion-comet/pull/6365) (andygrove) +- fix: gate array distinct and union signed-zero semantics by Spark version [#5750](https://github.com/apache/datafusion-comet/pull/5750) (peterxcli) +- fix: hash NaN the way Spark does in hash, xxhash64 and native shuffle [#6413](https://github.com/apache/datafusion-comet/pull/6413) (andygrove) +- fix: return -0.0 for signum(-0.0), as Spark does [#6556](https://github.com/apache/datafusion-comet/pull/6556) (zhangfengcdt) +- fix: validate release build ref before building [#6571](https://github.com/apache/datafusion-comet/pull/6571) (andygrove) +- fix: draw the cached plan below CometInMemoryTableScan in EXPLAIN [#6574](https://github.com/apache/datafusion-comet/pull/6574) (comphead) +- fix: stop the leak of Arrow memory from struct outputs of the codegen dispatcher [#6552](https://github.com/apache/datafusion-comet/pull/6552) (peterxcli) +- fix: keep Spark's cache format when Kryo would reject Comet's or Comet disables itself [#6537](https://github.com/apache/datafusion-comet/pull/6537) (andygrove) +- fix: add native metrics from every plan a task runs instead of keeping the last [#6416](https://github.com/apache/datafusion-comet/pull/6416) (dwsmith1983) +- fix: keep a folded single-entry map literal with a null key on Spark [#6585](https://github.com/apache/datafusion-comet/pull/6585) (Smallfu666) +- fix: match Spark's float ordering in min, max, greatest and least [#6457](https://github.com/apache/datafusion-comet/pull/6457) (andygrove) +- fix: convert native ReadAncientDatetime errors to SparkUpgradeException [#6559](https://github.com/apache/datafusion-comet/pull/6559) (dwsmith1983) +- fix: choose get_json_object number-limit behavior from the runtime Jackson version [#6600](https://github.com/apache/datafusion-comet/pull/6600) (andygrove) +- fix: align wide-decimal ANSI overflow value with Spark [#5276](https://github.com/apache/datafusion-comet/pull/5276) (sam-1112) +- fix: draw the cached plan below CometInMemoryTableScan in the SQL tab and event log [#6577](https://github.com/apache/datafusion-comet/pull/6577) (comphead) +- fix: preserve DPP filters during transition revert [#6137](https://github.com/apache/datafusion-comet/pull/6137) (sam-1112) +- fix: close Iceberg cleanup ownership gap during JVM handoff [#6582](https://github.com/apache/datafusion-comet/pull/6582) (unikdahal) +- fix(iceberg): preserve Spark 4.2 write transactions [#6587](https://github.com/apache/datafusion-comet/pull/6587) (unikdahal) +- fix: read the Iceberg write gate's storage scheme the same way the native factory does [#6502](https://github.com/apache/datafusion-comet/pull/6502) (0lai0) +- fix: keep parquet view types out of the shuffle benchmark input [#5804](https://github.com/apache/datafusion-comet/pull/5804) (peterxcli) +- fix: make the native Iceberg per-I/O timeout configurable [#6649](https://github.com/apache/datafusion-comet/pull/6649) (comphead) +- fix: refresh stale shuffle scans when AQE prepares a stage, not while it re-plans [#6646](https://github.com/apache/datafusion-comet/pull/6646) (andygrove) +- fix: compact already-sliced batches before the native Iceberg writer counts NaNs [#6238](https://github.com/apache/datafusion-comet/pull/6238) (andygrove) +- fix: ship NOTICE.txt as META-INF/NOTICE in the comet-spark jar and sources jar [#6657](https://github.com/apache/datafusion-comet/pull/6657) (andygrove) +- fix: match Spark ANSI integral overflow errors (#6217) [#6249](https://github.com/apache/datafusion-comet/pull/6249) (LinSimon-901101) +- fix: fall back for RANGE frames over nested keys that can hold a null [#6684](https://github.com/apache/datafusion-comet/pull/6684) (andygrove) +- fix: decline native Iceberg writes through a void partition field whose source column was dropped [#6677](https://github.com/apache/datafusion-comet/pull/6677) (andygrove) +- fix: let a spilled final aggregate read its spill files back past its memory share [#6544](https://github.com/apache/datafusion-comet/pull/6544) (andygrove) +- fix: drop values under NULL parents before the native Iceberg writer counts NaNs [#6658](https://github.com/apache/datafusion-comet/pull/6658) (andygrove) +- fix: have native shuffle read a converted child's Arrow stream [#6689](https://github.com/apache/datafusion-comet/pull/6689) (andygrove) +- fix: coalesce shuffle partitions under Comet unions the way Spark does [#6459](https://github.com/apache/datafusion-comet/pull/6459) (andygrove) +- fix: fall back for nested sort keys whose null elements Spark orders differently [#6683](https://github.com/apache/datafusion-comet/pull/6683) (andygrove) +- fix: gate unsupported Iceberg S3 write settings [#6441](https://github.com/apache/datafusion-comet/pull/6441) (sam-1112) +- fix: rethrow the exception a JVM input throws instead of a CometNativeException [#6243](https://github.com/apache/datafusion-comet/pull/6243) (andygrove) +- fix: initialize dispatched kernels with the partition the native plan computes [#6700](https://github.com/apache/datafusion-comet/pull/6700) (comphead) +- fix: restore Spark write execs when reverting transition-heavy stages [#5957](https://github.com/apache/datafusion-comet/pull/5957) (sam-1112) +- fix: count the days partition transform in UTC [#6348](https://github.com/apache/datafusion-comet/pull/6348) (andygrove) +- fix: normalize float operands in the native comparison builder [#6447](https://github.com/apache/datafusion-comet/pull/6447) (andygrove) +- fix: keep a Spark scan unconverted when the plan reads input_file_name [#6703](https://github.com/apache/datafusion-comet/pull/6703) (kazuyukitanimura) +- fix: load the Iceberg storage scheme lists from the native factory [#6065](https://github.com/apache/datafusion-comet/pull/6065) (dwsmith1983) +- fix: Route collated array element comparisons through the codegen dispatcher [#6471](https://github.com/apache/datafusion-comet/pull/6471) (viirya) +- fix: match Spark 4.2 date_trunc overflow for SECOND and MILLISECOND [#6740](https://github.com/apache/datafusion-comet/pull/6740) (andygrove) +- fix: make Iceberg delete-file reflection failures fatal [#5515](https://github.com/apache/datafusion-comet/pull/5515) (unikdahal) +- fix: honor Spark’s commit protocol in Spark 3.x native writes [#4746](https://github.com/apache/datafusion-comet/pull/4746) (peterxcli, andygrove) +- fix: retry a shuffle page allocation that Spark failed after dropping the task's entry [#6403](https://github.com/apache/datafusion-comet/pull/6403) (dwsmith1983) +- fix: truncate per-row date_trunc formats with the literal-format kernel [#6354](https://github.com/apache/datafusion-comet/pull/6354) (andygrove) +- fix: don't let S3A committer, delete and read defaults block native Iceberg writes [#6808](https://github.com/apache/datafusion-comet/pull/6808) (andygrove) +- fix: run InsertIntoHadoopFsRelationCommand's post-write steps in the Spark 3.x native writer [#6821](https://github.com/apache/datafusion-comet/pull/6821) (viirya) +- fix: retry a native acquire that Spark failed after dropping the task's entry [#6310](https://github.com/apache/datafusion-comet/pull/6310) (dwsmith1983) +- fix: check native UDF types at registration and explain planning-time evaluation [#6777](https://github.com/apache/datafusion-comet/pull/6777) (andygrove) +- fix: match Spark percentile ordering for negative NaNs [#6829](https://github.com/apache/datafusion-comet/pull/6829) (rich7420) +- fix: resolve Azure ABFS authentication through Hadoop and keep ambient credentials out [#6059](https://github.com/apache/datafusion-comet/pull/6059) (dwsmith1983) +- fix: fall back from native V2 scans when the plan reads input_file_name [#6754](https://github.com/apache/datafusion-comet/pull/6754) (zhangfengcdt) +- fix: accept all BMP Unicode decimal digits in cast(string as decimal) [#6841](https://github.com/apache/datafusion-comet/pull/6841) (viirya) +- fix: preserve CASE WHEN struct field names from the first THEN branch [#6827](https://github.com/apache/datafusion-comet/pull/6827) (pingzh) +- fix: keep one row per key in a native PartialMerge that runs short of memory [#6867](https://github.com/apache/datafusion-comet/pull/6867) (andygrove) +- fix: use Spark-compatible shuffle for wide decimal hash keys [#6005](https://github.com/apache/datafusion-comet/pull/6005) (sunchao, andygrove) +- fix: [spark34] preserve empty Iceberg partitions after dynamic pruning [#6814](https://github.com/apache/datafusion-comet/pull/6814) (LinSimon-901101) +- fix: restore localReads assertion in AdaptiveQueryExecSuite diffs [#6382](https://github.com/apache/datafusion-comet/pull/6382) (mizulun) +- fix: do not resolve DPP subqueries when AQE reads a native scan's partitioning [#6268](https://github.com/apache/datafusion-comet/pull/6268) (dwsmith1983) +- fix: skip an array or map function's later arguments where the array or map is NULL [#6716](https://github.com/apache/datafusion-comet/pull/6716) (andygrove) +- fix: share the missing task entry retry and stop retrying it after the task has ended [#6813](https://github.com/apache/datafusion-comet/pull/6813) (viirya) +- fix: return an empty array from CASE WHEN for an empty batch [#6763](https://github.com/apache/datafusion-comet/pull/6763) (viirya) +- fix: compare greatest and least struct arguments by position [#6760](https://github.com/apache/datafusion-comet/pull/6760) (viirya) +- fix: order nulls inside nested values first in least [#6761](https://github.com/apache/datafusion-comet/pull/6761) (viirya) +- fix: handle truncation operands in scalar evaluation and rewrites [#6831](https://github.com/apache/datafusion-comet/pull/6831) (nat-openai) +- fix: match Spark statistical aggregate updates and merges [#6076](https://github.com/apache/datafusion-comet/pull/6076) (rich7420) +- fix: make Comet's in-memory cache opt-in again [#6963](https://github.com/apache/datafusion-comet/pull/6963) (andygrove) +- fix: keep the values of sliced batches when coalescing a broadcast [#6900](https://github.com/apache/datafusion-comet/pull/6900) (comphead) +- fix: compact oversized nested list_extract buffers [#6498](https://github.com/apache/datafusion-comet/pull/6498) (rich7420) +- fix: raise DIVIDE_BY_ZERO from native corr on a zero denominator in ANSI mode [#6830](https://github.com/apache/datafusion-comet/pull/6830) (kazuyukitanimura, rich7420) +- fix: report row contents to native write statistics trackers [#6383](https://github.com/apache/datafusion-comet/pull/6383) (LinSimon-901101) +- fix: fall back to Spark for decimal AVG cases that return wrong results [#7017](https://github.com/apache/datafusion-comet/pull/7017) (andygrove) +- fix: do not fail planning on a cast of a literal that fails to evaluate [#6826](https://github.com/apache/datafusion-comet/pull/6826) (viirya) +- fix: read each native scan partition through the object store its files live in [#6753](https://github.com/apache/datafusion-comet/pull/6753) (dwsmith1983) +- fix: compare a Remainder divisor with zero in Spark's ordering without ANSI [#6896](https://github.com/apache/datafusion-comet/pull/6896) (andygrove) +- fix: include try_add suggestion in ANSI SUM overflow error [#6732](https://github.com/apache/datafusion-comet/pull/6732) (DeviousCardi) +- fix: give each nondeterministic dispatched expression its own kernel [#6712](https://github.com/apache/datafusion-comet/pull/6712) (dwsmith1983) +- fix: honor subexpression elimination off in the native null short-circuit [#6986](https://github.com/apache/datafusion-comet/pull/6986) (dwsmith1983) +- fix: read fs.comet.libhdfs.schemes for the scan rule from the Hadoop conf [#6838](https://github.com/apache/datafusion-comet/pull/6838) (dwsmith1983) +- fix: keep unconverted DPP filters in a scan's canonical form [#6270](https://github.com/apache/datafusion-comet/pull/6270) (dwsmith1983) +- fix: fall back to Spark for NOT IN over an ordered streamed side [#6810](https://github.com/apache/datafusion-comet/pull/6810) (dwsmith1983) +- fix: stop building maps natively when the key type holds a FLOAT or DOUBLE [#6992](https://github.com/apache/datafusion-comet/pull/6992) (andygrove) +- fix: return errors instead of panicking in native planner [#7087](https://github.com/apache/datafusion-comet/pull/7087) (0lai0) +- fix: handle NullType struct fields in JVM columnar shuffle [#7099](https://github.com/apache/datafusion-comet/pull/7099) (jadenfix) +- fix: size the native runtime for standalone executors without spark.executor.cores [#6368](https://github.com/apache/datafusion-comet/pull/6368) (andygrove) +- fix: fall back to Spark when Iceberg reports an output ordering [#7043](https://github.com/apache/datafusion-comet/pull/7043) (parthchandra, andygrove) +- fix: build release jars from the same commit as the native libraries [#6377](https://github.com/apache/datafusion-comet/pull/6377) (andygrove) +- fix: count grouped integer SUM state in the aggregate's memory reservation [#6370](https://github.com/apache/datafusion-comet/pull/6370) (andygrove) +- fix: preserve Parquet root field IDs after nested pruning [#7081](https://github.com/apache/datafusion-comet/pull/7081) (LinSimon-901101) +- fix: serialize large-offset string and binary vectors with 32-bit offsets [#6371](https://github.com/apache/datafusion-comet/pull/6371) (andygrove) +- fix: keep a streamed side sorted outside the native plan in order through outer hash joins [#6835](https://github.com/apache/datafusion-comet/pull/6835) (andygrove) +- fix: treat -0.0 and 0.0 as one value in arrays_overlap on Spark 4.0.5, 4.1.4 and 4.2.1 [#6989](https://github.com/apache/datafusion-comet/pull/6989) (andygrove) +- fix: preserve Spark nested null ordering in native sort [#6968](https://github.com/apache/datafusion-comet/pull/6968) (Weijun-H) +- fix: fall back when an Iceberg equality delete is keyed on a nested field [#7122](https://github.com/apache/datafusion-comet/pull/7122) (yunchipang) +- fix: fall back for duplicate from_json fields [#6651](https://github.com/apache/datafusion-comet/pull/6651) (Samearth17) +- fix: evaluate the array_append item on rows whose array is NULL [#7025](https://github.com/apache/datafusion-comet/pull/7025) (kazuyukitanimura) +- fix: fall back to Spark for negative-scale decimal casts and arithmetic [#5050](https://github.com/apache/datafusion-comet/pull/5050) (0lai0, andygrove) +- fix: extract bundled native libraries into private directories [#6987](https://github.com/apache/datafusion-comet/pull/6987) (manuzhang) +- fix: preserve Spark AVG buffer and overflow semantics [#5420](https://github.com/apache/datafusion-comet/pull/5420) (sunchao) +- fix: log native panic backtraces instead of attaching them to the exception [#7042](https://github.com/apache/datafusion-comet/pull/7042) (viirya) +- fix: report output rows and time of the native window operator [#7140](https://github.com/apache/datafusion-comet/pull/7140) (comphead) +- fix: [branch-1.2] keep Text build sides on Spark by default and add Text conversion configs (#7123) [#7205](https://github.com/apache/datafusion-comet/pull/7205) (andygrove) +- fix: [branch-1.2] don't treat planner-inserted casts as Parquet schema conversions (#7159) [#7262](https://github.com/apache/datafusion-comet/pull/7262) (snmvaughan) +- fix: [branch-1.2] match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast (#7171) [#7263](https://github.com/apache/datafusion-comet/pull/7263) (snmvaughan) +- fix: [branch-1.2] fall back to Spark when the driver cannot use the ABFS authentication (#7221) [#7285](https://github.com/apache/datafusion-comet/pull/7285) (andygrove) +- fix: [branch-1.2] spill a native PartialMerge whose input is sorted on some of its keys (#7222) [#7286](https://github.com/apache/datafusion-comet/pull/7286) (andygrove) + +**Performance related:** + +- perf: optimize string contains function [#5322](https://github.com/apache/datafusion-comet/pull/5322) (kazantsev-maksim) +- perf: cache Iceberg FileIO per executor instead of building one per task [#6106](https://github.com/apache/datafusion-comet/pull/6106) (mixermt) +- perf: copy off-heap strings directly into Arrow [#6281](https://github.com/apache/datafusion-comet/pull/6281) (pingzh) +- perf: reuse Variant reconstruction buffers [#6342](https://github.com/apache/datafusion-comet/pull/6342) (peterxcli) +- perf: memoize Comet-subtree check in EliminateRedundantTransitions [#5213](https://github.com/apache/datafusion-comet/pull/5213) (andygrove) +- perf: optimize native `CASE WHEN` and `IF` (up to 12x faster) [#6350](https://github.com/apache/datafusion-comet/pull/6350) (andygrove) +- perf: optimize levenshtein function [#5042](https://github.com/apache/datafusion-comet/pull/5042) (kazantsev-maksim) +- perf: read the shuffle natively in operators AQE reuses from the initial plan [#6547](https://github.com/apache/datafusion-comet/pull/6547) (dwsmith1983) +- perf: classify string-to-timestamp shapes in one byte scan (up to 5.6x faster) [#5130](https://github.com/apache/datafusion-comet/pull/5130) (andygrove) +- perf: write Spark columns to Arrow in bulk in CometSparkToColumnarExec [#6566](https://github.com/apache/datafusion-comet/pull/6566) (andygrove) +- perf: prune unused filter output columns [#6396](https://github.com/apache/datafusion-comet/pull/6396) (rich7420) +- perf: speed up string, map and array conversion in JVM columnar shuffle [#6604](https://github.com/apache/datafusion-comet/pull/6604) (andygrove) +- test: cover IF branch selection and short-circuiting [#6397](https://github.com/apache/datafusion-comet/pull/6397) (rich7420) +- perf: fuse Comet cache vector reads into Spark codegen [#5859](https://github.com/apache/datafusion-comet/pull/5859) (peterxcli) +- perf: choose dictionary encoding like iceberg-java in the native Iceberg writer [#6241](https://github.com/apache/datafusion-comet/pull/6241) (andygrove) +- perf: skip Spark's encoders for boxed primitive Scala UDF values in the codegen dispatcher [#6714](https://github.com/apache/datafusion-comet/pull/6714) (andygrove) +- perf: key the codegen dispatcher's kernel cache on a digest of the expression [#6709](https://github.com/apache/datafusion-comet/pull/6709) (andygrove) +- perf: convert Spark's unsafe rows to Arrow straight from their memory [#6730](https://github.com/apache/datafusion-comet/pull/6730) (andygrove) +- perf: keep the sort-merge join when the forced hash join's build side is too large [#6384](https://github.com/apache/datafusion-comet/pull/6384) (dwsmith1983) +- perf: compare floats in Spark's order without normalizing copies of the operands [#6691](https://github.com/apache/datafusion-comet/pull/6691) (andygrove) +- perf: run Spark's null guard for a Scala UDF inside the codegen kernel [#6715](https://github.com/apache/datafusion-comet/pull/6715) (andygrove) +- perf: bulk-append long nullable primitive arrays in SparkUnsafeArray conversion [#6824](https://github.com/apache/datafusion-comet/pull/6824) (andygrove, sandugood) +- perf: skip DecimalPrecision.promote's rewrite when there is no decimal arithmetic [#5216](https://github.com/apache/datafusion-comet/pull/5216) (andygrove) +- perf: integral sequence optimisation [#6833](https://github.com/apache/datafusion-comet/pull/6833) (nat-openai) +- perf: generate the columnar-to-row projection once per executor instead of per partition [#6858](https://github.com/apache/datafusion-comet/pull/6858) (andygrove) +- perf: read shuffle blocks in 64 KiB pieces instead of through `Channels.newChannel` [#6805](https://github.com/apache/datafusion-comet/pull/6805) (comphead) +- perf: presize collections and reuse buffers in native loops [#6861](https://github.com/apache/datafusion-comet/pull/6861) (comphead) +- perf: read native-written Parquet footers without parsing Hadoop's configuration [#6923](https://github.com/apache/datafusion-comet/pull/6923) (andygrove) +- perf: avoid duplicate Hadoop configs in Iceberg write detection [#6925](https://github.com/apache/datafusion-comet/pull/6925) (andygrove) +- perf: generate the in-memory cache reader's class once per executor [#6860](https://github.com/apache/datafusion-comet/pull/6860) (andygrove) +- perf: store shuffle offsets as lengths with zstd [#6943](https://github.com/apache/datafusion-comet/pull/6943) (andygrove) +- perf: read only the nested fields Spark keeps in the native Iceberg scan [#6725](https://github.com/apache/datafusion-comet/pull/6725) (comphead) +- perf: make the native Iceberg writer's dictionary pre-pass 1.8x to 3.9x faster [#6937](https://github.com/apache/datafusion-comet/pull/6937) (andygrove) +- perf: reuse validity buffers and use unary kernels in native expressions [#6953](https://github.com/apache/datafusion-comet/pull/6953) (comphead) +- perf: optimize JVM columnar-to-row conversion [#5496](https://github.com/apache/datafusion-comet/pull/5496) (peterxcli) +- perf: start bypass shuffle partition writers on small pages that grow [#7059](https://github.com/apache/datafusion-comet/pull/7059) (andygrove) +- perf: write every partition of a JVM shuffle spill in one native call [#7065](https://github.com/apache/datafusion-comet/pull/7065) (andygrove) +- perf: create a native plan's spill directories on its first spill [#7066](https://github.com/apache/datafusion-comet/pull/7066) (andygrove) +- perf: vectorize int, decimal, float and boolean casts to timestamp [#6969](https://github.com/apache/datafusion-comet/pull/6969) (Ersa-tech) +- perf: sort JVM shuffle spill pointers in the sorter's scratch space [#7125](https://github.com/apache/datafusion-comet/pull/7125) (viirya) +- perf: load the next Parquet file's footer and page index while the current file is read [#7094](https://github.com/apache/datafusion-comet/pull/7094) (andygrove) +- perf: default the native Iceberg scan to two concurrent data files [#7132](https://github.com/apache/datafusion-comet/pull/7132) (andygrove) +- perf: share one decoded copy of a broadcast among the tasks of an executor [#7016](https://github.com/apache/datafusion-comet/pull/7016) (andygrove) + +**Implemented enhancements:** + +- test: report which writer ran each Iceberg write in the Iceberg CI jobs [#6155](https://github.com/apache/datafusion-comet/pull/6155) (andygrove) +- test: wait for Parquet write plan callbacks [#6108](https://github.com/apache/datafusion-comet/pull/6108) (rich7420) +- feat: add opt-in local TopK fusion for native Parquet scans [#6181](https://github.com/apache/datafusion-comet/pull/6181) (pingzh) +- feat: route timestamp_seconds decimal, byte and short input through codegen dispatch [#6204](https://github.com/apache/datafusion-comet/pull/6204) (0lai0) +- chore: add branch protection for branch-1.0 and branch-1.1 [#6267](https://github.com/apache/datafusion-comet/pull/6267) (andygrove) +- test: cover native Iceberg mid-write retry cleanup [#6111](https://github.com/apache/datafusion-comet/pull/6111) (sam-1112) +- feat: built-in S3 credential provider adapters for the native Parquet scan [#6023](https://github.com/apache/datafusion-comet/pull/6023) (parthchandra) +- feat: report JVM Arrow memory in the executor memory usage log [#6250](https://github.com/apache/datafusion-comet/pull/6250) (andygrove) +- test: cover matching nested nulls in collect aggregates [#5970](https://github.com/apache/datafusion-comet/pull/5970) (sunchao) +- test: fix flaky mixed field id directory test on Spark 3.4 and 3.5 [#6312](https://github.com/apache/datafusion-comet/pull/6312) (andygrove) +- feat: support Spark HyperLogLog sketch functions (hll_sketch_agg, hll_union_agg, hll_sketch_estimate, hll_union) [#4802](https://github.com/apache/datafusion-comet/pull/4802) (andygrove) +- build: enable Spark SQL tests for released Spark 4.2.0 [#4950](https://github.com/apache/datafusion-comet/pull/4950) (andygrove) +- feat: support divide_dt_interval with codegen dispatch [#4901](https://github.com/apache/datafusion-comet/pull/4901) (peterxcli) +- chore: start 1.2.0 development [#6311](https://github.com/apache/datafusion-comet/pull/6311) (andygrove) +- test: compare in-memory caches with independent answers [#6262](https://github.com/apache/datafusion-comet/pull/6262) (LinSimon-901101) +- test: cover native Iceberg write handoff cleanup [#6117](https://github.com/apache/datafusion-comet/pull/6117) (sam-1112) +- chore: Fix Scala code warnings [#5876](https://github.com/apache/datafusion-comet/pull/5876) (athlcode) +- test: backport the upstream fix for a flaky Iceberg 1.9.1 rewrite test [#6242](https://github.com/apache/datafusion-comet/pull/6242) (andygrove) +- ci: retry dependency downloads in the Spark SQL test jobs and the Iceberg Gradle wrapper [#6394](https://github.com/apache/datafusion-comet/pull/6394) (andygrove) +- ci: rebalance the Spark SQL test shards [#6393](https://github.com/apache/datafusion-comet/pull/6393) (andygrove) +- ci: raise the merge queue's build concurrency from 2 to 4 [#6391](https://github.com/apache/datafusion-comet/pull/6391) (andygrove) +- ci: stop main's cargo build cache from being evicted [#6392](https://github.com/apache/datafusion-comet/pull/6392) (andygrove) +- refactor: move native -0.0 and NaN handling into one float_semantics module [#6400](https://github.com/apache/datafusion-comet/pull/6400) (andygrove) +- build: bump the Spark 4.2 test diff to Comet 1.2.0-SNAPSHOT [#6398](https://github.com/apache/datafusion-comet/pull/6398) (andygrove) +- test: check Iceberg DPP pruning by planned file tasks [#6280](https://github.com/apache/datafusion-comet/pull/6280) (dwsmith1983) +- ci: keep a macOS native build cache on main [#6395](https://github.com/apache/datafusion-comet/pull/6395) (andygrove) +- test: cover integer bounds in cache pruning tests [#6381](https://github.com/apache/datafusion-comet/pull/6381) (LinSimon-901101) +- chore: remove AlignedArrowStreamReader and fix the Native to JVM FFI docs [#6364](https://github.com/apache/datafusion-comet/pull/6364) (andygrove) +- feat: push local TopK thresholds into Parquet readers [#6263](https://github.com/apache/datafusion-comet/pull/6263) (pingzh) +- feat: write the executor memory usage log to the Spark event log [#6401](https://github.com/apache/datafusion-comet/pull/6401) (comphead) +- refactor: separate JNI-free core logic and error classification at the native boundary [#6435](https://github.com/apache/datafusion-comet/pull/6435) (viirya) +- feat: admit string maps in Spark-to-Comet conversion [#6036](https://github.com/apache/datafusion-comet/pull/6036) (sunchao, jeffw13) +- test: cover Iceberg driver commit cleanup [#6119](https://github.com/apache/datafusion-comet/pull/6119) (sam-1112) +- refactor: move the remaining PhysicalExpr impls from core into spark-expr [#6450](https://github.com/apache/datafusion-comet/pull/6450) (andygrove) +- build: bump semanticdb to 4.13.10 so scalafix runs on Spark 4.1 and 4.2 [#6448](https://github.com/apache/datafusion-comet/pull/6448) (andygrove) +- test: cover booleans in native explode output past the first batch [#6473](https://github.com/apache/datafusion-comet/pull/6473) (andygrove) +- test: cover typed Dataset filters over sliced aggregate batches [#6497](https://github.com/apache/datafusion-comet/pull/6497) (rich7420) +- chore: move native write configs under spark.comet.write [#6494](https://github.com/apache/datafusion-comet/pull/6494) (zhangfengcdt) +- refactor: move the self-contained operators from core into a new operators crate [#6503](https://github.com/apache/datafusion-comet/pull/6503) (andygrove) +- refactor: move write planning into dedicated module [#6529](https://github.com/apache/datafusion-comet/pull/6529) (andygrove) +- feat: wire native existence join support [#4587](https://github.com/apache/datafusion-comet/pull/4587) (coderfender) +- test: sweep Spark's -0.0 and NaN semantics across operators and float expressions [#6521](https://github.com/apache/datafusion-comet/pull/6521) (andygrove) +- feat: reuse S3 credentials until shortly before their reported expiry [#6509](https://github.com/apache/datafusion-comet/pull/6509) (snmvaughan) +- test: run Comet in Spark SQL and Iceberg suites that build their own sessions [#6415](https://github.com/apache/datafusion-comet/pull/6415) (andygrove) +- feat: record why Spark scans a relation cached in Comet's format when Comet is off [#6538](https://github.com/apache/datafusion-comet/pull/6538) (andygrove) +- refactor: move shuffle planning into dedicated module [#6535](https://github.com/apache/datafusion-comet/pull/6535) (andygrove) +- ci: grant label runs the contents: write the docs job requests [#6409](https://github.com/apache/datafusion-comet/pull/6409) (andygrove) +- feat: per-location credentials for native Iceberg reads and writes [#6478](https://github.com/apache/datafusion-comet/pull/6478) (snmvaughan) +- feat: add a native Comet version of RangeExec [#6553](https://github.com/apache/datafusion-comet/pull/6553) (andygrove) +- feat: support multiply_ym_interval with YearMonth interval codegen dispatch [#4715](https://github.com/apache/datafusion-comet/pull/4715) (peterxcli) +- feat: add native distinct-combined collect_list aggregate support [#4727](https://github.com/apache/datafusion-comet/pull/4727) (peterxcli) +- test: add seeded fuzz coverage for PyArrow UDFs [#6074](https://github.com/apache/datafusion-comet/pull/6074) (LinSimon-901101) +- feat: add native support for MergeRowsExec (row-level MERGE INTO) [#5318](https://github.com/apache/datafusion-comet/pull/5318) (unikdahal) +- feat: give each default Spark-to-Arrow conversion its own spark.comet.convert config [#6602](https://github.com/apache/datafusion-comet/pull/6602) (andygrove) +- test: use the renamed native write flag in the Iceberg 1.11 WAP-branch suite [#6644](https://github.com/apache/datafusion-comet/pull/6644) (andygrove) +- feat: support Spark 4.1 TIME type and expressions via codegen dispatch [#4821](https://github.com/apache/datafusion-comet/pull/4821) (andygrove) +- refactor: move the remaining JNI-free entry point logic into core functions [#6534](https://github.com/apache/datafusion-comet/pull/6534) (viirya) +- feat: support native MergeSummary on Spark 4.1+ [#6640](https://github.com/apache/datafusion-comet/pull/6640) (unikdahal) +- ci: check licenses on release source tarball [#6639](https://github.com/apache/datafusion-comet/pull/6639) (andygrove) +- test: cover sequence's per-batch element limit and the batch size remedy [#6302](https://github.com/apache/datafusion-comet/pull/6302) (dwsmith1983) +- ci: drop non-existent suites from the Linux and macOS test matrices [#6099](https://github.com/apache/datafusion-comet/pull/6099) (ErikBPF) +- feat: run the operators above a typed Dataset operation natively [#6564](https://github.com/apache/datafusion-comet/pull/6564) (andygrove) +- chore: update the NOTICE copyright year to 2024-2026 [#6675](https://github.com/apache/datafusion-comet/pull/6675) (manuzhang) +- chore: Add contributor sentiment to contribute [#6688](https://github.com/apache/datafusion-comet/pull/6688) (comphead) +- feat: support SortAggregateExec [#4565](https://github.com/apache/datafusion-comet/pull/4565) (andygrove) +- feat: custom Rust UDFs via arrow-ffi [experimental] [#4459](https://github.com/apache/datafusion-comet/pull/4459) (andygrove) +- test: keep the Iceberg job-abort test's wait off Comet's Tokio workers [#6699](https://github.com/apache/datafusion-comet/pull/6699) (andygrove) +- chore: raise the native rust-version to 1.95, which iceberg-rust requires [#6690](https://github.com/apache/datafusion-comet/pull/6690) (manuzhang) +- feat: add a spark.comet.convert config for RowDataSourceScanExec [#6662](https://github.com/apache/datafusion-comet/pull/6662) (andygrove) +- feat: support native Iceberg writes to format-version 3 tables [#6672](https://github.com/apache/datafusion-comet/pull/6672) (andygrove) +- feat: convert OneRowRelation to Arrow by default [#6666](https://github.com/apache/datafusion-comet/pull/6666) (andygrove) +- feat: charge native write buffers to the task's memory pool [#6247](https://github.com/apache/datafusion-comet/pull/6247) (andygrove) +- test: move mixed-type shift coverage to SQL fixtures [#6652](https://github.com/apache/datafusion-comet/pull/6652) (rich7420) +- feat: Use DataFusion date_trunc for scalar-format timestamp truncation [#5956](https://github.com/apache/datafusion-comet/pull/5956) (sam-1112) +- chore: add benches for remaining comet native kernels [#6648](https://github.com/apache/datafusion-comet/pull/6648) (coderfender) +- chore: update documentation links for 1.1.0 release [#6736](https://github.com/apache/datafusion-comet/pull/6736) (andygrove) +- feat: add a spill compression codec config, defaulting to lz4 [#6698](https://github.com/apache/datafusion-comet/pull/6698) (comphead) +- feat(iceberg): plan WriteDelta in the split write path and keep it on the JVM DeltaWriter [#6693](https://github.com/apache/datafusion-comet/pull/6693) (unikdahal) +- feat: use native shuffle for row input by converting it to Arrow [#6607](https://github.com/apache/datafusion-comet/pull/6607) (andygrove) +- test: preserve regex escapes in SQL fixtures [#6745](https://github.com/apache/datafusion-comet/pull/6745) (rich7420) +- test: enable SPARK-57298 collect_set tests [#5228](https://github.com/apache/datafusion-comet/pull/5228) (peterxcli) +- test: add optimizer controls and config isolation to SQL fixtures [#6731](https://github.com/apache/datafusion-comet/pull/6731) (rich7420) +- chore: move CometCollationSuite to spark-4.1+ test shim [#4864](https://github.com/apache/datafusion-comet/pull/4864) (0lai0) +- test: drain the listener bus before registering Iceberg test listeners [#6809](https://github.com/apache/datafusion-comet/pull/6809) (viirya) +- feat: run nullable nested float sort keys natively in strict floating-point mode [#6735](https://github.com/apache/datafusion-comet/pull/6735) (andygrove) +- feat: enable Comet's in-memory cache by default [#5634](https://github.com/apache/datafusion-comet/pull/5634) (andygrove) +- feat: run array_contains on float elements natively with Spark's equa… [#6599](https://github.com/apache/datafusion-comet/pull/6599) (zhangfengcdt) +- refactor: move the plan entry point logic into core functions [#6695](https://github.com/apache/datafusion-comet/pull/6695) (viirya) +- refactor: remove unused cast compatibility flag [#6135](https://github.com/apache/datafusion-comet/pull/6135) (sam-1112, andygrove) +- test: move bit_count boundary coverage to SQL fixtures [#6655](https://github.com/apache/datafusion-comet/pull/6655) (rich7420) +- test: cover container field ids next to INT96 timestamps in the native Parquet scan [#6862](https://github.com/apache/datafusion-comet/pull/6862) (dwsmith1983) +- feat: filter decoded probe batches before intermediate joins [#6433](https://github.com/apache/datafusion-comet/pull/6433) (sunchao) +- test: count the partitions and batches the native cache scan reads [#6836](https://github.com/apache/datafusion-comet/pull/6836) (andygrove) +- feat: add lz4 as an opt-in codec for Comet's in-memory cache [#6859](https://github.com/apache/datafusion-comet/pull/6859) (andygrove) +- chore: re-enable unused-parameter warnings in strict profile [#6846](https://github.com/apache/datafusion-comet/pull/6846) (rich7420) +- test: migrate required-child hash regressions to SQL fixtures [#6864](https://github.com/apache/datafusion-comet/pull/6864) (rich7420) +- test: fail Comet tests when a native plan mislabels a timestamp's timezone [#6868](https://github.com/apache/datafusion-comet/pull/6868) (andygrove) +- test: migrate regex extraction and position coverage to SQL fixtures [#6882](https://github.com/apache/datafusion-comet/pull/6882) (rich7420) +- refactor: stop normalizing float divisors now that percentile_approx orders NaN as Spark does [#6866](https://github.com/apache/datafusion-comet/pull/6866) (andygrove) +- test: migrate scalar hash coverage to SQL fixtures [#6881](https://github.com/apache/datafusion-comet/pull/6881) (rich7420) +- test: migrate Java RLIKE semantics and batch coverage to SQL fixtures [#6863](https://github.com/apache/datafusion-comet/pull/6863) (rich7420) +- test: add SQL error-class parity checks [#6650](https://github.com/apache/datafusion-comet/pull/6650) (rich7420) +- test: run the S3 suites against an available MinIO image [#6887](https://github.com/apache/datafusion-comet/pull/6887) (dwsmith1983) +- build: stop publishing comet-common, the parent pom and comet-spark-integration [#6888](https://github.com/apache/datafusion-comet/pull/6888) (andygrove) +- feat: close fanout partitions early when the memory pool refuses the native Iceberg writer [#6773](https://github.com/apache/datafusion-comet/pull/6773) (andygrove) +- test: expect Spark 3.5's empty LocalTableScan in the single-split DPP join test [#6902](https://github.com/apache/datafusion-comet/pull/6902) (dwsmith1983) +- feat: add native Delta Lake scan contrib module (page/row-group pruning) [#5365](https://github.com/apache/datafusion-comet/pull/5365) (dwsmith1983, sunchao, andygrove) +- test: Deflake SQLAppStatusListenerMemoryLeakSuite [#6905](https://github.com/apache/datafusion-comet/pull/6905) (LinSimon-901101) +- feat: enable native Iceberg writes by default [#6664](https://github.com/apache/datafusion-comet/pull/6664) (andygrove) +- test: add an end-to-end Iceberg read, repartition, and write benchmark [#6910](https://github.com/apache/datafusion-comet/pull/6910) (andygrove) +- ci: retry the scalafix lint when Maven cannot resolve the plugin [#6061](https://github.com/apache/datafusion-comet/pull/6061) (andygrove) +- feat: run min and max over string and binary columns natively [#6927](https://github.com/apache/datafusion-comet/pull/6927) (comphead) +- build: drop the compiled maven-wrapper.jar by switching to the script-only wrapper [#6678](https://github.com/apache/datafusion-comet/pull/6678) (manuzhang) +- feat: run global aggregates with no grouping keys or functions natively [#6899](https://github.com/apache/datafusion-comet/pull/6899) (andygrove) +- feat: normalize floats in native array_distinct and array_union [#6563](https://github.com/apache/datafusion-comet/pull/6563) (andygrove) +- test: tag the Azure ABFS suites with AzureTest [#6942](https://github.com/apache/datafusion-comet/pull/6942) (comphead) +- feat: make native Iceberg writes opt-in for 1.2.0 [#6962](https://github.com/apache/datafusion-comet/pull/6962) (andygrove) +- feat: run broadcast joins natively when an unsupported scan is on the build side [#6786](https://github.com/apache/datafusion-comet/pull/6786) (parthchandra) +- feat: expose per-scan S3 HTTP coverage and retries [#6429](https://github.com/apache/datafusion-comet/pull/6429) (sunchao) +- build: use a single jni version across the native workspace [#6998](https://github.com/apache/datafusion-comet/pull/6998) (andygrove) +- feat: add nested benchmark derived from TPC-H [#7044](https://github.com/apache/datafusion-comet/pull/7044) (comphead) +- refactor: simplify shuffle support predicates [#7045](https://github.com/apache/datafusion-comet/pull/7045) (0lai0) +- feat: native RANGE window frames with explicit offset on DATE ORDER BY [#4974](https://github.com/apache/datafusion-comet/pull/4974) (0lai0) +- feat: support PivotFirst aggregate for the optimized PIVOT fast path [#4798](https://github.com/apache/datafusion-comet/pull/4798) (andygrove) +- feat: bump iceberg-rust and write float and double identity partitions natively [#6851](https://github.com/apache/datafusion-comet/pull/6851) (andygrove, mbutrovich) +- refactor: move MergeRows, Explode, aggregate, join and window planning into dedicated modules [#7014](https://github.com/apache/datafusion-comet/pull/7014) (andygrove) +- ci: label Delta pull requests with area:Delta [#6965](https://github.com/apache/datafusion-comet/pull/6965) (dwsmith1983) +- test: write ancient dates with CORRECTED rebase mode in the DATE RANGE window test [#7103](https://github.com/apache/datafusion-comet/pull/7103) (andygrove) +- refactor: share window frame bound offset serialization between lower and upper bounds [#7107](https://github.com/apache/datafusion-comet/pull/7107) (0lai0) +- ci: give the Delta contrib Maven cache its own key prefix [#7111](https://github.com/apache/datafusion-comet/pull/7111) (andygrove) +- chore: fix stddev accumulator size and resolve session libhdfs schemes once per scan rule [#7120](https://github.com/apache/datafusion-comet/pull/7120) (viirya) +- feat: add config to disable or restrict native UDF library loading, and load libraries without a global lock [#6775](https://github.com/apache/datafusion-comet/pull/6775) (andygrove) +- feat: support native wide-decimal hashing [#6548](https://github.com/apache/datafusion-comet/pull/6548) (rich7420) +- refactor: deduplicate split list-array construction [#7004](https://github.com/apache/datafusion-comet/pull/7004) (arminazimi) +- test: verify native timestamp output timezone labels [#6930](https://github.com/apache/datafusion-comet/pull/6930) (LinSimon-901101) +- feat: enable native replace for safe inputs [#6680](https://github.com/apache/datafusion-comet/pull/6680) (sam-1112) +- feat: opt in to join filter pushdown through deterministic residuals [#6431](https://github.com/apache/datafusion-comet/pull/6431) (sunchao) +- chore: limit open PRs for users without write access to 3 [#7135](https://github.com/apache/datafusion-comet/pull/7135) (andygrove) +- test: add window operator benchmarks [#7139](https://github.com/apache/datafusion-comet/pull/7139) (comphead) +- test: cover positional struct reconciliation in native CASE WHEN and COALESCE [#7149](https://github.com/apache/datafusion-comet/pull/7149) (kazuyukitanimura) +- refactor: share Parquet field-id and name resolution between root and nested fields [#6990](https://github.com/apache/datafusion-comet/pull/6990) (andygrove) +- feat: execute scalar Arrow Python UDFs in native Comet [#6130](https://github.com/apache/datafusion-comet/pull/6130) (viirya) +- feat: support AtLeastNNonNulls natively [#6180](https://github.com/apache/datafusion-comet/pull/6180) (rich7420) +- chore: [branch-1.2] change version from 1.2.0-SNAPSHOT to 1.2.0 [#7165](https://github.com/apache/datafusion-comet/pull/7165) (andygrove) +- feat: [branch-1.2] support the S3A magic committer in native Parquet writes (#6940) [#7281](https://github.com/apache/datafusion-comet/pull/7281) (andygrove) + +**Documentation updates:** + +- docs: add a simple diagram of how Comet uses executor memory [#6237](https://github.com/apache/datafusion-comet/pull/6237) (andygrove) +- docs: split the tuning guide into multiple pages [#6244](https://github.com/apache/datafusion-comet/pull/6244) (andygrove) +- docs: fix redirects that lead to missing pages [#6245](https://github.com/apache/datafusion-comet/pull/6245) (andygrove) +- docs: add 1.1.0 upgrade notes for fixed, deprecated and removed settings [#6248](https://github.com/apache/datafusion-comet/pull/6248) (andygrove) +- ci: run every tier on release-branch pull requests, and the full suite before an RC [#6218](https://github.com/apache/datafusion-comet/pull/6218) (andygrove) +- docs: remove nonexistent GROUPS frame references [#6299](https://github.com/apache/datafusion-comet/pull/6299) (manuzhang) +- docs: add contributor guide page on timezone handling [#6337](https://github.com/apache/datafusion-comet/pull/6337) (andygrove) +- docs: simplify the memory tuning page [#6343](https://github.com/apache/datafusion-comet/pull/6343) (andygrove) +- docs: correct the Tokio worker default and release_runtime's shutdown comment [#6300](https://github.com/apache/datafusion-comet/pull/6300) (andygrove) +- docs: add a backporting policy to the contributor guide [#6357](https://github.com/apache/datafusion-comet/pull/6357) (andygrove) +- docs: explain operator plan identity and exchange reuse [#6352](https://github.com/apache/datafusion-comet/pull/6352) (rich7420) +- docs: redraw the home page architecture diagram as a simple SVG [#6322](https://github.com/apache/datafusion-comet/pull/6322) (andygrove) +- docs: update TPC-DS benchmark results for Comet 1.1.0 on Spark 4.2 [#6308](https://github.com/apache/datafusion-comet/pull/6308) (andygrove) +- docs: correct what the Iceberg split-operator plan moves into AQE [#6407](https://github.com/apache/datafusion-comet/pull/6407) (andygrove) +- docs: refresh stale roadmap entries [#6309](https://github.com/apache/datafusion-comet/pull/6309) (andygrove) +- docs: add a Celeborn integration guide [#6373](https://github.com/apache/datafusion-comet/pull/6373) (pingzh) +- docs: request changes for correctness and performance findings in the review skill [#6483](https://github.com/apache/datafusion-comet/pull/6483) (andygrove) +- docs: promote Spark 4.2 from experimental to supported [#6417](https://github.com/apache/datafusion-comet/pull/6417) (andygrove) +- docs: cut the release branch before fixing regressions, and add a diagram [#6517](https://github.com/apache/datafusion-comet/pull/6517) (andygrove) +- docs: explain which Spark operators Comet leaves in place [#6551](https://github.com/apache/datafusion-comet/pull/6551) (andygrove) +- docs: note that the peak native aggregate memory metric is not reported [#6557](https://github.com/apache/datafusion-comet/pull/6557) (andygrove) +- docs: add release-behavior checks and the 1.1.0 audit's lessons to the review skills [#6516](https://github.com/apache/datafusion-comet/pull/6516) (andygrove) +- docs: note in the 1.1.0 upgrade guide that native Iceberg reads no longer fall back from IRSA [#6603](https://github.com/apache/datafusion-comet/pull/6603) (andygrove) +- docs: name the AQE query stages on the operator support page [#6660](https://github.com/apache/datafusion-comet/pull/6660) (andygrove) +- docs: date the float value and null count divergence to Iceberg 1.9 [#6681](https://github.com/apache/datafusion-comet/pull/6681) (andygrove) +- docs: update the timezone page for the fixes from the timezone EPIC [#6686](https://github.com/apache/datafusion-comet/pull/6686) (andygrove) +- docs: check timezone handling against the contributor guide in the PR review skills [#6340](https://github.com/apache/datafusion-comet/pull/6340) (andygrove) +- docs: add 1.1.0 changelog [#6728](https://github.com/apache/datafusion-comet/pull/6728) (andygrove) +- docs: add an FAQ page with an answer on vectorized Java/Scala UDFs [#6778](https://github.com/apache/datafusion-comet/pull/6778) (andygrove) +- docs: expand the FAQ and refresh the Gluten comparison [#6783](https://github.com/apache/datafusion-comet/pull/6783) (andygrove) +- docs: list the S3A settings the Iceberg native-write gate ignores [#6828](https://github.com/apache/datafusion-comet/pull/6828) (andygrove) +- docs: prefer behavior-named parameters over Spark versions [#6776](https://github.com/apache/datafusion-comet/pull/6776) (andygrove) +- docs: add proposal to establish Apache Comet as a top-level project [#6817](https://github.com/apache/datafusion-comet/pull/6817) (andygrove) +- docs: correct native Iceberg write docs that drifted from the code [#7009](https://github.com/apache/datafusion-comet/pull/7009) (andygrove) +- docs: document casts involving arrays, structs, and maps [#6320](https://github.com/apache/datafusion-comet/pull/6320) (andygrove) +- docs: list known Parquet and Iceberg issues and add a data source audit log [#7052](https://github.com/apache/datafusion-comet/pull/7052) (andygrove) +- docs: consolidate Parquet and Iceberg documentation into one page each [#7090](https://github.com/apache/datafusion-comet/pull/7090) (andygrove) +- docs: [branch-1.2] document known differences new in 1.2.0 in the upgrade guide [#7287](https://github.com/apache/datafusion-comet/pull/7287) (andygrove) + +**Other:** + +- chore: add benches remaining scalars [#6541](https://github.com/apache/datafusion-comet/pull/6541) (coderfender) +- chore: add benches for remaining agg functions [#6647](https://github.com/apache/datafusion-comet/pull/6647) (coderfender) +- build(deps): bump rustls from 0.23.44 to 0.23.45 in /native [#6701](https://github.com/apache/datafusion-comet/pull/6701) (dependabot[bot]) +- deps: bump to datafusion 55.2.0 [#6766](https://github.com/apache/datafusion-comet/pull/6766) (andygrove) +- build(deps): bump actions/setup-python from 6 to 7 [#7046](https://github.com/apache/datafusion-comet/pull/7046) (dependabot[bot]) + +## Credits + +Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. A PR with commits from more than one person counts for each of them. + +``` + 212 Andy Grove + 25 dustin + 21 KUAN-HAO HUANG + 18 Oleks V + 17 Liang-Chi Hsieh + 13 Peter Lee + 11 Chao Sun + 10 LinSimon-901101 + 8 ChenChen Lai + 6 Feng Zhang + 6 Unik Dahal + 5 Manu Zhang + 5 Ping Zhang + 5 Yu Hsun Hung + 5 sam-1112 + 4 Bhargava Vadlamani + 4 KAZUYUKI TANIMURA + 4 Parth Chandra + 4 Steve Vaughan + 2 Kazantsev Maksim + 2 dependabot[bot] + 2 nat-openai + 1 Alex Huang + 1 Armin Azimi + 1 Bill + 1 DeviousCardi + 1 Erik Bogado + 1 Han-Yin Chang + 1 Jaden Fix + 1 Jeff Wang + 1 Krishna Sudarshan J + 1 Matt Butrovich + 1 Michael Taranov + 1 Po-Han (Stan) Shih · 施博瀚 + 1 Samarth Mukhija + 1 YuLun Mao + 1 Yunchi Pang + 1 kid + 1 liupoyi-1031 + 1 sandudb +``` + +Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. diff --git a/docs/source/changelog/index.md b/docs/source/changelog/index.md index 792dfa2df3b..baadd676a83 100644 --- a/docs/source/changelog/index.md +++ b/docs/source/changelog/index.md @@ -24,6 +24,7 @@ Per-release change logs for Apache DataFusion Comet. ```{toctree} :maxdepth: 1 +1.2.0 <1.2.0> 1.1.0 <1.1.0> 1.0.0 <1.0.0> 0.17.1 <0.17.1> diff --git a/docs/source/user-guide/latest/migration-guide.md b/docs/source/user-guide/latest/migration-guide.md index 99b357d7cf4..feaa15aab06 100644 --- a/docs/source/user-guide/latest/migration-guide.md +++ b/docs/source/user-guide/latest/migration-guide.md @@ -146,6 +146,42 @@ without an `ORDER BY` can return rows in a different order than before, because files interleave. Spark does not guarantee the order of such a result. To read one file at a time in the order Spark assigned them, as before, set the setting to `1`. See [Iceberg tuning](iceberg.md#tuning). +### Known Differences From Spark That Are New in 1.2.0 + +The three differences below were not in Comet `1.1.0`. They come from operators and expressions that +now run natively where `1.1.0` fell back to Spark, so they are listed here although they are not +behavior changes that a legacy key restores. Each one has an issue, and each has a setting that +makes Comet use Spark's implementation instead. + +- **`collect_set` over `FLOAT` or `DOUBLE` next to `count(DISTINCT ...)` + ([#7203](https://github.com/apache/datafusion-comet/issues/7203)).** On Spark before 4.2, a query + that mixes `count(DISTINCT ...)` with other aggregates, such as + `SELECT x, count(DISTINCT y), collect_set(z) FROM t GROUP BY x`, now runs `collect_set` natively. + Comet `1.1.0` ran that query in Spark. Native `collect_set` keeps `-0.0` and `0.0` as two entries + and merges every `NaN` into one, where Spark collapses the zeros and keeps each `NaN`. The + [compatibility guide](compatibility/floating-point.md) already lists this difference for + `collect_set` on its own. Spark 4.2 is not affected. Set `spark.comet.exec.strictFloatingPoint=true` + to make `collect_set` over floating-point values fall back to Spark. +- **`RANGE` window frames over a `DATE` column with a very large offset + ([#5022](https://github.com/apache/datafusion-comet/issues/5022)).** A `RANGE` frame with an + explicit `PRECEDING` or `FOLLOWING` offset over a `DATE` `ORDER BY` column now runs natively, + where `1.1.0` fell back for every such frame. When the offset is roughly 92 million days (about + 250,000 years) or more, the frame bound overflows the date range, and Comet can put rows with a + `NULL` date in the frame where Spark does not. Comet falls back above 2,144,550,751 days. Offsets + of a realistic size are not affected. To avoid it for larger offsets, set + `spark.comet.exec.window.enabled=false`, which runs every window operator in Spark. +- **Native `min` and `max` over `STRING` or `BINARY` under-report their memory + ([#6936](https://github.com/apache/datafusion-comet/issues/6936)).** Comet `1.2.0` runs these + natively, where `1.1.0` fell back. In a hash aggregate, DataFusion's accumulator counts the + length of each group's current value rather than the memory it holds, so a group whose extreme + value gets shorter, such as the `min` of URLs where a prefix arrives after a longer value, holds + more memory than Comet's memory pool sees. The aggregate then spills later than it should, and a + task can run out of memory. Results are not affected. Spark plans a hash aggregate for this when + the query also has an aggregate such as `collect_set`, as in + `SELECT k, min(s), collect_set(tag) FROM t GROUP BY k`. A plain `min` or `max` over a string is + not affected. Set `spark.sql.execution.useObjectHashAggregateExec=false` to make Spark plan a + sort aggregate for these queries, which is not affected. + ## Upgrading to Comet 1.1.0 Comet `1.1.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The changes below