Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions docs/source/user-guide/latest/migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,42 @@ without an `ORDER BY` can return rows in a different order than before, because
files interleave. Spark does not guarantee the order of such a result. To read one file at a time in
the order Spark assigned them, as before, set the setting to `1`. See [Iceberg tuning](iceberg.md#tuning).

### Known Differences From Spark That Are New in 1.2.0

The three differences below were not in Comet `1.1.0`. They come from operators and expressions that
now run natively where `1.1.0` fell back to Spark, so they are listed here although they are not
behavior changes that a legacy key restores. Each one has an issue, and each has a setting that
makes Comet use Spark's implementation instead.

- **`collect_set` over `FLOAT` or `DOUBLE` next to `count(DISTINCT ...)`
([#7203](https://github.com/apache/datafusion-comet/issues/7203)).** On Spark before 4.2, a query
that mixes `count(DISTINCT ...)` with other aggregates, such as
`SELECT x, count(DISTINCT y), collect_set(z) FROM t GROUP BY x`, now runs `collect_set` natively.
Comet `1.1.0` ran that query in Spark. Native `collect_set` keeps `-0.0` and `0.0` as two entries
and merges every `NaN` into one, where Spark collapses the zeros and keeps each `NaN`. The
[compatibility guide](compatibility/floating-point.md) already lists this difference for
`collect_set` on its own. Spark 4.2 is not affected. Set `spark.comet.exec.strictFloatingPoint=true`
to make `collect_set` over floating-point values fall back to Spark.
- **`RANGE` window frames over a `DATE` column with a very large offset
([#5022](https://github.com/apache/datafusion-comet/issues/5022)).** A `RANGE` frame with an
explicit `PRECEDING` or `FOLLOWING` offset over a `DATE` `ORDER BY` column now runs natively,
where `1.1.0` fell back for every such frame. When the offset is roughly 92 million days (about
250,000 years) or more, the frame bound overflows the date range, and Comet can put rows with a
`NULL` date in the frame where Spark does not. Comet falls back above 2,144,550,751 days. Offsets
of a realistic size are not affected. To avoid it for larger offsets, set
`spark.comet.exec.window.enabled=false`, which runs every window operator in Spark.
- **Native `min` and `max` over `STRING` or `BINARY` under-report their memory
([#6936](https://github.com/apache/datafusion-comet/issues/6936)).** Comet `1.2.0` runs these
natively, where `1.1.0` fell back. In a hash aggregate, DataFusion's accumulator counts the
length of each group's current value rather than the memory it holds, so a group whose extreme
value gets shorter, such as the `min` of URLs where a prefix arrives after a longer value, holds
more memory than Comet's memory pool sees. The aggregate then spills later than it should, and a
task can run out of memory. Results are not affected. Spark plans a hash aggregate for this when
the query also has an aggregate such as `collect_set`, as in
`SELECT k, min(s), collect_set(tag) FROM t GROUP BY k`. A plain `min` or `max` over a string is
not affected. Set `spark.sql.execution.useObjectHashAggregateExec=false` to make Spark plan a
sort aggregate for these queries, which is not affected.

## Upgrading to Comet 1.1.0

Comet `1.1.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The changes below
Expand Down