diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/aggregate.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/aggregate.md new file mode 100644 index 00000000000..d8d4966d6a2 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/aggregate.md @@ -0,0 +1,123 @@ + + +# Aggregate Expressions + + + +## ApproximatePercentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The percentage argument must be a foldable literal. +- The accuracy argument must be a foldable literal. +- Only byte, short, int, long, float, and double input types are supported. + +## Average + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- YearMonthIntervalType and DayTimeIntervalType inputs are not supported + +## CollectSet + +The following incompatibilities cause `CollectSet` to fall back to Spark by default. Set `spark.comet.expression.CollectSet.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Before Spark 4.2, Comet deduplicates NaN values (treats `NaN == NaN`) while Spark treats each NaN as a distinct value. Comet treats -0.0 and 0.0 as distinct while Spark treats them as equal. When `spark.comet.exec.strictFloatingPoint=true`, `collect_set` on floating-point types falls back to Spark on those versions unless `spark.comet.expression.CollectSet.allowIncompatible=true` is set. + +## First + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## HyperLogLogPlusPlus + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input type must be boolean, integral, floating-point, decimal, date/time, string, or binary +- `DecimalType` with precision > 18 is not supported +- Collated (non-UTF8_BINARY) strings are not supported + +## Last + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## Max + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MaxBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the maximum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Min + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MinBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the minimum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Mode + +The following incompatibilities cause `Mode` to fall back to Spark by default. Set `spark.comet.expression.Mode.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- mode breaks ties non-deterministically in Spark (the result depends on JVM hash-map iteration order); Comet returns the smallest of the tied values instead (https://github.com/apache/datafusion-comet/issues/3970) + +## Percentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- An array of percentages is not supported. +- The percentage argument must be a literal. +- A frequency argument is not supported. +- Descending order in `WITHIN GROUP (ORDER BY ... DESC)` is not supported. +- Only numeric input types are supported. + +## PivotFirst + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Value data type other than Boolean, Byte, Short, Int, Long, Float, Double or Decimal +- Pivot column of a type other than Boolean, Byte, Short, Int, Long, Float, Double, Decimal, String (UTF8_BINARY collation), Date, Timestamp or TimestampNTZ, which includes the struct that a pivot on more than one column builds +- Pivot values list is empty +- Pivot values list contains duplicates, including `0.0` next to `-0.0` + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/array.md new file mode 100644 index 00000000000..3e94336b308 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/array.md @@ -0,0 +1,113 @@ + + +# Array Expressions + + + +## ArrayAppend + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- array_append with a nullable array and a nondeterministic operand such as `rand()`: the native NULL guard evaluates the operands more than once, on subsets of the rows + +## ArrayContains + +For applicable cases that are not selected for native execution automatically, `ArrayContains` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayContains.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares array elements with ordering.equiv, so -0.0 matches +0.0 and all NaNs match each other; Comet's native array_contains applies that only to flat FLOAT and DOUBLE elements, and nested float elements stay on Spark's comparison +- array_contains evaluates its value eagerly, while Spark skips it when the array is null +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_contains compares raw bytes + +## ArrayDistinct + +The following incompatibilities cause `ArrayDistinct` to fall back to Spark by default. Set `spark.comet.expression.ArrayDistinct.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_distinct compares raw bytes + +## ArrayExcept + +By default, `ArrayExcept` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayExcept.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Null handling and ordering may differ from Spark + +## ArrayIntersect + +By default, `ArrayIntersect` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayIntersect.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Result array element order may differ from Spark when the right array is longer than the left (DataFusion probes the longer side). +- array_intersect does not propagate non-UTF8_BINARY collations to the output array elements (https://github.com/apache/datafusion-comet/issues/2190) + +## ArrayJoin + +By default, `ArrayJoin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayJoin.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190) +- array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178) + +## ArrayMax + +For applicable cases that are not selected for native execution automatically, `ArrayMax` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMax.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayMin + +For applicable cases that are not selected for native execution automatically, `ArrayMin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMin.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayUnion + +The following incompatibilities cause `ArrayUnion` to fall back to Spark by default. Set `spark.comet.expression.ArrayUnion.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_union compares raw bytes + +## ArraysOverlap + +For applicable cases that are not selected for native execution automatically, `ArraysOverlap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArraysOverlap.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native arrays_overlap compares raw bytes + +## ArraysZip + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all input data types are supported; falls back to Spark for unsupported types + +## Sequence + +The following differences from Spark are always present and do not require any additional configuration: + +- When `stop - start` overflows a long but the sequence is short, Spark 4.0.5, 4.1.4, 4.2.1 and later return its elements (SPARK-58821), and earlier releases raise `INTERNAL_ERROR`. Comet chooses the behavior by the Spark version number. On a vendor build that reports an older version but includes SPARK-58821, Comet raises the error where Spark returns the elements. + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- date and timestamp element types run through the JVM codegen dispatcher +- sequence arguments must be literals or column references; other shapes run through the JVM codegen dispatcher to preserve Spark's per-row null short-circuit + +## Shuffle + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- shuffle requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/cast.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/cast.md new file mode 100644 index 00000000000..c99186ccaff --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/cast.md @@ -0,0 +1,324 @@ + + +# Cast + +Cast operations in Comet fall into three levels of support: + +- **C (Compatible)**: The results match Apache Spark +- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By + default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark. + Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended + for production use. +- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated + cast inside the Comet pipeline. +- **N/A**: Spark does not support this cast. + +Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for +example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`. + +The tables at the end of this page cover casts between primitive types. See [Complex Types](#complex-types) for +casts involving arrays, structs, and maps. + +## ANSI Mode Support + +Enabling ANSI mode does not by itself make a cast fall back to Spark or require +`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support +level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the +[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting. + +## Whitespace Trimming in Casts from String + +Spark uses two different notions of whitespace when casting from string, and Comet reproduces +both: + +| Trim set | Cast targets | +| -------------------------------------- | --------------------------------------------------------------------------------------- | +| Bytes `0x00`-`0x20` and `0x7F` (`DEL`) | `BOOLEAN`, `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_NTZ` | +| Bytes `0x00`-`0x20` only | `FLOAT`, `DOUBLE`, `DECIMAL` | + +Neither set includes any non-ASCII whitespace, so padding a value with `U+0085`, `U+00A0`, +`U+1680`, `U+2000`-`U+200A`, `U+2028`, `U+2029`, `U+202F`, `U+205F` or `U+3000` produces `NULL` +(or raises under ANSI mode), even though those codepoints are whitespace to Unicode. Whitespace +inside a value is never trimmed. + +## String to Decimal + +Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's behavior, +including: + +- Leading and trailing bytes `0x00`-`0x20` are trimmed before parsing, matching the + `java.lang.String.trim` that Spark applies before handing the value to `BigDecimal`. +- Null bytes (`\u0000`) at the start or end of a string are therefore trimmed. Null bytes + embedded in the middle of a string produce `NULL`. +- Non-ASCII decimal digits that Java's `BigDecimal` accepts, i.e. every Unicode `Nd` digit in + the Basic Multilingual Plane (fullwidth `１２３.４５`, Arabic-Indic `١٢٣`, Devanagari `१२३`, + ...), are treated as their ASCII equivalents in both the mantissa and the exponent, so + `CAST('١٢٣.٤٥' AS DECIMAL(10,2))` returns `123.45`. Digits outside the Basic Multilingual + Plane, such as U+1D7CE MATHEMATICAL BOLD DIGIT ZERO, are rejected as they are by Spark. +- Scientific notation (e.g. `1.23E+5`) is supported. +- Special values (`inf`, `infinity`, `nan`) produce `NULL`. + +A zero with a large exponent, such as `'0e39'`, casts to `0` on Spark 4.0.5, 4.1.4, 4.2.1 and later +(SPARK-60118). Earlier releases return `NULL` for it, or raise `NUMERIC_OUT_OF_SUPPORTED_RANGE` +under ANSI mode. Comet chooses the behavior by the Spark version number. On a vendor build that +reports an older version but includes SPARK-60118, Comet returns `NULL` (or raises) where Spark +returns `0`. + +## String to Date + +Comet's native `CAST(string AS DATE)` implementation matches Apache Spark's behavior for years +between 262143 BC and 262142 AD. This range limitation comes from the underlying chrono library's +`NaiveDate` type. Spark itself supports a wider range. All three eval modes (Legacy, ANSI, Try) +are supported. + +Supported input formats match Spark exactly: + +- `yyyy`, `yyyy-[m]m`, `yyyy-[m]m-[d]d` +- Optional `T` suffix with arbitrary trailing text (e.g. `2020-01-01T12:34:56`) +- Leading/trailing bytes `0x00`-`0x20` and `0x7F` are trimmed +- Optional sign prefix (`-` for negative years) +- Leading zeros (e.g. `0002020-01-01` is year 2020) + +## Date to Timestamp + +Comet's native `CAST(date AS TIMESTAMP)` is compatible with Spark. The cast interprets each +date as midnight in the session timezone and converts to a UTC epoch value. DST transitions +are handled correctly, including spring-forward gaps (where midnight may not exist) and +fall-back ambiguity (where Comet picks the earlier/DST occurrence, matching Spark's +`LocalDate.atStartOfDay(zoneId)` behavior). + +## Date to TimestampNTZ + +Comet's native `CAST(date AS TIMESTAMP_NTZ)` is compatible with Spark. The cast is +timezone-independent: each date is converted to midnight as pure arithmetic +(`days * 86,400,000,000` microseconds) with no session timezone offset applied. The result +is the same regardless of the session timezone setting. + +## Date to Numeric Types + +In Legacy mode, `CAST(date AS INT)`, `CAST(date AS LONG)`, and casts to all other numeric +types (Boolean, Byte, Short, Float, Double, Decimal) always return `NULL`. Comet handles +this by short-circuiting to a null literal during query planning, so no native execution +is needed. In ANSI and Try modes, Spark rejects these casts at analysis time (before +execution reaches Comet). The short-circuit applies only to a top-level cast. See +[Complex Types](#complex-types) for `DATE` values inside structs and maps. + +## String to Timestamp + +Comet's native `CAST(string AS TIMESTAMP)` implementation supports all timestamp formats accepted +by Apache Spark, including ISO 8601 date-time strings, date-only strings, time-only strings +(`HH:MM:SS`), embedded timezone offsets (e.g. `+07:30`, `GMT-01:00`, `UTC`), named timezone +suffixes (e.g. `Europe/Moscow`), and the full Spark timestamp year range +(-290308 to 294247). + +## String to TimestampNTZ + +Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior. +Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in +the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time +components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`. +The result is always a wall-clock timestamp with no timezone conversion or DST adjustment. + +## TimestampNTZ Casts + +Comet supports the following `TIMESTAMP_NTZ` casts natively: + +| Cast | Compatible | Notes | +| ---------------------------------- | ---------- | ----------------------------------------------------------------- | +| `CAST(timestamp_ntz AS STRING)` | Yes | Formats local time as-is, timezone-independent | +| `CAST(timestamp_ntz AS DATE)` | Yes | Extracts the date component, timezone-independent | +| `CAST(timestamp_ntz AS TIMESTAMP)` | Yes | Interprets NTZ as local time in session TZ, converts to UTC epoch | +| `CAST(date AS TIMESTAMP_NTZ)` | Yes | Pure arithmetic, timezone-independent | +| `CAST(timestamp AS TIMESTAMP_NTZ)` | Yes | Shifts UTC epoch to local time in session TZ | +| `CAST(string AS TIMESTAMP_NTZ)` | Yes | See [String to TimestampNTZ](#string-to-timestampntz) above | + +The NTZ-to-Timestamp and Timestamp-to-NTZ casts are session-timezone-dependent (the session +timezone determines the UTC offset). All other NTZ casts are timezone-independent and produce +the same result regardless of the session timezone. + +## Date to String + +Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000 are +zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are +rendered without truncation. The cast is timezone-independent. + +## Decimal with Negative Scale to String + +Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when +`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is +disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's +code-generated cast instead of the native one. + +When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches +Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific +notation (e.g. a value of 12300 stored as `Decimal(7,-2)` with unscaled value 123 is rendered +as `"1.23E+4"`). + +## Decimal with Negative Scale to Integer and Timestamp + +Casting a `DecimalType` with a negative scale to `ByteType`, `ShortType`, `IntegerType`, +`LongType` or `TimestampType` has no native implementation, and neither does casting any of the +integer types to a negative-scale `DecimalType`. Comet reports these as unsupported, which +routes them through the JVM codegen dispatcher (Spark's own generated code) inside the Comet +pipeline, so the results match Spark exactly. + +The native kernels align scale by computing `10^|scale|` in fixed-point arithmetic, which +overflows when the scale is negative. See +[#5013](https://github.com/apache/datafusion-comet/issues/5013). + +All other cast directions to and from negative-scale decimals run natively: `FloatType` and +`DoubleType`, `StringType` in both directions (subject to the config gate described above), +casts from a negative-scale decimal to `BooleanType`, and widening to another `DecimalType`. +Casts from `BooleanType` to a `DecimalType` route through the dispatcher at any scale, which is +not specific to a negative one. + +## Complex Types + +Comet casts arrays, structs, and maps by casting each element, field, key, and value with the +rules for primitive types, using the table for the same eval mode. A complex-type cast is +therefore compatible only when every cast it contains is compatible. For example, casting +`ARRAY` to `ARRAY` has the same support level as casting `INT` to `BIGINT`, and a +struct cast with one unsupported field cast is unsupported. The rules apply at every level of +nesting, so casting `ARRAY>` to `ARRAY>` also has the same +support level as casting `INT` to `BIGINT`. + +| From | To | Support level | +| ----------- | ------------- | ---------------------------------------------------------------------------- | +| `ARRAY` | `ARRAY` | Same as `T` to `U` | +| `ARRAY` | `STRING` | Same as `T` to `STRING` | +| `STRUCT` | `STRUCT` | Compatible if every field cast is compatible. Fields are matched by position | +| `STRUCT` | `STRING` | Compatible if the cast from every field to `STRING` is compatible | +| `MAP` | `MAP` | Compatible if `K` to `K2` and `V` to `V2` are both compatible | +| `MAP` | `STRING` | Unsupported | + +The following casts are exceptions to these rules: + +- `ARRAY` can only be cast natively to `ARRAY` or `ARRAY`. Casts to other array + types are unsupported. +- A cast from a `DATE` struct field or map value to a numeric or boolean type is unsupported. In + Legacy mode these casts always return `NULL` (see [Date to Numeric Types](#date-to-numeric-types)). +- When `spark.sql.legacy.castComplexTypesToString.enabled=true`, casts from arrays, structs, and + maps to `STRING` are unsupported, because Comet's native cast only produces Spark's default + string format. + +In Try mode, a cast between maps fails with an error when a key cannot be cast, where Spark returns +a result ([#5995](https://github.com/apache/datafusion-comet/issues/5995)). This only affects key +casts that Spark accepts under `TRY_CAST`, such as narrowing `BIGINT` to `INT`. Spark rejects a +`STRING` key cast at analysis. + +## Decimal with Negative Scale to Integral Types + +Casting a `DecimalType` with a negative scale, which Spark only creates when +`spark.sql.legacy.allowNegativeScaleOfDecimal=true`, to `ByteType`, `ShortType`, `IntegerType` +or `LongType` is not supported natively in any eval mode. The native kernels divide by +`10^scale`, which overflows for a negative scale, so Comet leaves the cast to Spark. + +## Legacy Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | C | N/A | +| byte | C | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | C | C | - | C | C | C | C | C | C | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | C | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | C | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | C | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## Try Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## ANSI Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md new file mode 100644 index 00000000000..b147e6e546a --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md @@ -0,0 +1,156 @@ + + +# Date/Time Expressions + +- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is marked Incompatible and + routes through the JVM codegen dispatcher by default, producing Spark-identical results. The + native path is itself correct for dates within chrono-tz's DST horizon (approximately year 2100; + see "Date and Time Functions" below) and can be enabled by setting + `spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ inputs are handled + correctly regardless of session timezone (timezone-independent truncation). + +## Date and Time Functions + +Comet's native implementation of date and time functions may produce different results than Spark for dates +far in the future (approximately beyond year 2100). This is because Comet uses the chrono-tz library for +timezone calculations, which has limited support for Daylight Saving Time (DST) rules beyond the IANA +time zone database's explicit transitions. + +For dates within a reasonable range (approximately 1970-2100), Comet's date and time functions are compatible +with Spark. For dates beyond this range, functions that involve timezone-aware calculations (such as +`date_trunc` with timezone-aware timestamps) may produce results with incorrect DST offsets. + +If you need to process dates far in the future with accurate timezone handling, consider: + +- Using timezone-naive types (`timestamp_ntz`) when timezone conversion is not required +- Falling back to Spark for these specific operations + +### Timezone Database Versions + +Comet's native code converts between instants and local time with the IANA timezone database that +chrono-tz compiles into the Comet library. Spark uses the JVM's timezone database (`tzdb.dat`), whose +version depends on the JDK build and on whether its timezone data has been updated. When the two versions +have different rules for a timezone, local times that Comet computes natively for that timezone can differ +from Spark's for the affected dates. That covers `hour`, casts between timestamps and strings or dates, +`date_trunc`, and parsing strings as timestamps. Comet logs a warning at startup when the two versions +differ. + + + +## ConvertTimezone + +By default, `ConvertTimezone` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ConvertTimezone.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## DateFormatClass + +The following differences from Spark are always present and do not require any additional configuration: + +- Format strings in a curated allow-list run natively via DataFusion's `to_char` for UTC sessions. Other format strings (including non-literal formats), as well as non-UTC sessions, route through Spark's own `DateFormatClass.doGenCode` via the Arrow-direct codegen dispatcher when `spark.comet.exec.scalaUDF.codegen.enabled=true` (the default). When the codegen dispatcher is disabled the operator falls back to Spark in those cases. + +By default, `DateFormatClass` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.DateFormatClass.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-UTC timezones may produce different results than Spark + +## Days + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DateType` and `TimestampType` inputs are supported. `TimestampNTZType` is not supported. + +## FromUTCTimestamp + +By default, `FromUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## FromUnixTime + +By default, `FromUnixTime` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUnixTime.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's valid timestamp range differs from Spark (https://github.com/apache/datafusion/issues/16594) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the default datetime format pattern `yyyy-MM-dd HH:mm:ss` is supported + +## Hours + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `TimestampType` and `TimestampNTZType` inputs are supported. + +## MakeInterval + +The following differences from Spark are always present and do not require any additional configuration: + +- Both the default JVM codegen-dispatch path and the native path currently limit the elapsed-time component to about 292 years in either direction. This only affects extreme intervals and is tracked in [#5279](https://github.com/apache/datafusion-comet/issues/5279). + +By default, `MakeInterval` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MakeInterval.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- The native implementation converts seconds to `Float64`, which can lose microsecond precision, and stores time in nanoseconds, which overflows for large time components (hours, minutes, seconds) that Spark can represent. + +## PreciseTimestampConversion + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only reinterprets between TimestampType/TimestampNTZType and LongType are supported. + +## SecondsToTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `DecimalType`, `ByteType`, and `ShortType` inputs + +## ToUTCTimestamp + +By default, `ToUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ToUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## TruncDate + +By default, `TruncDate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncDate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-literal format strings will throw an exception instead of returning NULL + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week + +## TruncTimestamp + +By default, `TruncTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Produces incorrect results when used with non-UTC timezones. Compatible when timezone is UTC. (https://github.com/apache/datafusion-comet/issues/2649) +- Non-literal format strings will throw an exception instead of returning NULL + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week, day, dd, hour, minute, second, millisecond, microsecond + +## UnixTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- String inputs, including collated strings, have no native implementation. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/map.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/map.md new file mode 100644 index 00000000000..d16f9df138b --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/map.md @@ -0,0 +1,66 @@ + + +# Map Expressions + +## MapSort (Spark 4.0+) + +Spark 4.0 inserts `MapSort` to normalize map values when they appear in shuffle hash partitioning +keys, in `try_element_at`, and in other contexts where map ordering must be deterministic. Comet +runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0. + +When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose +keys contain `Float` or `Double`. Arrow's sort uses +IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for +`NaN` and `-0.0`. + + + +## MapFromArrays + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## MapFromEntries + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromEntries.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `BinaryType` is not supported as a map key in `map_from_entries` +- `BinaryType` is not supported as a map value in `map_from_entries` +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## StringToMap + +By default, `StringToMap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringToMap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `spark.sql.legacy.truncateForEmptyRegexSplit` is enabled, so trailing empty split entries may differ from Spark. +- `str_to_map` does not support non-UTF8_BINARY collations on the input string or delimiters. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/math.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/math.md new file mode 100644 index 00000000000..b0e2ec1d3b1 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/math.md @@ -0,0 +1,79 @@ + + +# Math Expressions + + + +## Abs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `INTERVAL YEAR TO MONTH` and `INTERVAL DAY TO SECOND` inputs + +## Divide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Greatest + +For applicable cases that are not selected for native execution automatically, `Greatest` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Greatest.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native greatest compares raw bytes + +## IntegralDivide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Least + +For applicable cases that are not selected for native execution automatically, `Least` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Least.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native least compares raw bytes + +## Rand + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Randn + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Round + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Float and double inputs. Spark rounds them through a BigDecimal built from `java.lang.Double.toString()` rather than from the exact binary value, and that shortened decimal string can round differently than the value it came from +- Negative-scale decimal inputs, which are only creatable with spark.sql.legacy.allowNegativeScaleOfDecimal=true + +## Signum + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DoubleType` is supported + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/misc.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/misc.md new file mode 100644 index 00000000000..127be9f4131 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/misc.md @@ -0,0 +1,66 @@ + + +# Miscellaneous Expressions + + + +## CheckOverflow + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DecimalType` is supported + +## Literal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for literal values + +## MakeDecimal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `LongType` input is supported + +## ScalarSubquery + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for scalar subquery results + +## SortOrder + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- A nested sort key with nondefault root null placement must be deterministic: the wrapper can evaluate the key twice, for isnull(key) and its value, which could disagree + +## StaticInvoke + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 10 is one such value: the result has 19 digits. The other `truncate` input types, and `bucket` on decimals, are unaffected. + +## Uuid + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- uuid requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/string.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/string.md new file mode 100644 index 00000000000..bdd6d1f9e20 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/string.md @@ -0,0 +1,217 @@ + + +# String Expressions + + + +## BitLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## Concat + +By default, `Concat` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Concat.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- concat does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- CONCAT supports only string input parameters + +## ConcatWs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- all arguments are foldable + +## Contains + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## EndsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## GetJsonObject + +By default, `GetJsonObject` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.GetJsonObject.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark allows single-quoted JSON and unescaped control characters which Comet does not support +- Very long numbers near Jackson's 1000-digit limit can depend on Spark's recycled parser buffer, which Comet cannot reproduce from the input alone +- Selected JSON integers outside the 64-bit range and very long numbers can lose precision or fail during Comet's native JSON materialization +- Some selected floating-point values can differ from Spark at decimal parsing or Java-version formatting boundaries +- When a returned object or array contains duplicate keys, Spark preserves them while Comet's native JSON materialization keeps only the last value + +## InitCap + +By default, `InitCap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.InitCap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Treats hyphen as a word separator (e.g. `robert rose-smith` produces `Robert Rose-Smith` instead of Spark's `Robert Rose-smith`) (https://github.com/apache/datafusion-comet/issues/1052) + +## Levenshtein + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-default (non-UTF8_BINARY) collated input. The native kernel compares raw bytes, so collation-aware comparison has no native path. + +## Like + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- LIKE with a custom escape character (only `\` is supported natively) +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## Lower + +By default, `Lower` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + +## OctetLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## RLike + +The following cases use Comet's native implementation by default: + +- A `UTF8_BINARY` literal pattern admitted by the [plan-time compatibility analyzer](../../regex.md#when-the-rust-engine-is-safe) is evaluated natively by default. + +For applicable cases that are not selected for native execution automatically, `RLike` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.RLike.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- For applicable literal patterns outside the automatically admitted subset, the native Rust regex engine may behave differently from Java regex. + +## RegExpReplace + +By default, `RegExpReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.RegExpReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regexp pattern may not be compatible with Spark + +## Reverse + +By default, `Reverse` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Reverse.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- native reverse does not support arrays whose element type contains binary, struct, or map +- reverse does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- reverse has no native implementation for binary input + +## StartsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## StringInstr + +For applicable cases that are not selected for native execution automatically, `StringInstr` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringInstr.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native instr compares raw bytes + +## StringLPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRepeat + +The following differences from Spark are always present and do not require any additional configuration: + +- A negative argument for the number of times to repeat throws an exception instead of returning an empty string as Spark does + +## StringReplace + +The following cases use Comet's native implementation by default: + +- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default. + +For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast + +## StringSplit + +By default, `StringSplit` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringSplit.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regex engine differences between Java and Rust + +## StringTranslate + +By default, `StringTranslate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel + +## StringTrim + +For applicable cases that are not selected for native execution automatically, `StringTrim` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrim.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native trim with a trim string compares raw bytes + +## StringTrimLeft + +For applicable cases that are not selected for native execution automatically, `StringTrimLeft` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimLeft.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native ltrim with a trim string compares raw bytes + +## StringTrimRight + +For applicable cases that are not selected for native execution automatically, `StringTrimRight` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimRight.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native rtrim with a trim string compares raw bytes + +## SubstringIndex + +For applicable cases that are not selected for native execution automatically, `SubstringIndex` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.SubstringIndex.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native substring_index compares raw bytes + +## UnBase64 + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- unbase64 with failOnError = true uses stricter RFC 4648 validation that is not yet implemented natively +- unbase64 with a non-trivial child expression uses the JVM codegen dispatcher to preserve Spark's short-circuit evaluation (native path is limited to column and literal children) + +## Upper + +By default, `Upper` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/struct.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/struct.md new file mode 100644 index 00000000000..c530b12a1a8 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/struct.md @@ -0,0 +1,52 @@ + + +# Struct Expressions + + + +## CreateNamedStruct + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `CreateNamedStruct` with duplicate field names is not supported + +## JsonToStructs + +By default, `JsonToStructs` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.JsonToStructs.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Partially implemented and not comprehensively tested + +## StructsToCsv + +By default, `StructsToCsv` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToCsv.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Date, Timestamp, TimestampNTZ, and Binary data types may produce different results (https://github.com/apache/datafusion-comet/issues/3232) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Complex types (arrays, maps, structs) in the schema are not supported + +## StructsToJson + +By default, `StructsToJson` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToJson.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Does not support `+Infinity` and `-Infinity` for numeric types (float, double). (https://github.com/apache/datafusion-comet/issues/3016) + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/url.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/url.md new file mode 100644 index 00000000000..765e12941fd --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/url.md @@ -0,0 +1,23 @@ + + +# URL Expressions + + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/aggregate.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/aggregate.md new file mode 100644 index 00000000000..d8d4966d6a2 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/aggregate.md @@ -0,0 +1,123 @@ + + +# Aggregate Expressions + + + +## ApproximatePercentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The percentage argument must be a foldable literal. +- The accuracy argument must be a foldable literal. +- Only byte, short, int, long, float, and double input types are supported. + +## Average + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- YearMonthIntervalType and DayTimeIntervalType inputs are not supported + +## CollectSet + +The following incompatibilities cause `CollectSet` to fall back to Spark by default. Set `spark.comet.expression.CollectSet.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Before Spark 4.2, Comet deduplicates NaN values (treats `NaN == NaN`) while Spark treats each NaN as a distinct value. Comet treats -0.0 and 0.0 as distinct while Spark treats them as equal. When `spark.comet.exec.strictFloatingPoint=true`, `collect_set` on floating-point types falls back to Spark on those versions unless `spark.comet.expression.CollectSet.allowIncompatible=true` is set. + +## First + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## HyperLogLogPlusPlus + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input type must be boolean, integral, floating-point, decimal, date/time, string, or binary +- `DecimalType` with precision > 18 is not supported +- Collated (non-UTF8_BINARY) strings are not supported + +## Last + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## Max + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MaxBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the maximum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Min + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MinBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the minimum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Mode + +The following incompatibilities cause `Mode` to fall back to Spark by default. Set `spark.comet.expression.Mode.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- mode breaks ties non-deterministically in Spark (the result depends on JVM hash-map iteration order); Comet returns the smallest of the tied values instead (https://github.com/apache/datafusion-comet/issues/3970) + +## Percentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- An array of percentages is not supported. +- The percentage argument must be a literal. +- A frequency argument is not supported. +- Descending order in `WITHIN GROUP (ORDER BY ... DESC)` is not supported. +- Only numeric input types are supported. + +## PivotFirst + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Value data type other than Boolean, Byte, Short, Int, Long, Float, Double or Decimal +- Pivot column of a type other than Boolean, Byte, Short, Int, Long, Float, Double, Decimal, String (UTF8_BINARY collation), Date, Timestamp or TimestampNTZ, which includes the struct that a pivot on more than one column builds +- Pivot values list is empty +- Pivot values list contains duplicates, including `0.0` next to `-0.0` + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/array.md new file mode 100644 index 00000000000..3e94336b308 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/array.md @@ -0,0 +1,113 @@ + + +# Array Expressions + + + +## ArrayAppend + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- array_append with a nullable array and a nondeterministic operand such as `rand()`: the native NULL guard evaluates the operands more than once, on subsets of the rows + +## ArrayContains + +For applicable cases that are not selected for native execution automatically, `ArrayContains` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayContains.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares array elements with ordering.equiv, so -0.0 matches +0.0 and all NaNs match each other; Comet's native array_contains applies that only to flat FLOAT and DOUBLE elements, and nested float elements stay on Spark's comparison +- array_contains evaluates its value eagerly, while Spark skips it when the array is null +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_contains compares raw bytes + +## ArrayDistinct + +The following incompatibilities cause `ArrayDistinct` to fall back to Spark by default. Set `spark.comet.expression.ArrayDistinct.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_distinct compares raw bytes + +## ArrayExcept + +By default, `ArrayExcept` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayExcept.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Null handling and ordering may differ from Spark + +## ArrayIntersect + +By default, `ArrayIntersect` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayIntersect.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Result array element order may differ from Spark when the right array is longer than the left (DataFusion probes the longer side). +- array_intersect does not propagate non-UTF8_BINARY collations to the output array elements (https://github.com/apache/datafusion-comet/issues/2190) + +## ArrayJoin + +By default, `ArrayJoin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayJoin.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190) +- array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178) + +## ArrayMax + +For applicable cases that are not selected for native execution automatically, `ArrayMax` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMax.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayMin + +For applicable cases that are not selected for native execution automatically, `ArrayMin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMin.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayUnion + +The following incompatibilities cause `ArrayUnion` to fall back to Spark by default. Set `spark.comet.expression.ArrayUnion.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_union compares raw bytes + +## ArraysOverlap + +For applicable cases that are not selected for native execution automatically, `ArraysOverlap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArraysOverlap.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native arrays_overlap compares raw bytes + +## ArraysZip + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all input data types are supported; falls back to Spark for unsupported types + +## Sequence + +The following differences from Spark are always present and do not require any additional configuration: + +- When `stop - start` overflows a long but the sequence is short, Spark 4.0.5, 4.1.4, 4.2.1 and later return its elements (SPARK-58821), and earlier releases raise `INTERNAL_ERROR`. Comet chooses the behavior by the Spark version number. On a vendor build that reports an older version but includes SPARK-58821, Comet raises the error where Spark returns the elements. + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- date and timestamp element types run through the JVM codegen dispatcher +- sequence arguments must be literals or column references; other shapes run through the JVM codegen dispatcher to preserve Spark's per-row null short-circuit + +## Shuffle + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- shuffle requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/cast.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/cast.md new file mode 100644 index 00000000000..c99186ccaff --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/cast.md @@ -0,0 +1,324 @@ + + +# Cast + +Cast operations in Comet fall into three levels of support: + +- **C (Compatible)**: The results match Apache Spark +- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By + default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark. + Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended + for production use. +- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated + cast inside the Comet pipeline. +- **N/A**: Spark does not support this cast. + +Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for +example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`. + +The tables at the end of this page cover casts between primitive types. See [Complex Types](#complex-types) for +casts involving arrays, structs, and maps. + +## ANSI Mode Support + +Enabling ANSI mode does not by itself make a cast fall back to Spark or require +`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support +level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the +[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting. + +## Whitespace Trimming in Casts from String + +Spark uses two different notions of whitespace when casting from string, and Comet reproduces +both: + +| Trim set | Cast targets | +| -------------------------------------- | --------------------------------------------------------------------------------------- | +| Bytes `0x00`-`0x20` and `0x7F` (`DEL`) | `BOOLEAN`, `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_NTZ` | +| Bytes `0x00`-`0x20` only | `FLOAT`, `DOUBLE`, `DECIMAL` | + +Neither set includes any non-ASCII whitespace, so padding a value with `U+0085`, `U+00A0`, +`U+1680`, `U+2000`-`U+200A`, `U+2028`, `U+2029`, `U+202F`, `U+205F` or `U+3000` produces `NULL` +(or raises under ANSI mode), even though those codepoints are whitespace to Unicode. Whitespace +inside a value is never trimmed. + +## String to Decimal + +Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's behavior, +including: + +- Leading and trailing bytes `0x00`-`0x20` are trimmed before parsing, matching the + `java.lang.String.trim` that Spark applies before handing the value to `BigDecimal`. +- Null bytes (`\u0000`) at the start or end of a string are therefore trimmed. Null bytes + embedded in the middle of a string produce `NULL`. +- Non-ASCII decimal digits that Java's `BigDecimal` accepts, i.e. every Unicode `Nd` digit in + the Basic Multilingual Plane (fullwidth `１２３.４５`, Arabic-Indic `١٢٣`, Devanagari `१२३`, + ...), are treated as their ASCII equivalents in both the mantissa and the exponent, so + `CAST('١٢٣.٤٥' AS DECIMAL(10,2))` returns `123.45`. Digits outside the Basic Multilingual + Plane, such as U+1D7CE MATHEMATICAL BOLD DIGIT ZERO, are rejected as they are by Spark. +- Scientific notation (e.g. `1.23E+5`) is supported. +- Special values (`inf`, `infinity`, `nan`) produce `NULL`. + +A zero with a large exponent, such as `'0e39'`, casts to `0` on Spark 4.0.5, 4.1.4, 4.2.1 and later +(SPARK-60118). Earlier releases return `NULL` for it, or raise `NUMERIC_OUT_OF_SUPPORTED_RANGE` +under ANSI mode. Comet chooses the behavior by the Spark version number. On a vendor build that +reports an older version but includes SPARK-60118, Comet returns `NULL` (or raises) where Spark +returns `0`. + +## String to Date + +Comet's native `CAST(string AS DATE)` implementation matches Apache Spark's behavior for years +between 262143 BC and 262142 AD. This range limitation comes from the underlying chrono library's +`NaiveDate` type. Spark itself supports a wider range. All three eval modes (Legacy, ANSI, Try) +are supported. + +Supported input formats match Spark exactly: + +- `yyyy`, `yyyy-[m]m`, `yyyy-[m]m-[d]d` +- Optional `T` suffix with arbitrary trailing text (e.g. `2020-01-01T12:34:56`) +- Leading/trailing bytes `0x00`-`0x20` and `0x7F` are trimmed +- Optional sign prefix (`-` for negative years) +- Leading zeros (e.g. `0002020-01-01` is year 2020) + +## Date to Timestamp + +Comet's native `CAST(date AS TIMESTAMP)` is compatible with Spark. The cast interprets each +date as midnight in the session timezone and converts to a UTC epoch value. DST transitions +are handled correctly, including spring-forward gaps (where midnight may not exist) and +fall-back ambiguity (where Comet picks the earlier/DST occurrence, matching Spark's +`LocalDate.atStartOfDay(zoneId)` behavior). + +## Date to TimestampNTZ + +Comet's native `CAST(date AS TIMESTAMP_NTZ)` is compatible with Spark. The cast is +timezone-independent: each date is converted to midnight as pure arithmetic +(`days * 86,400,000,000` microseconds) with no session timezone offset applied. The result +is the same regardless of the session timezone setting. + +## Date to Numeric Types + +In Legacy mode, `CAST(date AS INT)`, `CAST(date AS LONG)`, and casts to all other numeric +types (Boolean, Byte, Short, Float, Double, Decimal) always return `NULL`. Comet handles +this by short-circuiting to a null literal during query planning, so no native execution +is needed. In ANSI and Try modes, Spark rejects these casts at analysis time (before +execution reaches Comet). The short-circuit applies only to a top-level cast. See +[Complex Types](#complex-types) for `DATE` values inside structs and maps. + +## String to Timestamp + +Comet's native `CAST(string AS TIMESTAMP)` implementation supports all timestamp formats accepted +by Apache Spark, including ISO 8601 date-time strings, date-only strings, time-only strings +(`HH:MM:SS`), embedded timezone offsets (e.g. `+07:30`, `GMT-01:00`, `UTC`), named timezone +suffixes (e.g. `Europe/Moscow`), and the full Spark timestamp year range +(-290308 to 294247). + +## String to TimestampNTZ + +Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior. +Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in +the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time +components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`. +The result is always a wall-clock timestamp with no timezone conversion or DST adjustment. + +## TimestampNTZ Casts + +Comet supports the following `TIMESTAMP_NTZ` casts natively: + +| Cast | Compatible | Notes | +| ---------------------------------- | ---------- | ----------------------------------------------------------------- | +| `CAST(timestamp_ntz AS STRING)` | Yes | Formats local time as-is, timezone-independent | +| `CAST(timestamp_ntz AS DATE)` | Yes | Extracts the date component, timezone-independent | +| `CAST(timestamp_ntz AS TIMESTAMP)` | Yes | Interprets NTZ as local time in session TZ, converts to UTC epoch | +| `CAST(date AS TIMESTAMP_NTZ)` | Yes | Pure arithmetic, timezone-independent | +| `CAST(timestamp AS TIMESTAMP_NTZ)` | Yes | Shifts UTC epoch to local time in session TZ | +| `CAST(string AS TIMESTAMP_NTZ)` | Yes | See [String to TimestampNTZ](#string-to-timestampntz) above | + +The NTZ-to-Timestamp and Timestamp-to-NTZ casts are session-timezone-dependent (the session +timezone determines the UTC offset). All other NTZ casts are timezone-independent and produce +the same result regardless of the session timezone. + +## Date to String + +Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000 are +zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are +rendered without truncation. The cast is timezone-independent. + +## Decimal with Negative Scale to String + +Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when +`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is +disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's +code-generated cast instead of the native one. + +When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches +Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific +notation (e.g. a value of 12300 stored as `Decimal(7,-2)` with unscaled value 123 is rendered +as `"1.23E+4"`). + +## Decimal with Negative Scale to Integer and Timestamp + +Casting a `DecimalType` with a negative scale to `ByteType`, `ShortType`, `IntegerType`, +`LongType` or `TimestampType` has no native implementation, and neither does casting any of the +integer types to a negative-scale `DecimalType`. Comet reports these as unsupported, which +routes them through the JVM codegen dispatcher (Spark's own generated code) inside the Comet +pipeline, so the results match Spark exactly. + +The native kernels align scale by computing `10^|scale|` in fixed-point arithmetic, which +overflows when the scale is negative. See +[#5013](https://github.com/apache/datafusion-comet/issues/5013). + +All other cast directions to and from negative-scale decimals run natively: `FloatType` and +`DoubleType`, `StringType` in both directions (subject to the config gate described above), +casts from a negative-scale decimal to `BooleanType`, and widening to another `DecimalType`. +Casts from `BooleanType` to a `DecimalType` route through the dispatcher at any scale, which is +not specific to a negative one. + +## Complex Types + +Comet casts arrays, structs, and maps by casting each element, field, key, and value with the +rules for primitive types, using the table for the same eval mode. A complex-type cast is +therefore compatible only when every cast it contains is compatible. For example, casting +`ARRAY` to `ARRAY` has the same support level as casting `INT` to `BIGINT`, and a +struct cast with one unsupported field cast is unsupported. The rules apply at every level of +nesting, so casting `ARRAY>` to `ARRAY>` also has the same +support level as casting `INT` to `BIGINT`. + +| From | To | Support level | +| ----------- | ------------- | ---------------------------------------------------------------------------- | +| `ARRAY` | `ARRAY` | Same as `T` to `U` | +| `ARRAY` | `STRING` | Same as `T` to `STRING` | +| `STRUCT` | `STRUCT` | Compatible if every field cast is compatible. Fields are matched by position | +| `STRUCT` | `STRING` | Compatible if the cast from every field to `STRING` is compatible | +| `MAP` | `MAP` | Compatible if `K` to `K2` and `V` to `V2` are both compatible | +| `MAP` | `STRING` | Unsupported | + +The following casts are exceptions to these rules: + +- `ARRAY` can only be cast natively to `ARRAY` or `ARRAY`. Casts to other array + types are unsupported. +- A cast from a `DATE` struct field or map value to a numeric or boolean type is unsupported. In + Legacy mode these casts always return `NULL` (see [Date to Numeric Types](#date-to-numeric-types)). +- When `spark.sql.legacy.castComplexTypesToString.enabled=true`, casts from arrays, structs, and + maps to `STRING` are unsupported, because Comet's native cast only produces Spark's default + string format. + +In Try mode, a cast between maps fails with an error when a key cannot be cast, where Spark returns +a result ([#5995](https://github.com/apache/datafusion-comet/issues/5995)). This only affects key +casts that Spark accepts under `TRY_CAST`, such as narrowing `BIGINT` to `INT`. Spark rejects a +`STRING` key cast at analysis. + +## Decimal with Negative Scale to Integral Types + +Casting a `DecimalType` with a negative scale, which Spark only creates when +`spark.sql.legacy.allowNegativeScaleOfDecimal=true`, to `ByteType`, `ShortType`, `IntegerType` +or `LongType` is not supported natively in any eval mode. The native kernels divide by +`10^scale`, which overflows for a negative scale, so Comet leaves the cast to Spark. + +## Legacy Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | C | N/A | +| byte | C | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | C | C | - | C | C | C | C | C | C | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | C | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | C | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | C | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## Try Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## ANSI Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md new file mode 100644 index 00000000000..b147e6e546a --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md @@ -0,0 +1,156 @@ + + +# Date/Time Expressions + +- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is marked Incompatible and + routes through the JVM codegen dispatcher by default, producing Spark-identical results. The + native path is itself correct for dates within chrono-tz's DST horizon (approximately year 2100; + see "Date and Time Functions" below) and can be enabled by setting + `spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ inputs are handled + correctly regardless of session timezone (timezone-independent truncation). + +## Date and Time Functions + +Comet's native implementation of date and time functions may produce different results than Spark for dates +far in the future (approximately beyond year 2100). This is because Comet uses the chrono-tz library for +timezone calculations, which has limited support for Daylight Saving Time (DST) rules beyond the IANA +time zone database's explicit transitions. + +For dates within a reasonable range (approximately 1970-2100), Comet's date and time functions are compatible +with Spark. For dates beyond this range, functions that involve timezone-aware calculations (such as +`date_trunc` with timezone-aware timestamps) may produce results with incorrect DST offsets. + +If you need to process dates far in the future with accurate timezone handling, consider: + +- Using timezone-naive types (`timestamp_ntz`) when timezone conversion is not required +- Falling back to Spark for these specific operations + +### Timezone Database Versions + +Comet's native code converts between instants and local time with the IANA timezone database that +chrono-tz compiles into the Comet library. Spark uses the JVM's timezone database (`tzdb.dat`), whose +version depends on the JDK build and on whether its timezone data has been updated. When the two versions +have different rules for a timezone, local times that Comet computes natively for that timezone can differ +from Spark's for the affected dates. That covers `hour`, casts between timestamps and strings or dates, +`date_trunc`, and parsing strings as timestamps. Comet logs a warning at startup when the two versions +differ. + + + +## ConvertTimezone + +By default, `ConvertTimezone` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ConvertTimezone.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## DateFormatClass + +The following differences from Spark are always present and do not require any additional configuration: + +- Format strings in a curated allow-list run natively via DataFusion's `to_char` for UTC sessions. Other format strings (including non-literal formats), as well as non-UTC sessions, route through Spark's own `DateFormatClass.doGenCode` via the Arrow-direct codegen dispatcher when `spark.comet.exec.scalaUDF.codegen.enabled=true` (the default). When the codegen dispatcher is disabled the operator falls back to Spark in those cases. + +By default, `DateFormatClass` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.DateFormatClass.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-UTC timezones may produce different results than Spark + +## Days + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DateType` and `TimestampType` inputs are supported. `TimestampNTZType` is not supported. + +## FromUTCTimestamp + +By default, `FromUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## FromUnixTime + +By default, `FromUnixTime` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUnixTime.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's valid timestamp range differs from Spark (https://github.com/apache/datafusion/issues/16594) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the default datetime format pattern `yyyy-MM-dd HH:mm:ss` is supported + +## Hours + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `TimestampType` and `TimestampNTZType` inputs are supported. + +## MakeInterval + +The following differences from Spark are always present and do not require any additional configuration: + +- Both the default JVM codegen-dispatch path and the native path currently limit the elapsed-time component to about 292 years in either direction. This only affects extreme intervals and is tracked in [#5279](https://github.com/apache/datafusion-comet/issues/5279). + +By default, `MakeInterval` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MakeInterval.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- The native implementation converts seconds to `Float64`, which can lose microsecond precision, and stores time in nanoseconds, which overflows for large time components (hours, minutes, seconds) that Spark can represent. + +## PreciseTimestampConversion + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only reinterprets between TimestampType/TimestampNTZType and LongType are supported. + +## SecondsToTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `DecimalType`, `ByteType`, and `ShortType` inputs + +## ToUTCTimestamp + +By default, `ToUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ToUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## TruncDate + +By default, `TruncDate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncDate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-literal format strings will throw an exception instead of returning NULL + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week + +## TruncTimestamp + +By default, `TruncTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Produces incorrect results when used with non-UTC timezones. Compatible when timezone is UTC. (https://github.com/apache/datafusion-comet/issues/2649) +- Non-literal format strings will throw an exception instead of returning NULL + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week, day, dd, hour, minute, second, millisecond, microsecond + +## UnixTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- String inputs, including collated strings, have no native implementation. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/map.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/map.md new file mode 100644 index 00000000000..d16f9df138b --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/map.md @@ -0,0 +1,66 @@ + + +# Map Expressions + +## MapSort (Spark 4.0+) + +Spark 4.0 inserts `MapSort` to normalize map values when they appear in shuffle hash partitioning +keys, in `try_element_at`, and in other contexts where map ordering must be deterministic. Comet +runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0. + +When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose +keys contain `Float` or `Double`. Arrow's sort uses +IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for +`NaN` and `-0.0`. + + + +## MapFromArrays + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## MapFromEntries + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromEntries.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `BinaryType` is not supported as a map key in `map_from_entries` +- `BinaryType` is not supported as a map value in `map_from_entries` +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## StringToMap + +By default, `StringToMap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringToMap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `spark.sql.legacy.truncateForEmptyRegexSplit` is enabled, so trailing empty split entries may differ from Spark. +- `str_to_map` does not support non-UTF8_BINARY collations on the input string or delimiters. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/math.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/math.md new file mode 100644 index 00000000000..b0e2ec1d3b1 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/math.md @@ -0,0 +1,79 @@ + + +# Math Expressions + + + +## Abs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `INTERVAL YEAR TO MONTH` and `INTERVAL DAY TO SECOND` inputs + +## Divide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Greatest + +For applicable cases that are not selected for native execution automatically, `Greatest` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Greatest.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native greatest compares raw bytes + +## IntegralDivide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Least + +For applicable cases that are not selected for native execution automatically, `Least` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Least.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native least compares raw bytes + +## Rand + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Randn + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Round + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Float and double inputs. Spark rounds them through a BigDecimal built from `java.lang.Double.toString()` rather than from the exact binary value, and that shortened decimal string can round differently than the value it came from +- Negative-scale decimal inputs, which are only creatable with spark.sql.legacy.allowNegativeScaleOfDecimal=true + +## Signum + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DoubleType` is supported + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/misc.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/misc.md new file mode 100644 index 00000000000..021051487b5 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/misc.md @@ -0,0 +1,72 @@ + + +# Miscellaneous Expressions + + + +## CheckOverflow + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DecimalType` is supported + +## Literal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for literal values + +## MakeDecimal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `LongType` input is supported + +## ScalarSubquery + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for scalar subquery results + +## SortOrder + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- A nested sort key with nondefault root null placement must be deterministic: the wrapper can evaluate the key twice, for isnull(key) and its value, which could disagree + +## StaticInvoke + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 10 is one such value: the result has 19 digits. The other `truncate` input types, and `bucket` on decimals, are unaffected. + +## ToPrettyString + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Falls back to Spark when the input type cannot be cast to string. + +## Uuid + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- uuid requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/string.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/string.md new file mode 100644 index 00000000000..bdd6d1f9e20 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/string.md @@ -0,0 +1,217 @@ + + +# String Expressions + + + +## BitLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## Concat + +By default, `Concat` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Concat.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- concat does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- CONCAT supports only string input parameters + +## ConcatWs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- all arguments are foldable + +## Contains + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## EndsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## GetJsonObject + +By default, `GetJsonObject` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.GetJsonObject.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark allows single-quoted JSON and unescaped control characters which Comet does not support +- Very long numbers near Jackson's 1000-digit limit can depend on Spark's recycled parser buffer, which Comet cannot reproduce from the input alone +- Selected JSON integers outside the 64-bit range and very long numbers can lose precision or fail during Comet's native JSON materialization +- Some selected floating-point values can differ from Spark at decimal parsing or Java-version formatting boundaries +- When a returned object or array contains duplicate keys, Spark preserves them while Comet's native JSON materialization keeps only the last value + +## InitCap + +By default, `InitCap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.InitCap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Treats hyphen as a word separator (e.g. `robert rose-smith` produces `Robert Rose-Smith` instead of Spark's `Robert Rose-smith`) (https://github.com/apache/datafusion-comet/issues/1052) + +## Levenshtein + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-default (non-UTF8_BINARY) collated input. The native kernel compares raw bytes, so collation-aware comparison has no native path. + +## Like + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- LIKE with a custom escape character (only `\` is supported natively) +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## Lower + +By default, `Lower` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + +## OctetLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## RLike + +The following cases use Comet's native implementation by default: + +- A `UTF8_BINARY` literal pattern admitted by the [plan-time compatibility analyzer](../../regex.md#when-the-rust-engine-is-safe) is evaluated natively by default. + +For applicable cases that are not selected for native execution automatically, `RLike` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.RLike.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- For applicable literal patterns outside the automatically admitted subset, the native Rust regex engine may behave differently from Java regex. + +## RegExpReplace + +By default, `RegExpReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.RegExpReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regexp pattern may not be compatible with Spark + +## Reverse + +By default, `Reverse` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Reverse.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- native reverse does not support arrays whose element type contains binary, struct, or map +- reverse does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- reverse has no native implementation for binary input + +## StartsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## StringInstr + +For applicable cases that are not selected for native execution automatically, `StringInstr` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringInstr.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native instr compares raw bytes + +## StringLPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRepeat + +The following differences from Spark are always present and do not require any additional configuration: + +- A negative argument for the number of times to repeat throws an exception instead of returning an empty string as Spark does + +## StringReplace + +The following cases use Comet's native implementation by default: + +- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default. + +For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast + +## StringSplit + +By default, `StringSplit` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringSplit.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regex engine differences between Java and Rust + +## StringTranslate + +By default, `StringTranslate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel + +## StringTrim + +For applicable cases that are not selected for native execution automatically, `StringTrim` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrim.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native trim with a trim string compares raw bytes + +## StringTrimLeft + +For applicable cases that are not selected for native execution automatically, `StringTrimLeft` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimLeft.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native ltrim with a trim string compares raw bytes + +## StringTrimRight + +For applicable cases that are not selected for native execution automatically, `StringTrimRight` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimRight.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native rtrim with a trim string compares raw bytes + +## SubstringIndex + +For applicable cases that are not selected for native execution automatically, `SubstringIndex` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.SubstringIndex.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native substring_index compares raw bytes + +## UnBase64 + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- unbase64 with failOnError = true uses stricter RFC 4648 validation that is not yet implemented natively +- unbase64 with a non-trivial child expression uses the JVM codegen dispatcher to preserve Spark's short-circuit evaluation (native path is limited to column and literal children) + +## Upper + +By default, `Upper` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/struct.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/struct.md new file mode 100644 index 00000000000..c530b12a1a8 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/struct.md @@ -0,0 +1,52 @@ + + +# Struct Expressions + + + +## CreateNamedStruct + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `CreateNamedStruct` with duplicate field names is not supported + +## JsonToStructs + +By default, `JsonToStructs` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.JsonToStructs.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Partially implemented and not comprehensively tested + +## StructsToCsv + +By default, `StructsToCsv` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToCsv.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Date, Timestamp, TimestampNTZ, and Binary data types may produce different results (https://github.com/apache/datafusion-comet/issues/3232) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Complex types (arrays, maps, structs) in the schema are not supported + +## StructsToJson + +By default, `StructsToJson` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToJson.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Does not support `+Infinity` and `-Infinity` for numeric types (float, double). (https://github.com/apache/datafusion-comet/issues/3016) + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/url.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/url.md new file mode 100644 index 00000000000..765e12941fd --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/url.md @@ -0,0 +1,23 @@ + + +# URL Expressions + + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/aggregate.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/aggregate.md new file mode 100644 index 00000000000..093a70ac586 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/aggregate.md @@ -0,0 +1,156 @@ + + +# Aggregate Expressions + + + +## ApproximatePercentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The percentage argument must be a foldable literal. +- The accuracy argument must be a foldable literal. +- Only byte, short, int, long, float, and double input types are supported. + +## Average + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- YearMonthIntervalType and DayTimeIntervalType inputs are not supported + +## CollectSet + +The following incompatibilities cause `CollectSet` to fall back to Spark by default. Set `spark.comet.expression.CollectSet.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Before Spark 4.2, Comet deduplicates NaN values (treats `NaN == NaN`) while Spark treats each NaN as a distinct value. Comet treats -0.0 and 0.0 as distinct while Spark treats them as equal. When `spark.comet.exec.strictFloatingPoint=true`, `collect_set` on floating-point types falls back to Spark on those versions unless `spark.comet.expression.CollectSet.allowIncompatible=true` is set. + +## First + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## HllSketchAgg + +The following incompatibilities cause `HllSketchAgg` to fall back to Spark by default. Set `spark.comet.expression.HllSketchAgg.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The lgConfigK argument must be a foldable literal. +- The lgConfigK argument must be in the range [4, 21]. +- Only int, long, string, and binary input types are supported. + +## HllUnionAgg + +The following incompatibilities cause `HllUnionAgg` to fall back to Spark by default. Set `spark.comet.expression.HllUnionAgg.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark. +- Errors surface as plain Comet execution errors rather than Spark's SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail. +- An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The allowDifferentLgConfigK argument must be a foldable literal. + +## HyperLogLogPlusPlus + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input type must be boolean, integral, floating-point, decimal, date/time, string, or binary +- `DecimalType` with precision > 18 is not supported +- Collated (non-UTF8_BINARY) strings are not supported + +## Last + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## ListAgg + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` inputs are not supported. +- `WITHIN GROUP (ORDER BY ...)` is not supported. +- Non-literal delimiters are not supported. +- Non-default string collations are not supported. + +## Max + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MaxBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the maximum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Min + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MinBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the minimum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Mode + +The following incompatibilities cause `Mode` to fall back to Spark by default. Set `spark.comet.expression.Mode.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- mode breaks ties non-deterministically in Spark (the result depends on JVM hash-map iteration order); Comet returns the smallest of the tied values instead (https://github.com/apache/datafusion-comet/issues/3970) + +## Percentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- An array of percentages is not supported. +- The percentage argument must be a literal. +- A frequency argument is not supported. +- Descending order in `WITHIN GROUP (ORDER BY ... DESC)` is not supported. +- Only numeric input types are supported. + +## PivotFirst + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Value data type other than Boolean, Byte, Short, Int, Long, Float, Double or Decimal +- Pivot column of a type other than Boolean, Byte, Short, Int, Long, Float, Double, Decimal, String (UTF8_BINARY collation), Date, Timestamp or TimestampNTZ, which includes the struct that a pivot on more than one column builds +- Pivot values list is empty +- Pivot values list contains duplicates, including `0.0` next to `-0.0` + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/array.md new file mode 100644 index 00000000000..3e94336b308 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/array.md @@ -0,0 +1,113 @@ + + +# Array Expressions + + + +## ArrayAppend + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- array_append with a nullable array and a nondeterministic operand such as `rand()`: the native NULL guard evaluates the operands more than once, on subsets of the rows + +## ArrayContains + +For applicable cases that are not selected for native execution automatically, `ArrayContains` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayContains.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares array elements with ordering.equiv, so -0.0 matches +0.0 and all NaNs match each other; Comet's native array_contains applies that only to flat FLOAT and DOUBLE elements, and nested float elements stay on Spark's comparison +- array_contains evaluates its value eagerly, while Spark skips it when the array is null +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_contains compares raw bytes + +## ArrayDistinct + +The following incompatibilities cause `ArrayDistinct` to fall back to Spark by default. Set `spark.comet.expression.ArrayDistinct.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_distinct compares raw bytes + +## ArrayExcept + +By default, `ArrayExcept` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayExcept.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Null handling and ordering may differ from Spark + +## ArrayIntersect + +By default, `ArrayIntersect` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayIntersect.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Result array element order may differ from Spark when the right array is longer than the left (DataFusion probes the longer side). +- array_intersect does not propagate non-UTF8_BINARY collations to the output array elements (https://github.com/apache/datafusion-comet/issues/2190) + +## ArrayJoin + +By default, `ArrayJoin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayJoin.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190) +- array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178) + +## ArrayMax + +For applicable cases that are not selected for native execution automatically, `ArrayMax` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMax.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayMin + +For applicable cases that are not selected for native execution automatically, `ArrayMin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMin.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayUnion + +The following incompatibilities cause `ArrayUnion` to fall back to Spark by default. Set `spark.comet.expression.ArrayUnion.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_union compares raw bytes + +## ArraysOverlap + +For applicable cases that are not selected for native execution automatically, `ArraysOverlap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArraysOverlap.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native arrays_overlap compares raw bytes + +## ArraysZip + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all input data types are supported; falls back to Spark for unsupported types + +## Sequence + +The following differences from Spark are always present and do not require any additional configuration: + +- When `stop - start` overflows a long but the sequence is short, Spark 4.0.5, 4.1.4, 4.2.1 and later return its elements (SPARK-58821), and earlier releases raise `INTERNAL_ERROR`. Comet chooses the behavior by the Spark version number. On a vendor build that reports an older version but includes SPARK-58821, Comet raises the error where Spark returns the elements. + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- date and timestamp element types run through the JVM codegen dispatcher +- sequence arguments must be literals or column references; other shapes run through the JVM codegen dispatcher to preserve Spark's per-row null short-circuit + +## Shuffle + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- shuffle requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/cast.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/cast.md new file mode 100644 index 00000000000..c99186ccaff --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/cast.md @@ -0,0 +1,324 @@ + + +# Cast + +Cast operations in Comet fall into three levels of support: + +- **C (Compatible)**: The results match Apache Spark +- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By + default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark. + Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended + for production use. +- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated + cast inside the Comet pipeline. +- **N/A**: Spark does not support this cast. + +Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for +example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`. + +The tables at the end of this page cover casts between primitive types. See [Complex Types](#complex-types) for +casts involving arrays, structs, and maps. + +## ANSI Mode Support + +Enabling ANSI mode does not by itself make a cast fall back to Spark or require +`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support +level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the +[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting. + +## Whitespace Trimming in Casts from String + +Spark uses two different notions of whitespace when casting from string, and Comet reproduces +both: + +| Trim set | Cast targets | +| -------------------------------------- | --------------------------------------------------------------------------------------- | +| Bytes `0x00`-`0x20` and `0x7F` (`DEL`) | `BOOLEAN`, `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_NTZ` | +| Bytes `0x00`-`0x20` only | `FLOAT`, `DOUBLE`, `DECIMAL` | + +Neither set includes any non-ASCII whitespace, so padding a value with `U+0085`, `U+00A0`, +`U+1680`, `U+2000`-`U+200A`, `U+2028`, `U+2029`, `U+202F`, `U+205F` or `U+3000` produces `NULL` +(or raises under ANSI mode), even though those codepoints are whitespace to Unicode. Whitespace +inside a value is never trimmed. + +## String to Decimal + +Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's behavior, +including: + +- Leading and trailing bytes `0x00`-`0x20` are trimmed before parsing, matching the + `java.lang.String.trim` that Spark applies before handing the value to `BigDecimal`. +- Null bytes (`\u0000`) at the start or end of a string are therefore trimmed. Null bytes + embedded in the middle of a string produce `NULL`. +- Non-ASCII decimal digits that Java's `BigDecimal` accepts, i.e. every Unicode `Nd` digit in + the Basic Multilingual Plane (fullwidth `１２３.４５`, Arabic-Indic `١٢٣`, Devanagari `१२३`, + ...), are treated as their ASCII equivalents in both the mantissa and the exponent, so + `CAST('١٢٣.٤٥' AS DECIMAL(10,2))` returns `123.45`. Digits outside the Basic Multilingual + Plane, such as U+1D7CE MATHEMATICAL BOLD DIGIT ZERO, are rejected as they are by Spark. +- Scientific notation (e.g. `1.23E+5`) is supported. +- Special values (`inf`, `infinity`, `nan`) produce `NULL`. + +A zero with a large exponent, such as `'0e39'`, casts to `0` on Spark 4.0.5, 4.1.4, 4.2.1 and later +(SPARK-60118). Earlier releases return `NULL` for it, or raise `NUMERIC_OUT_OF_SUPPORTED_RANGE` +under ANSI mode. Comet chooses the behavior by the Spark version number. On a vendor build that +reports an older version but includes SPARK-60118, Comet returns `NULL` (or raises) where Spark +returns `0`. + +## String to Date + +Comet's native `CAST(string AS DATE)` implementation matches Apache Spark's behavior for years +between 262143 BC and 262142 AD. This range limitation comes from the underlying chrono library's +`NaiveDate` type. Spark itself supports a wider range. All three eval modes (Legacy, ANSI, Try) +are supported. + +Supported input formats match Spark exactly: + +- `yyyy`, `yyyy-[m]m`, `yyyy-[m]m-[d]d` +- Optional `T` suffix with arbitrary trailing text (e.g. `2020-01-01T12:34:56`) +- Leading/trailing bytes `0x00`-`0x20` and `0x7F` are trimmed +- Optional sign prefix (`-` for negative years) +- Leading zeros (e.g. `0002020-01-01` is year 2020) + +## Date to Timestamp + +Comet's native `CAST(date AS TIMESTAMP)` is compatible with Spark. The cast interprets each +date as midnight in the session timezone and converts to a UTC epoch value. DST transitions +are handled correctly, including spring-forward gaps (where midnight may not exist) and +fall-back ambiguity (where Comet picks the earlier/DST occurrence, matching Spark's +`LocalDate.atStartOfDay(zoneId)` behavior). + +## Date to TimestampNTZ + +Comet's native `CAST(date AS TIMESTAMP_NTZ)` is compatible with Spark. The cast is +timezone-independent: each date is converted to midnight as pure arithmetic +(`days * 86,400,000,000` microseconds) with no session timezone offset applied. The result +is the same regardless of the session timezone setting. + +## Date to Numeric Types + +In Legacy mode, `CAST(date AS INT)`, `CAST(date AS LONG)`, and casts to all other numeric +types (Boolean, Byte, Short, Float, Double, Decimal) always return `NULL`. Comet handles +this by short-circuiting to a null literal during query planning, so no native execution +is needed. In ANSI and Try modes, Spark rejects these casts at analysis time (before +execution reaches Comet). The short-circuit applies only to a top-level cast. See +[Complex Types](#complex-types) for `DATE` values inside structs and maps. + +## String to Timestamp + +Comet's native `CAST(string AS TIMESTAMP)` implementation supports all timestamp formats accepted +by Apache Spark, including ISO 8601 date-time strings, date-only strings, time-only strings +(`HH:MM:SS`), embedded timezone offsets (e.g. `+07:30`, `GMT-01:00`, `UTC`), named timezone +suffixes (e.g. `Europe/Moscow`), and the full Spark timestamp year range +(-290308 to 294247). + +## String to TimestampNTZ + +Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior. +Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in +the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time +components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`. +The result is always a wall-clock timestamp with no timezone conversion or DST adjustment. + +## TimestampNTZ Casts + +Comet supports the following `TIMESTAMP_NTZ` casts natively: + +| Cast | Compatible | Notes | +| ---------------------------------- | ---------- | ----------------------------------------------------------------- | +| `CAST(timestamp_ntz AS STRING)` | Yes | Formats local time as-is, timezone-independent | +| `CAST(timestamp_ntz AS DATE)` | Yes | Extracts the date component, timezone-independent | +| `CAST(timestamp_ntz AS TIMESTAMP)` | Yes | Interprets NTZ as local time in session TZ, converts to UTC epoch | +| `CAST(date AS TIMESTAMP_NTZ)` | Yes | Pure arithmetic, timezone-independent | +| `CAST(timestamp AS TIMESTAMP_NTZ)` | Yes | Shifts UTC epoch to local time in session TZ | +| `CAST(string AS TIMESTAMP_NTZ)` | Yes | See [String to TimestampNTZ](#string-to-timestampntz) above | + +The NTZ-to-Timestamp and Timestamp-to-NTZ casts are session-timezone-dependent (the session +timezone determines the UTC offset). All other NTZ casts are timezone-independent and produce +the same result regardless of the session timezone. + +## Date to String + +Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000 are +zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are +rendered without truncation. The cast is timezone-independent. + +## Decimal with Negative Scale to String + +Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when +`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is +disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's +code-generated cast instead of the native one. + +When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches +Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific +notation (e.g. a value of 12300 stored as `Decimal(7,-2)` with unscaled value 123 is rendered +as `"1.23E+4"`). + +## Decimal with Negative Scale to Integer and Timestamp + +Casting a `DecimalType` with a negative scale to `ByteType`, `ShortType`, `IntegerType`, +`LongType` or `TimestampType` has no native implementation, and neither does casting any of the +integer types to a negative-scale `DecimalType`. Comet reports these as unsupported, which +routes them through the JVM codegen dispatcher (Spark's own generated code) inside the Comet +pipeline, so the results match Spark exactly. + +The native kernels align scale by computing `10^|scale|` in fixed-point arithmetic, which +overflows when the scale is negative. See +[#5013](https://github.com/apache/datafusion-comet/issues/5013). + +All other cast directions to and from negative-scale decimals run natively: `FloatType` and +`DoubleType`, `StringType` in both directions (subject to the config gate described above), +casts from a negative-scale decimal to `BooleanType`, and widening to another `DecimalType`. +Casts from `BooleanType` to a `DecimalType` route through the dispatcher at any scale, which is +not specific to a negative one. + +## Complex Types + +Comet casts arrays, structs, and maps by casting each element, field, key, and value with the +rules for primitive types, using the table for the same eval mode. A complex-type cast is +therefore compatible only when every cast it contains is compatible. For example, casting +`ARRAY` to `ARRAY` has the same support level as casting `INT` to `BIGINT`, and a +struct cast with one unsupported field cast is unsupported. The rules apply at every level of +nesting, so casting `ARRAY>` to `ARRAY>` also has the same +support level as casting `INT` to `BIGINT`. + +| From | To | Support level | +| ----------- | ------------- | ---------------------------------------------------------------------------- | +| `ARRAY` | `ARRAY` | Same as `T` to `U` | +| `ARRAY` | `STRING` | Same as `T` to `STRING` | +| `STRUCT` | `STRUCT` | Compatible if every field cast is compatible. Fields are matched by position | +| `STRUCT` | `STRING` | Compatible if the cast from every field to `STRING` is compatible | +| `MAP` | `MAP` | Compatible if `K` to `K2` and `V` to `V2` are both compatible | +| `MAP` | `STRING` | Unsupported | + +The following casts are exceptions to these rules: + +- `ARRAY` can only be cast natively to `ARRAY` or `ARRAY`. Casts to other array + types are unsupported. +- A cast from a `DATE` struct field or map value to a numeric or boolean type is unsupported. In + Legacy mode these casts always return `NULL` (see [Date to Numeric Types](#date-to-numeric-types)). +- When `spark.sql.legacy.castComplexTypesToString.enabled=true`, casts from arrays, structs, and + maps to `STRING` are unsupported, because Comet's native cast only produces Spark's default + string format. + +In Try mode, a cast between maps fails with an error when a key cannot be cast, where Spark returns +a result ([#5995](https://github.com/apache/datafusion-comet/issues/5995)). This only affects key +casts that Spark accepts under `TRY_CAST`, such as narrowing `BIGINT` to `INT`. Spark rejects a +`STRING` key cast at analysis. + +## Decimal with Negative Scale to Integral Types + +Casting a `DecimalType` with a negative scale, which Spark only creates when +`spark.sql.legacy.allowNegativeScaleOfDecimal=true`, to `ByteType`, `ShortType`, `IntegerType` +or `LongType` is not supported natively in any eval mode. The native kernels divide by +`10^scale`, which overflows for a negative scale, so Comet leaves the cast to Spark. + +## Legacy Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | C | N/A | +| byte | C | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | C | C | - | C | C | C | C | C | C | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | C | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | C | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | C | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## Try Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## ANSI Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md new file mode 100644 index 00000000000..68eb6b4a8a0 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md @@ -0,0 +1,166 @@ + + +# Date/Time Expressions + +- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is marked Incompatible and + routes through the JVM codegen dispatcher by default, producing Spark-identical results. The + native path is itself correct for dates within chrono-tz's DST horizon (approximately year 2100; + see "Date and Time Functions" below) and can be enabled by setting + `spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ inputs are handled + correctly regardless of session timezone (timezone-independent truncation). + +## Date and Time Functions + +Comet's native implementation of date and time functions may produce different results than Spark for dates +far in the future (approximately beyond year 2100). This is because Comet uses the chrono-tz library for +timezone calculations, which has limited support for Daylight Saving Time (DST) rules beyond the IANA +time zone database's explicit transitions. + +For dates within a reasonable range (approximately 1970-2100), Comet's date and time functions are compatible +with Spark. For dates beyond this range, functions that involve timezone-aware calculations (such as +`date_trunc` with timezone-aware timestamps) may produce results with incorrect DST offsets. + +If you need to process dates far in the future with accurate timezone handling, consider: + +- Using timezone-naive types (`timestamp_ntz`) when timezone conversion is not required +- Falling back to Spark for these specific operations + +### Timezone Database Versions + +Comet's native code converts between instants and local time with the IANA timezone database that +chrono-tz compiles into the Comet library. Spark uses the JVM's timezone database (`tzdb.dat`), whose +version depends on the JDK build and on whether its timezone data has been updated. When the two versions +have different rules for a timezone, local times that Comet computes natively for that timezone can differ +from Spark's for the affected dates. That covers `hour`, casts between timestamps and strings or dates, +`date_trunc`, and parsing strings as timestamps. Comet logs a warning at startup when the two versions +differ. + + + +## ConvertTimezone + +By default, `ConvertTimezone` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ConvertTimezone.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. +- convert_timezone does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +## DateFormatClass + +The following differences from Spark are always present and do not require any additional configuration: + +- Format strings in a curated allow-list run natively via DataFusion's `to_char` for UTC sessions. Other format strings (including non-literal formats), as well as non-UTC sessions, route through Spark's own `DateFormatClass.doGenCode` via the Arrow-direct codegen dispatcher when `spark.comet.exec.scalaUDF.codegen.enabled=true` (the default). When the codegen dispatcher is disabled the operator falls back to Spark in those cases. + +By default, `DateFormatClass` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.DateFormatClass.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-UTC timezones may produce different results than Spark +- date_format does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +## Days + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DateType` and `TimestampType` inputs are supported. `TimestampNTZType` is not supported. + +## FromUTCTimestamp + +By default, `FromUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## FromUnixTime + +By default, `FromUnixTime` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUnixTime.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's valid timestamp range differs from Spark (https://github.com/apache/datafusion/issues/16594) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the default datetime format pattern `yyyy-MM-dd HH:mm:ss` is supported + +## Hours + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `TimestampType` and `TimestampNTZType` inputs are supported. + +## MakeInterval + +The following differences from Spark are always present and do not require any additional configuration: + +- Both the default JVM codegen-dispatch path and the native path currently limit the elapsed-time component to about 292 years in either direction. This only affects extreme intervals and is tracked in [#5279](https://github.com/apache/datafusion-comet/issues/5279). + +By default, `MakeInterval` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MakeInterval.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- The native implementation converts seconds to `Float64`, which can lose microsecond precision, and stores time in nanoseconds, which overflows for large time components (hours, minutes, seconds) that Spark can represent. + +## NextDay + +By default, `NextDay` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.NextDay.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- next_day does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +## PreciseTimestampConversion + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only reinterprets between TimestampType/TimestampNTZType and LongType are supported. + +## SecondsToTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `DecimalType`, `ByteType`, and `ShortType` inputs + +## ToUTCTimestamp + +By default, `ToUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ToUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## TruncDate + +By default, `TruncDate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncDate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-literal format strings will throw an exception instead of returning NULL +- trunc does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week + +## TruncTimestamp + +By default, `TruncTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Produces incorrect results when used with non-UTC timezones. Compatible when timezone is UTC. (https://github.com/apache/datafusion-comet/issues/2649) +- Non-literal format strings will throw an exception instead of returning NULL +- date_trunc does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week, day, dd, hour, minute, second, millisecond, microsecond + +## UnixTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- String inputs, including collated strings, have no native implementation. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/map.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/map.md new file mode 100644 index 00000000000..ece843d1a75 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/map.md @@ -0,0 +1,76 @@ + + +# Map Expressions + +## MapSort (Spark 4.0+) + +Spark 4.0 inserts `MapSort` to normalize map values when they appear in shuffle hash partitioning +keys, in `try_element_at`, and in other contexts where map ordering must be deterministic. Comet +runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0. + +When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose +keys contain `Float` or `Double`. Arrow's sort uses +IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for +`NaN` and `-0.0`. + + + +## MapFromArrays + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## MapFromEntries + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromEntries.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `BinaryType` is not supported as a map key in `map_from_entries` +- `BinaryType` is not supported as a map value in `map_from_entries` +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## MapSort + +The following incompatibilities cause `MapSort` to fall back to Spark by default. Set `spark.comet.expression.MapSort.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- MapSort on floating-point keys is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- MapSort is unsupported for non-scalar key types (struct, array, map, etc.). + +## StringToMap + +By default, `StringToMap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringToMap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `spark.sql.legacy.truncateForEmptyRegexSplit` is enabled, so trailing empty split entries may differ from Spark. +- `str_to_map` does not support non-UTF8_BINARY collations on the input string or delimiters. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/math.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/math.md new file mode 100644 index 00000000000..b0e2ec1d3b1 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/math.md @@ -0,0 +1,79 @@ + + +# Math Expressions + + + +## Abs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `INTERVAL YEAR TO MONTH` and `INTERVAL DAY TO SECOND` inputs + +## Divide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Greatest + +For applicable cases that are not selected for native execution automatically, `Greatest` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Greatest.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native greatest compares raw bytes + +## IntegralDivide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Least + +For applicable cases that are not selected for native execution automatically, `Least` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Least.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native least compares raw bytes + +## Rand + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Randn + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Round + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Float and double inputs. Spark rounds them through a BigDecimal built from `java.lang.Double.toString()` rather than from the exact binary value, and that shortened decimal string can round differently than the value it came from +- Negative-scale decimal inputs, which are only creatable with spark.sql.legacy.allowNegativeScaleOfDecimal=true + +## Signum + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DoubleType` is supported + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/misc.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/misc.md new file mode 100644 index 00000000000..f477a93886d --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/misc.md @@ -0,0 +1,92 @@ + + +# Miscellaneous Expressions + + + +## CheckOverflow + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DecimalType` is supported + +## HllSketchEstimate + +The following incompatibilities cause `HllSketchEstimate` to fall back to Spark by default. Set `spark.comet.expression.HllSketchEstimate.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL estimates may differ slightly from Spark. +- Errors surface as plain Comet execution errors rather than Spark's SparkRuntimeException with condition HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail. +- An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere. + +## HllUnion + +The following incompatibilities cause `HllUnion` to fall back to Spark by default. Set `spark.comet.expression.HllUnion.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark. +- Errors surface as plain Comet execution errors rather than Spark's SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail. +- An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The allowDifferentLgConfigK argument must be a foldable literal. + +## Literal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for literal values + +## MakeDecimal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `LongType` input is supported + +## ScalarSubquery + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for scalar subquery results + +## SortOrder + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- A nested sort key with nondefault root null placement must be deterministic: the wrapper can evaluate the key twice, for isnull(key) and its value, which could disagree + +## StaticInvoke + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 10 is one such value: the result has 19 digits. The other `truncate` input types, and `bucket` on decimals, are unaffected. + +## ToPrettyString + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Falls back to Spark when the input type cannot be cast to string. + +## Uuid + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- uuid requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/string.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/string.md new file mode 100644 index 00000000000..57d39cc2678 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/string.md @@ -0,0 +1,225 @@ + + +# String Expressions + + + +## BitLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## Concat + +By default, `Concat` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Concat.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- concat does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- CONCAT supports only string input parameters + +## ConcatWs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- all arguments are foldable + +## Contains + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## EndsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## GetJsonObject + +By default, `GetJsonObject` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.GetJsonObject.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark allows single-quoted JSON and unescaped control characters which Comet does not support +- Very long numbers near Jackson's 1000-digit limit can depend on Spark's recycled parser buffer, which Comet cannot reproduce from the input alone +- Selected JSON integers outside the 64-bit range and very long numbers can lose precision or fail during Comet's native JSON materialization +- Some selected floating-point values can differ from Spark at decimal parsing or Java-version formatting boundaries +- When a returned object or array contains duplicate keys, Spark preserves them while Comet's native JSON materialization keeps only the last value + +## InitCap + +By default, `InitCap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.InitCap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Treats hyphen as a word separator (e.g. `robert rose-smith` produces `Robert Rose-Smith` instead of Spark's `Robert Rose-smith`) (https://github.com/apache/datafusion-comet/issues/1052) + +## Levenshtein + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-default (non-UTF8_BINARY) collated input. The native kernel compares raw bytes, so collation-aware comparison has no native path. + +## Like + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- LIKE with a custom escape character (only `\` is supported natively) +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## Lower + +By default, `Lower` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + +## OctetLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## RLike + +The following cases use Comet's native implementation by default: + +- A `UTF8_BINARY` literal pattern admitted by the [plan-time compatibility analyzer](../../regex.md#when-the-rust-engine-is-safe) is evaluated natively by default. + +For applicable cases that are not selected for native execution automatically, `RLike` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.RLike.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- For applicable literal patterns outside the automatically admitted subset, the native Rust regex engine may behave differently from Java regex. + +## RandStr + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `length` must be a literal value +- `length` must be non-negative +- `seed` must be a literal value + +## RegExpReplace + +By default, `RegExpReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.RegExpReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regexp pattern may not be compatible with Spark + +## Reverse + +By default, `Reverse` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Reverse.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- native reverse does not support arrays whose element type contains binary, struct, or map +- reverse does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- reverse has no native implementation for binary input + +## StartsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## StringInstr + +For applicable cases that are not selected for native execution automatically, `StringInstr` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringInstr.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native instr compares raw bytes + +## StringLPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRepeat + +The following differences from Spark are always present and do not require any additional configuration: + +- A negative argument for the number of times to repeat throws an exception instead of returning an empty string as Spark does + +## StringReplace + +The following cases use Comet's native implementation by default: + +- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default. + +For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast + +## StringSplit + +By default, `StringSplit` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringSplit.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regex engine differences between Java and Rust + +## StringTranslate + +By default, `StringTranslate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel + +## StringTrim + +For applicable cases that are not selected for native execution automatically, `StringTrim` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrim.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native trim with a trim string compares raw bytes + +## StringTrimLeft + +For applicable cases that are not selected for native execution automatically, `StringTrimLeft` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimLeft.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native ltrim with a trim string compares raw bytes + +## StringTrimRight + +For applicable cases that are not selected for native execution automatically, `StringTrimRight` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimRight.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native rtrim with a trim string compares raw bytes + +## SubstringIndex + +For applicable cases that are not selected for native execution automatically, `SubstringIndex` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.SubstringIndex.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native substring_index compares raw bytes + +## UnBase64 + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- unbase64 with failOnError = true uses stricter RFC 4648 validation that is not yet implemented natively +- unbase64 with a non-trivial child expression uses the JVM codegen dispatcher to preserve Spark's short-circuit evaluation (native path is limited to column and literal children) + +## Upper + +By default, `Upper` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/struct.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/struct.md new file mode 100644 index 00000000000..c530b12a1a8 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/struct.md @@ -0,0 +1,52 @@ + + +# Struct Expressions + + + +## CreateNamedStruct + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `CreateNamedStruct` with duplicate field names is not supported + +## JsonToStructs + +By default, `JsonToStructs` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.JsonToStructs.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Partially implemented and not comprehensively tested + +## StructsToCsv + +By default, `StructsToCsv` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToCsv.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Date, Timestamp, TimestampNTZ, and Binary data types may produce different results (https://github.com/apache/datafusion-comet/issues/3232) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Complex types (arrays, maps, structs) in the schema are not supported + +## StructsToJson + +By default, `StructsToJson` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToJson.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Does not support `+Infinity` and `-Infinity` for numeric types (float, double). (https://github.com/apache/datafusion-comet/issues/3016) + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/url.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/url.md new file mode 100644 index 00000000000..765e12941fd --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/url.md @@ -0,0 +1,23 @@ + + +# URL Expressions + + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/aggregate.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/aggregate.md new file mode 100644 index 00000000000..093a70ac586 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/aggregate.md @@ -0,0 +1,156 @@ + + +# Aggregate Expressions + + + +## ApproximatePercentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The percentage argument must be a foldable literal. +- The accuracy argument must be a foldable literal. +- Only byte, short, int, long, float, and double input types are supported. + +## Average + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- YearMonthIntervalType and DayTimeIntervalType inputs are not supported + +## CollectSet + +The following incompatibilities cause `CollectSet` to fall back to Spark by default. Set `spark.comet.expression.CollectSet.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Before Spark 4.2, Comet deduplicates NaN values (treats `NaN == NaN`) while Spark treats each NaN as a distinct value. Comet treats -0.0 and 0.0 as distinct while Spark treats them as equal. When `spark.comet.exec.strictFloatingPoint=true`, `collect_set` on floating-point types falls back to Spark on those versions unless `spark.comet.expression.CollectSet.allowIncompatible=true` is set. + +## First + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## HllSketchAgg + +The following incompatibilities cause `HllSketchAgg` to fall back to Spark by default. Set `spark.comet.expression.HllSketchAgg.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The lgConfigK argument must be a foldable literal. +- The lgConfigK argument must be in the range [4, 21]. +- Only int, long, string, and binary input types are supported. + +## HllUnionAgg + +The following incompatibilities cause `HllUnionAgg` to fall back to Spark by default. Set `spark.comet.expression.HllUnionAgg.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark. +- Errors surface as plain Comet execution errors rather than Spark's SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail. +- An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The allowDifferentLgConfigK argument must be a foldable literal. + +## HyperLogLogPlusPlus + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input type must be boolean, integral, floating-point, decimal, date/time, string, or binary +- `DecimalType` with precision > 18 is not supported +- Collated (non-UTF8_BINARY) strings are not supported + +## Last + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## ListAgg + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` inputs are not supported. +- `WITHIN GROUP (ORDER BY ...)` is not supported. +- Non-literal delimiters are not supported. +- Non-default string collations are not supported. + +## Max + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MaxBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the maximum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Min + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MinBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the minimum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Mode + +The following incompatibilities cause `Mode` to fall back to Spark by default. Set `spark.comet.expression.Mode.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- mode breaks ties non-deterministically in Spark (the result depends on JVM hash-map iteration order); Comet returns the smallest of the tied values instead (https://github.com/apache/datafusion-comet/issues/3970) + +## Percentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- An array of percentages is not supported. +- The percentage argument must be a literal. +- A frequency argument is not supported. +- Descending order in `WITHIN GROUP (ORDER BY ... DESC)` is not supported. +- Only numeric input types are supported. + +## PivotFirst + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Value data type other than Boolean, Byte, Short, Int, Long, Float, Double or Decimal +- Pivot column of a type other than Boolean, Byte, Short, Int, Long, Float, Double, Decimal, String (UTF8_BINARY collation), Date, Timestamp or TimestampNTZ, which includes the struct that a pivot on more than one column builds +- Pivot values list is empty +- Pivot values list contains duplicates, including `0.0` next to `-0.0` + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/array.md new file mode 100644 index 00000000000..3e94336b308 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/array.md @@ -0,0 +1,113 @@ + + +# Array Expressions + + + +## ArrayAppend + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- array_append with a nullable array and a nondeterministic operand such as `rand()`: the native NULL guard evaluates the operands more than once, on subsets of the rows + +## ArrayContains + +For applicable cases that are not selected for native execution automatically, `ArrayContains` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayContains.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares array elements with ordering.equiv, so -0.0 matches +0.0 and all NaNs match each other; Comet's native array_contains applies that only to flat FLOAT and DOUBLE elements, and nested float elements stay on Spark's comparison +- array_contains evaluates its value eagerly, while Spark skips it when the array is null +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_contains compares raw bytes + +## ArrayDistinct + +The following incompatibilities cause `ArrayDistinct` to fall back to Spark by default. Set `spark.comet.expression.ArrayDistinct.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_distinct compares raw bytes + +## ArrayExcept + +By default, `ArrayExcept` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayExcept.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Null handling and ordering may differ from Spark + +## ArrayIntersect + +By default, `ArrayIntersect` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayIntersect.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Result array element order may differ from Spark when the right array is longer than the left (DataFusion probes the longer side). +- array_intersect does not propagate non-UTF8_BINARY collations to the output array elements (https://github.com/apache/datafusion-comet/issues/2190) + +## ArrayJoin + +By default, `ArrayJoin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayJoin.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190) +- array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178) + +## ArrayMax + +For applicable cases that are not selected for native execution automatically, `ArrayMax` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMax.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayMin + +For applicable cases that are not selected for native execution automatically, `ArrayMin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMin.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayUnion + +The following incompatibilities cause `ArrayUnion` to fall back to Spark by default. Set `spark.comet.expression.ArrayUnion.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_union compares raw bytes + +## ArraysOverlap + +For applicable cases that are not selected for native execution automatically, `ArraysOverlap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArraysOverlap.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native arrays_overlap compares raw bytes + +## ArraysZip + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all input data types are supported; falls back to Spark for unsupported types + +## Sequence + +The following differences from Spark are always present and do not require any additional configuration: + +- When `stop - start` overflows a long but the sequence is short, Spark 4.0.5, 4.1.4, 4.2.1 and later return its elements (SPARK-58821), and earlier releases raise `INTERNAL_ERROR`. Comet chooses the behavior by the Spark version number. On a vendor build that reports an older version but includes SPARK-58821, Comet raises the error where Spark returns the elements. + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- date and timestamp element types run through the JVM codegen dispatcher +- sequence arguments must be literals or column references; other shapes run through the JVM codegen dispatcher to preserve Spark's per-row null short-circuit + +## Shuffle + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- shuffle requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/cast.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/cast.md new file mode 100644 index 00000000000..c99186ccaff --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/cast.md @@ -0,0 +1,324 @@ + + +# Cast + +Cast operations in Comet fall into three levels of support: + +- **C (Compatible)**: The results match Apache Spark +- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By + default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark. + Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended + for production use. +- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated + cast inside the Comet pipeline. +- **N/A**: Spark does not support this cast. + +Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for +example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`. + +The tables at the end of this page cover casts between primitive types. See [Complex Types](#complex-types) for +casts involving arrays, structs, and maps. + +## ANSI Mode Support + +Enabling ANSI mode does not by itself make a cast fall back to Spark or require +`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support +level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the +[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting. + +## Whitespace Trimming in Casts from String + +Spark uses two different notions of whitespace when casting from string, and Comet reproduces +both: + +| Trim set | Cast targets | +| -------------------------------------- | --------------------------------------------------------------------------------------- | +| Bytes `0x00`-`0x20` and `0x7F` (`DEL`) | `BOOLEAN`, `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_NTZ` | +| Bytes `0x00`-`0x20` only | `FLOAT`, `DOUBLE`, `DECIMAL` | + +Neither set includes any non-ASCII whitespace, so padding a value with `U+0085`, `U+00A0`, +`U+1680`, `U+2000`-`U+200A`, `U+2028`, `U+2029`, `U+202F`, `U+205F` or `U+3000` produces `NULL` +(or raises under ANSI mode), even though those codepoints are whitespace to Unicode. Whitespace +inside a value is never trimmed. + +## String to Decimal + +Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's behavior, +including: + +- Leading and trailing bytes `0x00`-`0x20` are trimmed before parsing, matching the + `java.lang.String.trim` that Spark applies before handing the value to `BigDecimal`. +- Null bytes (`\u0000`) at the start or end of a string are therefore trimmed. Null bytes + embedded in the middle of a string produce `NULL`. +- Non-ASCII decimal digits that Java's `BigDecimal` accepts, i.e. every Unicode `Nd` digit in + the Basic Multilingual Plane (fullwidth `１２３.４５`, Arabic-Indic `١٢٣`, Devanagari `१२३`, + ...), are treated as their ASCII equivalents in both the mantissa and the exponent, so + `CAST('١٢٣.٤٥' AS DECIMAL(10,2))` returns `123.45`. Digits outside the Basic Multilingual + Plane, such as U+1D7CE MATHEMATICAL BOLD DIGIT ZERO, are rejected as they are by Spark. +- Scientific notation (e.g. `1.23E+5`) is supported. +- Special values (`inf`, `infinity`, `nan`) produce `NULL`. + +A zero with a large exponent, such as `'0e39'`, casts to `0` on Spark 4.0.5, 4.1.4, 4.2.1 and later +(SPARK-60118). Earlier releases return `NULL` for it, or raise `NUMERIC_OUT_OF_SUPPORTED_RANGE` +under ANSI mode. Comet chooses the behavior by the Spark version number. On a vendor build that +reports an older version but includes SPARK-60118, Comet returns `NULL` (or raises) where Spark +returns `0`. + +## String to Date + +Comet's native `CAST(string AS DATE)` implementation matches Apache Spark's behavior for years +between 262143 BC and 262142 AD. This range limitation comes from the underlying chrono library's +`NaiveDate` type. Spark itself supports a wider range. All three eval modes (Legacy, ANSI, Try) +are supported. + +Supported input formats match Spark exactly: + +- `yyyy`, `yyyy-[m]m`, `yyyy-[m]m-[d]d` +- Optional `T` suffix with arbitrary trailing text (e.g. `2020-01-01T12:34:56`) +- Leading/trailing bytes `0x00`-`0x20` and `0x7F` are trimmed +- Optional sign prefix (`-` for negative years) +- Leading zeros (e.g. `0002020-01-01` is year 2020) + +## Date to Timestamp + +Comet's native `CAST(date AS TIMESTAMP)` is compatible with Spark. The cast interprets each +date as midnight in the session timezone and converts to a UTC epoch value. DST transitions +are handled correctly, including spring-forward gaps (where midnight may not exist) and +fall-back ambiguity (where Comet picks the earlier/DST occurrence, matching Spark's +`LocalDate.atStartOfDay(zoneId)` behavior). + +## Date to TimestampNTZ + +Comet's native `CAST(date AS TIMESTAMP_NTZ)` is compatible with Spark. The cast is +timezone-independent: each date is converted to midnight as pure arithmetic +(`days * 86,400,000,000` microseconds) with no session timezone offset applied. The result +is the same regardless of the session timezone setting. + +## Date to Numeric Types + +In Legacy mode, `CAST(date AS INT)`, `CAST(date AS LONG)`, and casts to all other numeric +types (Boolean, Byte, Short, Float, Double, Decimal) always return `NULL`. Comet handles +this by short-circuiting to a null literal during query planning, so no native execution +is needed. In ANSI and Try modes, Spark rejects these casts at analysis time (before +execution reaches Comet). The short-circuit applies only to a top-level cast. See +[Complex Types](#complex-types) for `DATE` values inside structs and maps. + +## String to Timestamp + +Comet's native `CAST(string AS TIMESTAMP)` implementation supports all timestamp formats accepted +by Apache Spark, including ISO 8601 date-time strings, date-only strings, time-only strings +(`HH:MM:SS`), embedded timezone offsets (e.g. `+07:30`, `GMT-01:00`, `UTC`), named timezone +suffixes (e.g. `Europe/Moscow`), and the full Spark timestamp year range +(-290308 to 294247). + +## String to TimestampNTZ + +Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior. +Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in +the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time +components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`. +The result is always a wall-clock timestamp with no timezone conversion or DST adjustment. + +## TimestampNTZ Casts + +Comet supports the following `TIMESTAMP_NTZ` casts natively: + +| Cast | Compatible | Notes | +| ---------------------------------- | ---------- | ----------------------------------------------------------------- | +| `CAST(timestamp_ntz AS STRING)` | Yes | Formats local time as-is, timezone-independent | +| `CAST(timestamp_ntz AS DATE)` | Yes | Extracts the date component, timezone-independent | +| `CAST(timestamp_ntz AS TIMESTAMP)` | Yes | Interprets NTZ as local time in session TZ, converts to UTC epoch | +| `CAST(date AS TIMESTAMP_NTZ)` | Yes | Pure arithmetic, timezone-independent | +| `CAST(timestamp AS TIMESTAMP_NTZ)` | Yes | Shifts UTC epoch to local time in session TZ | +| `CAST(string AS TIMESTAMP_NTZ)` | Yes | See [String to TimestampNTZ](#string-to-timestampntz) above | + +The NTZ-to-Timestamp and Timestamp-to-NTZ casts are session-timezone-dependent (the session +timezone determines the UTC offset). All other NTZ casts are timezone-independent and produce +the same result regardless of the session timezone. + +## Date to String + +Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000 are +zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are +rendered without truncation. The cast is timezone-independent. + +## Decimal with Negative Scale to String + +Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when +`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is +disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's +code-generated cast instead of the native one. + +When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches +Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific +notation (e.g. a value of 12300 stored as `Decimal(7,-2)` with unscaled value 123 is rendered +as `"1.23E+4"`). + +## Decimal with Negative Scale to Integer and Timestamp + +Casting a `DecimalType` with a negative scale to `ByteType`, `ShortType`, `IntegerType`, +`LongType` or `TimestampType` has no native implementation, and neither does casting any of the +integer types to a negative-scale `DecimalType`. Comet reports these as unsupported, which +routes them through the JVM codegen dispatcher (Spark's own generated code) inside the Comet +pipeline, so the results match Spark exactly. + +The native kernels align scale by computing `10^|scale|` in fixed-point arithmetic, which +overflows when the scale is negative. See +[#5013](https://github.com/apache/datafusion-comet/issues/5013). + +All other cast directions to and from negative-scale decimals run natively: `FloatType` and +`DoubleType`, `StringType` in both directions (subject to the config gate described above), +casts from a negative-scale decimal to `BooleanType`, and widening to another `DecimalType`. +Casts from `BooleanType` to a `DecimalType` route through the dispatcher at any scale, which is +not specific to a negative one. + +## Complex Types + +Comet casts arrays, structs, and maps by casting each element, field, key, and value with the +rules for primitive types, using the table for the same eval mode. A complex-type cast is +therefore compatible only when every cast it contains is compatible. For example, casting +`ARRAY` to `ARRAY` has the same support level as casting `INT` to `BIGINT`, and a +struct cast with one unsupported field cast is unsupported. The rules apply at every level of +nesting, so casting `ARRAY>` to `ARRAY>` also has the same +support level as casting `INT` to `BIGINT`. + +| From | To | Support level | +| ----------- | ------------- | ---------------------------------------------------------------------------- | +| `ARRAY` | `ARRAY` | Same as `T` to `U` | +| `ARRAY` | `STRING` | Same as `T` to `STRING` | +| `STRUCT` | `STRUCT` | Compatible if every field cast is compatible. Fields are matched by position | +| `STRUCT` | `STRING` | Compatible if the cast from every field to `STRING` is compatible | +| `MAP` | `MAP` | Compatible if `K` to `K2` and `V` to `V2` are both compatible | +| `MAP` | `STRING` | Unsupported | + +The following casts are exceptions to these rules: + +- `ARRAY` can only be cast natively to `ARRAY` or `ARRAY`. Casts to other array + types are unsupported. +- A cast from a `DATE` struct field or map value to a numeric or boolean type is unsupported. In + Legacy mode these casts always return `NULL` (see [Date to Numeric Types](#date-to-numeric-types)). +- When `spark.sql.legacy.castComplexTypesToString.enabled=true`, casts from arrays, structs, and + maps to `STRING` are unsupported, because Comet's native cast only produces Spark's default + string format. + +In Try mode, a cast between maps fails with an error when a key cannot be cast, where Spark returns +a result ([#5995](https://github.com/apache/datafusion-comet/issues/5995)). This only affects key +casts that Spark accepts under `TRY_CAST`, such as narrowing `BIGINT` to `INT`. Spark rejects a +`STRING` key cast at analysis. + +## Decimal with Negative Scale to Integral Types + +Casting a `DecimalType` with a negative scale, which Spark only creates when +`spark.sql.legacy.allowNegativeScaleOfDecimal=true`, to `ByteType`, `ShortType`, `IntegerType` +or `LongType` is not supported natively in any eval mode. The native kernels divide by +`10^scale`, which overflows for a negative scale, so Comet leaves the cast to Spark. + +## Legacy Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | C | N/A | +| byte | C | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | C | C | - | C | C | C | C | C | C | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | C | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | C | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | C | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## Try Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## ANSI Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md new file mode 100644 index 00000000000..68eb6b4a8a0 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md @@ -0,0 +1,166 @@ + + +# Date/Time Expressions + +- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is marked Incompatible and + routes through the JVM codegen dispatcher by default, producing Spark-identical results. The + native path is itself correct for dates within chrono-tz's DST horizon (approximately year 2100; + see "Date and Time Functions" below) and can be enabled by setting + `spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ inputs are handled + correctly regardless of session timezone (timezone-independent truncation). + +## Date and Time Functions + +Comet's native implementation of date and time functions may produce different results than Spark for dates +far in the future (approximately beyond year 2100). This is because Comet uses the chrono-tz library for +timezone calculations, which has limited support for Daylight Saving Time (DST) rules beyond the IANA +time zone database's explicit transitions. + +For dates within a reasonable range (approximately 1970-2100), Comet's date and time functions are compatible +with Spark. For dates beyond this range, functions that involve timezone-aware calculations (such as +`date_trunc` with timezone-aware timestamps) may produce results with incorrect DST offsets. + +If you need to process dates far in the future with accurate timezone handling, consider: + +- Using timezone-naive types (`timestamp_ntz`) when timezone conversion is not required +- Falling back to Spark for these specific operations + +### Timezone Database Versions + +Comet's native code converts between instants and local time with the IANA timezone database that +chrono-tz compiles into the Comet library. Spark uses the JVM's timezone database (`tzdb.dat`), whose +version depends on the JDK build and on whether its timezone data has been updated. When the two versions +have different rules for a timezone, local times that Comet computes natively for that timezone can differ +from Spark's for the affected dates. That covers `hour`, casts between timestamps and strings or dates, +`date_trunc`, and parsing strings as timestamps. Comet logs a warning at startup when the two versions +differ. + + + +## ConvertTimezone + +By default, `ConvertTimezone` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ConvertTimezone.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. +- convert_timezone does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +## DateFormatClass + +The following differences from Spark are always present and do not require any additional configuration: + +- Format strings in a curated allow-list run natively via DataFusion's `to_char` for UTC sessions. Other format strings (including non-literal formats), as well as non-UTC sessions, route through Spark's own `DateFormatClass.doGenCode` via the Arrow-direct codegen dispatcher when `spark.comet.exec.scalaUDF.codegen.enabled=true` (the default). When the codegen dispatcher is disabled the operator falls back to Spark in those cases. + +By default, `DateFormatClass` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.DateFormatClass.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-UTC timezones may produce different results than Spark +- date_format does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +## Days + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DateType` and `TimestampType` inputs are supported. `TimestampNTZType` is not supported. + +## FromUTCTimestamp + +By default, `FromUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## FromUnixTime + +By default, `FromUnixTime` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUnixTime.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's valid timestamp range differs from Spark (https://github.com/apache/datafusion/issues/16594) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the default datetime format pattern `yyyy-MM-dd HH:mm:ss` is supported + +## Hours + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `TimestampType` and `TimestampNTZType` inputs are supported. + +## MakeInterval + +The following differences from Spark are always present and do not require any additional configuration: + +- Both the default JVM codegen-dispatch path and the native path currently limit the elapsed-time component to about 292 years in either direction. This only affects extreme intervals and is tracked in [#5279](https://github.com/apache/datafusion-comet/issues/5279). + +By default, `MakeInterval` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MakeInterval.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- The native implementation converts seconds to `Float64`, which can lose microsecond precision, and stores time in nanoseconds, which overflows for large time components (hours, minutes, seconds) that Spark can represent. + +## NextDay + +By default, `NextDay` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.NextDay.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- next_day does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +## PreciseTimestampConversion + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only reinterprets between TimestampType/TimestampNTZType and LongType are supported. + +## SecondsToTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `DecimalType`, `ByteType`, and `ShortType` inputs + +## ToUTCTimestamp + +By default, `ToUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ToUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## TruncDate + +By default, `TruncDate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncDate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-literal format strings will throw an exception instead of returning NULL +- trunc does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week + +## TruncTimestamp + +By default, `TruncTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Produces incorrect results when used with non-UTC timezones. Compatible when timezone is UTC. (https://github.com/apache/datafusion-comet/issues/2649) +- Non-literal format strings will throw an exception instead of returning NULL +- date_trunc does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week, day, dd, hour, minute, second, millisecond, microsecond + +## UnixTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- String inputs, including collated strings, have no native implementation. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/map.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/map.md new file mode 100644 index 00000000000..ece843d1a75 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/map.md @@ -0,0 +1,76 @@ + + +# Map Expressions + +## MapSort (Spark 4.0+) + +Spark 4.0 inserts `MapSort` to normalize map values when they appear in shuffle hash partitioning +keys, in `try_element_at`, and in other contexts where map ordering must be deterministic. Comet +runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0. + +When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose +keys contain `Float` or `Double`. Arrow's sort uses +IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for +`NaN` and `-0.0`. + + + +## MapFromArrays + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## MapFromEntries + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromEntries.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `BinaryType` is not supported as a map key in `map_from_entries` +- `BinaryType` is not supported as a map value in `map_from_entries` +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## MapSort + +The following incompatibilities cause `MapSort` to fall back to Spark by default. Set `spark.comet.expression.MapSort.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- MapSort on floating-point keys is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- MapSort is unsupported for non-scalar key types (struct, array, map, etc.). + +## StringToMap + +By default, `StringToMap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringToMap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `spark.sql.legacy.truncateForEmptyRegexSplit` is enabled, so trailing empty split entries may differ from Spark. +- `str_to_map` does not support non-UTF8_BINARY collations on the input string or delimiters. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/math.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/math.md new file mode 100644 index 00000000000..b0e2ec1d3b1 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/math.md @@ -0,0 +1,79 @@ + + +# Math Expressions + + + +## Abs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `INTERVAL YEAR TO MONTH` and `INTERVAL DAY TO SECOND` inputs + +## Divide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Greatest + +For applicable cases that are not selected for native execution automatically, `Greatest` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Greatest.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native greatest compares raw bytes + +## IntegralDivide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Least + +For applicable cases that are not selected for native execution automatically, `Least` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Least.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native least compares raw bytes + +## Rand + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Randn + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Round + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Float and double inputs. Spark rounds them through a BigDecimal built from `java.lang.Double.toString()` rather than from the exact binary value, and that shortened decimal string can round differently than the value it came from +- Negative-scale decimal inputs, which are only creatable with spark.sql.legacy.allowNegativeScaleOfDecimal=true + +## Signum + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DoubleType` is supported + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/misc.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/misc.md new file mode 100644 index 00000000000..f477a93886d --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/misc.md @@ -0,0 +1,92 @@ + + +# Miscellaneous Expressions + + + +## CheckOverflow + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DecimalType` is supported + +## HllSketchEstimate + +The following incompatibilities cause `HllSketchEstimate` to fall back to Spark by default. Set `spark.comet.expression.HllSketchEstimate.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL estimates may differ slightly from Spark. +- Errors surface as plain Comet execution errors rather than Spark's SparkRuntimeException with condition HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail. +- An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere. + +## HllUnion + +The following incompatibilities cause `HllUnion` to fall back to Spark by default. Set `spark.comet.expression.HllUnion.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark. +- Errors surface as plain Comet execution errors rather than Spark's SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail. +- An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The allowDifferentLgConfigK argument must be a foldable literal. + +## Literal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for literal values + +## MakeDecimal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `LongType` input is supported + +## ScalarSubquery + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for scalar subquery results + +## SortOrder + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- A nested sort key with nondefault root null placement must be deterministic: the wrapper can evaluate the key twice, for isnull(key) and its value, which could disagree + +## StaticInvoke + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 10 is one such value: the result has 19 digits. The other `truncate` input types, and `bucket` on decimals, are unaffected. + +## ToPrettyString + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Falls back to Spark when the input type cannot be cast to string. + +## Uuid + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- uuid requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/string.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/string.md new file mode 100644 index 00000000000..57d39cc2678 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/string.md @@ -0,0 +1,225 @@ + + +# String Expressions + + + +## BitLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## Concat + +By default, `Concat` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Concat.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- concat does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- CONCAT supports only string input parameters + +## ConcatWs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- all arguments are foldable + +## Contains + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## EndsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## GetJsonObject + +By default, `GetJsonObject` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.GetJsonObject.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark allows single-quoted JSON and unescaped control characters which Comet does not support +- Very long numbers near Jackson's 1000-digit limit can depend on Spark's recycled parser buffer, which Comet cannot reproduce from the input alone +- Selected JSON integers outside the 64-bit range and very long numbers can lose precision or fail during Comet's native JSON materialization +- Some selected floating-point values can differ from Spark at decimal parsing or Java-version formatting boundaries +- When a returned object or array contains duplicate keys, Spark preserves them while Comet's native JSON materialization keeps only the last value + +## InitCap + +By default, `InitCap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.InitCap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Treats hyphen as a word separator (e.g. `robert rose-smith` produces `Robert Rose-Smith` instead of Spark's `Robert Rose-smith`) (https://github.com/apache/datafusion-comet/issues/1052) + +## Levenshtein + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-default (non-UTF8_BINARY) collated input. The native kernel compares raw bytes, so collation-aware comparison has no native path. + +## Like + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- LIKE with a custom escape character (only `\` is supported natively) +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## Lower + +By default, `Lower` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + +## OctetLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## RLike + +The following cases use Comet's native implementation by default: + +- A `UTF8_BINARY` literal pattern admitted by the [plan-time compatibility analyzer](../../regex.md#when-the-rust-engine-is-safe) is evaluated natively by default. + +For applicable cases that are not selected for native execution automatically, `RLike` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.RLike.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- For applicable literal patterns outside the automatically admitted subset, the native Rust regex engine may behave differently from Java regex. + +## RandStr + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `length` must be a literal value +- `length` must be non-negative +- `seed` must be a literal value + +## RegExpReplace + +By default, `RegExpReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.RegExpReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regexp pattern may not be compatible with Spark + +## Reverse + +By default, `Reverse` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Reverse.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- native reverse does not support arrays whose element type contains binary, struct, or map +- reverse does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- reverse has no native implementation for binary input + +## StartsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## StringInstr + +For applicable cases that are not selected for native execution automatically, `StringInstr` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringInstr.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native instr compares raw bytes + +## StringLPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRepeat + +The following differences from Spark are always present and do not require any additional configuration: + +- A negative argument for the number of times to repeat throws an exception instead of returning an empty string as Spark does + +## StringReplace + +The following cases use Comet's native implementation by default: + +- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default. + +For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast + +## StringSplit + +By default, `StringSplit` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringSplit.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regex engine differences between Java and Rust + +## StringTranslate + +By default, `StringTranslate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel + +## StringTrim + +For applicable cases that are not selected for native execution automatically, `StringTrim` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrim.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native trim with a trim string compares raw bytes + +## StringTrimLeft + +For applicable cases that are not selected for native execution automatically, `StringTrimLeft` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimLeft.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native ltrim with a trim string compares raw bytes + +## StringTrimRight + +For applicable cases that are not selected for native execution automatically, `StringTrimRight` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimRight.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native rtrim with a trim string compares raw bytes + +## SubstringIndex + +For applicable cases that are not selected for native execution automatically, `SubstringIndex` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.SubstringIndex.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native substring_index compares raw bytes + +## UnBase64 + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- unbase64 with failOnError = true uses stricter RFC 4648 validation that is not yet implemented natively +- unbase64 with a non-trivial child expression uses the JVM codegen dispatcher to preserve Spark's short-circuit evaluation (native path is limited to column and literal children) + +## Upper + +By default, `Upper` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/struct.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/struct.md new file mode 100644 index 00000000000..c530b12a1a8 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/struct.md @@ -0,0 +1,52 @@ + + +# Struct Expressions + + + +## CreateNamedStruct + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `CreateNamedStruct` with duplicate field names is not supported + +## JsonToStructs + +By default, `JsonToStructs` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.JsonToStructs.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Partially implemented and not comprehensively tested + +## StructsToCsv + +By default, `StructsToCsv` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToCsv.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Date, Timestamp, TimestampNTZ, and Binary data types may produce different results (https://github.com/apache/datafusion-comet/issues/3232) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Complex types (arrays, maps, structs) in the schema are not supported + +## StructsToJson + +By default, `StructsToJson` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToJson.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Does not support `+Infinity` and `-Infinity` for numeric types (float, double). (https://github.com/apache/datafusion-comet/issues/3016) + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/url.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/url.md new file mode 100644 index 00000000000..765e12941fd --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/url.md @@ -0,0 +1,23 @@ + + +# URL Expressions + + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/aggregate.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/aggregate.md new file mode 100644 index 00000000000..07f2e833b98 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/aggregate.md @@ -0,0 +1,160 @@ + + +# Aggregate Expressions + + + +## ApproximatePercentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The percentage argument must be a foldable literal. +- The accuracy argument must be a foldable literal. +- Only byte, short, int, long, float, and double input types are supported. + +## Average + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- YearMonthIntervalType and DayTimeIntervalType inputs are not supported + +## CollectSet + +The following differences from Spark are always present and do not require any additional configuration: + +- On Spark 4.2+, `collect_set` inputs are normalized with Spark's recursive floating-point normalizer to match SPARK-57298. For array inputs containing floating-point values the normalizer produces `ArrayTransform`, which Comet executes through the JVM codegen dispatcher instead of fully natively (scalar and struct inputs stay native). When `spark.comet.exec.scalaUDF.codegen.enabled=false`, those array cases fall back to Spark. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `collect_set` with `RESPECT NULLS` falls back to Spark, since the native implementation always drops null inputs. + +## First + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## HllSketchAgg + +The following incompatibilities cause `HllSketchAgg` to fall back to Spark by default. Set `spark.comet.expression.HllSketchAgg.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The lgConfigK argument must be a foldable literal. +- The lgConfigK argument must be in the range [4, 21]. +- Only int, long, string, and binary input types are supported. + +## HllUnionAgg + +The following incompatibilities cause `HllUnionAgg` to fall back to Spark by default. Set `spark.comet.expression.HllUnionAgg.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark. +- Errors surface as plain Comet execution errors rather than Spark's SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail. +- An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The allowDifferentLgConfigK argument must be a foldable literal. + +## HyperLogLogPlusPlus + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input type must be boolean, integral, floating-point, decimal, date/time, string, or binary +- `DecimalType` with precision > 18 is not supported +- Collated (non-UTF8_BINARY) strings are not supported + +## Last + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is not deterministic. Results may not match Spark. + +## ListAgg + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` inputs are not supported. +- `WITHIN GROUP (ORDER BY ...)` is not supported. +- Non-literal delimiters are not supported. +- Non-default string collations are not supported. + +## Max + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MaxBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the maximum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Min + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Input of a type other than boolean, integral, floating-point, decimal, date, timestamp, string or binary, such as timestamp without time zone, intervals, arrays and structs. +- A string with a non-default collation (Spark 4.0+), which Spark orders by its collation while Comet compares raw bytes. + +## MinBy + +The following differences from Spark are always present and do not require any additional configuration: + +- This function is non-deterministic when multiple rows share the minimum ordering value. Results may differ from Spark in that case. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The value and ordering must both be fixed-length types (boolean, integral, floating-point, decimal, date, or timestamp). A variable-length or nested type such as string, binary, or struct falls back to Spark. + +## Mode + +The following incompatibilities cause `Mode` to fall back to Spark by default. Set `spark.comet.expression.Mode.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- mode breaks ties non-deterministically in Spark (the result depends on JVM hash-map iteration order); Comet returns the smallest of the tied values instead (https://github.com/apache/datafusion-comet/issues/3970) + +## Percentile + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- An array of percentages is not supported. +- The percentage argument must be a literal. +- A frequency argument is not supported. +- Descending order in `WITHIN GROUP (ORDER BY ... DESC)` is not supported. +- Only numeric input types are supported. + +## PivotFirst + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Value data type other than Boolean, Byte, Short, Int, Long, Float, Double or Decimal +- Pivot column of a type other than Boolean, Byte, Short, Int, Long, Float, Double, Decimal, String (UTF8_BINARY collation), Date, Timestamp or TimestampNTZ, which includes the struct that a pivot on more than one column builds +- Pivot values list is empty +- Pivot values list contains duplicates, including `0.0` next to `-0.0` + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/array.md new file mode 100644 index 00000000000..3e94336b308 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/array.md @@ -0,0 +1,113 @@ + + +# Array Expressions + + + +## ArrayAppend + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- array_append with a nullable array and a nondeterministic operand such as `rand()`: the native NULL guard evaluates the operands more than once, on subsets of the rows + +## ArrayContains + +For applicable cases that are not selected for native execution automatically, `ArrayContains` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayContains.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares array elements with ordering.equiv, so -0.0 matches +0.0 and all NaNs match each other; Comet's native array_contains applies that only to flat FLOAT and DOUBLE elements, and nested float elements stay on Spark's comparison +- array_contains evaluates its value eagerly, while Spark skips it when the array is null +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_contains compares raw bytes + +## ArrayDistinct + +The following incompatibilities cause `ArrayDistinct` to fall back to Spark by default. Set `spark.comet.expression.ArrayDistinct.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_distinct compares raw bytes + +## ArrayExcept + +By default, `ArrayExcept` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayExcept.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Null handling and ordering may differ from Spark + +## ArrayIntersect + +By default, `ArrayIntersect` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayIntersect.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Result array element order may differ from Spark when the right array is longer than the left (DataFusion probes the longer side). +- array_intersect does not propagate non-UTF8_BINARY collations to the output array elements (https://github.com/apache/datafusion-comet/issues/2190) + +## ArrayJoin + +By default, `ArrayJoin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayJoin.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190) +- array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178) + +## ArrayMax + +For applicable cases that are not selected for native execution automatically, `ArrayMax` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMax.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayMin + +For applicable cases that are not selected for native execution automatically, `ArrayMin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArrayMin.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Array extrema use binary string ordering for non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4496). + +## ArrayUnion + +The following incompatibilities cause `ArrayUnion` to fall back to Spark by default. Set `spark.comet.expression.ArrayUnion.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Floating-point elements match Spark's signed-zero semantics natively only on Spark 4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions (SPARK-54918, SPARK-59602) +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native array_union compares raw bytes + +## ArraysOverlap + +For applicable cases that are not selected for native execution automatically, `ArraysOverlap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.ArraysOverlap.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark compares non-UTF8_BINARY collated string elements under their collation, while Comet's native arrays_overlap compares raw bytes + +## ArraysZip + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all input data types are supported; falls back to Spark for unsupported types + +## Sequence + +The following differences from Spark are always present and do not require any additional configuration: + +- When `stop - start` overflows a long but the sequence is short, Spark 4.0.5, 4.1.4, 4.2.1 and later return its elements (SPARK-58821), and earlier releases raise `INTERNAL_ERROR`. Comet chooses the behavior by the Spark version number. On a vendor build that reports an older version but includes SPARK-58821, Comet raises the error where Spark returns the elements. + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- date and timestamp element types run through the JVM codegen dispatcher +- sequence arguments must be literals or column references; other shapes run through the JVM codegen dispatcher to preserve Spark's per-row null short-circuit + +## Shuffle + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- shuffle requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/cast.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/cast.md new file mode 100644 index 00000000000..c99186ccaff --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/cast.md @@ -0,0 +1,324 @@ + + +# Cast + +Cast operations in Comet fall into three levels of support: + +- **C (Compatible)**: The results match Apache Spark +- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By + default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark. + Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended + for production use. +- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated + cast inside the Comet pipeline. +- **N/A**: Spark does not support this cast. + +Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for +example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`. + +The tables at the end of this page cover casts between primitive types. See [Complex Types](#complex-types) for +casts involving arrays, structs, and maps. + +## ANSI Mode Support + +Enabling ANSI mode does not by itself make a cast fall back to Spark or require +`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support +level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the +[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting. + +## Whitespace Trimming in Casts from String + +Spark uses two different notions of whitespace when casting from string, and Comet reproduces +both: + +| Trim set | Cast targets | +| -------------------------------------- | --------------------------------------------------------------------------------------- | +| Bytes `0x00`-`0x20` and `0x7F` (`DEL`) | `BOOLEAN`, `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_NTZ` | +| Bytes `0x00`-`0x20` only | `FLOAT`, `DOUBLE`, `DECIMAL` | + +Neither set includes any non-ASCII whitespace, so padding a value with `U+0085`, `U+00A0`, +`U+1680`, `U+2000`-`U+200A`, `U+2028`, `U+2029`, `U+202F`, `U+205F` or `U+3000` produces `NULL` +(or raises under ANSI mode), even though those codepoints are whitespace to Unicode. Whitespace +inside a value is never trimmed. + +## String to Decimal + +Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's behavior, +including: + +- Leading and trailing bytes `0x00`-`0x20` are trimmed before parsing, matching the + `java.lang.String.trim` that Spark applies before handing the value to `BigDecimal`. +- Null bytes (`\u0000`) at the start or end of a string are therefore trimmed. Null bytes + embedded in the middle of a string produce `NULL`. +- Non-ASCII decimal digits that Java's `BigDecimal` accepts, i.e. every Unicode `Nd` digit in + the Basic Multilingual Plane (fullwidth `１２３.４５`, Arabic-Indic `١٢٣`, Devanagari `१२३`, + ...), are treated as their ASCII equivalents in both the mantissa and the exponent, so + `CAST('١٢٣.٤٥' AS DECIMAL(10,2))` returns `123.45`. Digits outside the Basic Multilingual + Plane, such as U+1D7CE MATHEMATICAL BOLD DIGIT ZERO, are rejected as they are by Spark. +- Scientific notation (e.g. `1.23E+5`) is supported. +- Special values (`inf`, `infinity`, `nan`) produce `NULL`. + +A zero with a large exponent, such as `'0e39'`, casts to `0` on Spark 4.0.5, 4.1.4, 4.2.1 and later +(SPARK-60118). Earlier releases return `NULL` for it, or raise `NUMERIC_OUT_OF_SUPPORTED_RANGE` +under ANSI mode. Comet chooses the behavior by the Spark version number. On a vendor build that +reports an older version but includes SPARK-60118, Comet returns `NULL` (or raises) where Spark +returns `0`. + +## String to Date + +Comet's native `CAST(string AS DATE)` implementation matches Apache Spark's behavior for years +between 262143 BC and 262142 AD. This range limitation comes from the underlying chrono library's +`NaiveDate` type. Spark itself supports a wider range. All three eval modes (Legacy, ANSI, Try) +are supported. + +Supported input formats match Spark exactly: + +- `yyyy`, `yyyy-[m]m`, `yyyy-[m]m-[d]d` +- Optional `T` suffix with arbitrary trailing text (e.g. `2020-01-01T12:34:56`) +- Leading/trailing bytes `0x00`-`0x20` and `0x7F` are trimmed +- Optional sign prefix (`-` for negative years) +- Leading zeros (e.g. `0002020-01-01` is year 2020) + +## Date to Timestamp + +Comet's native `CAST(date AS TIMESTAMP)` is compatible with Spark. The cast interprets each +date as midnight in the session timezone and converts to a UTC epoch value. DST transitions +are handled correctly, including spring-forward gaps (where midnight may not exist) and +fall-back ambiguity (where Comet picks the earlier/DST occurrence, matching Spark's +`LocalDate.atStartOfDay(zoneId)` behavior). + +## Date to TimestampNTZ + +Comet's native `CAST(date AS TIMESTAMP_NTZ)` is compatible with Spark. The cast is +timezone-independent: each date is converted to midnight as pure arithmetic +(`days * 86,400,000,000` microseconds) with no session timezone offset applied. The result +is the same regardless of the session timezone setting. + +## Date to Numeric Types + +In Legacy mode, `CAST(date AS INT)`, `CAST(date AS LONG)`, and casts to all other numeric +types (Boolean, Byte, Short, Float, Double, Decimal) always return `NULL`. Comet handles +this by short-circuiting to a null literal during query planning, so no native execution +is needed. In ANSI and Try modes, Spark rejects these casts at analysis time (before +execution reaches Comet). The short-circuit applies only to a top-level cast. See +[Complex Types](#complex-types) for `DATE` values inside structs and maps. + +## String to Timestamp + +Comet's native `CAST(string AS TIMESTAMP)` implementation supports all timestamp formats accepted +by Apache Spark, including ISO 8601 date-time strings, date-only strings, time-only strings +(`HH:MM:SS`), embedded timezone offsets (e.g. `+07:30`, `GMT-01:00`, `UTC`), named timezone +suffixes (e.g. `Europe/Moscow`), and the full Spark timestamp year range +(-290308 to 294247). + +## String to TimestampNTZ + +Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior. +Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in +the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time +components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`. +The result is always a wall-clock timestamp with no timezone conversion or DST adjustment. + +## TimestampNTZ Casts + +Comet supports the following `TIMESTAMP_NTZ` casts natively: + +| Cast | Compatible | Notes | +| ---------------------------------- | ---------- | ----------------------------------------------------------------- | +| `CAST(timestamp_ntz AS STRING)` | Yes | Formats local time as-is, timezone-independent | +| `CAST(timestamp_ntz AS DATE)` | Yes | Extracts the date component, timezone-independent | +| `CAST(timestamp_ntz AS TIMESTAMP)` | Yes | Interprets NTZ as local time in session TZ, converts to UTC epoch | +| `CAST(date AS TIMESTAMP_NTZ)` | Yes | Pure arithmetic, timezone-independent | +| `CAST(timestamp AS TIMESTAMP_NTZ)` | Yes | Shifts UTC epoch to local time in session TZ | +| `CAST(string AS TIMESTAMP_NTZ)` | Yes | See [String to TimestampNTZ](#string-to-timestampntz) above | + +The NTZ-to-Timestamp and Timestamp-to-NTZ casts are session-timezone-dependent (the session +timezone determines the UTC offset). All other NTZ casts are timezone-independent and produce +the same result regardless of the session timezone. + +## Date to String + +Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000 are +zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are +rendered without truncation. The cast is timezone-independent. + +## Decimal with Negative Scale to String + +Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when +`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is +disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's +code-generated cast instead of the native one. + +When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches +Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific +notation (e.g. a value of 12300 stored as `Decimal(7,-2)` with unscaled value 123 is rendered +as `"1.23E+4"`). + +## Decimal with Negative Scale to Integer and Timestamp + +Casting a `DecimalType` with a negative scale to `ByteType`, `ShortType`, `IntegerType`, +`LongType` or `TimestampType` has no native implementation, and neither does casting any of the +integer types to a negative-scale `DecimalType`. Comet reports these as unsupported, which +routes them through the JVM codegen dispatcher (Spark's own generated code) inside the Comet +pipeline, so the results match Spark exactly. + +The native kernels align scale by computing `10^|scale|` in fixed-point arithmetic, which +overflows when the scale is negative. See +[#5013](https://github.com/apache/datafusion-comet/issues/5013). + +All other cast directions to and from negative-scale decimals run natively: `FloatType` and +`DoubleType`, `StringType` in both directions (subject to the config gate described above), +casts from a negative-scale decimal to `BooleanType`, and widening to another `DecimalType`. +Casts from `BooleanType` to a `DecimalType` route through the dispatcher at any scale, which is +not specific to a negative one. + +## Complex Types + +Comet casts arrays, structs, and maps by casting each element, field, key, and value with the +rules for primitive types, using the table for the same eval mode. A complex-type cast is +therefore compatible only when every cast it contains is compatible. For example, casting +`ARRAY` to `ARRAY` has the same support level as casting `INT` to `BIGINT`, and a +struct cast with one unsupported field cast is unsupported. The rules apply at every level of +nesting, so casting `ARRAY>` to `ARRAY>` also has the same +support level as casting `INT` to `BIGINT`. + +| From | To | Support level | +| ----------- | ------------- | ---------------------------------------------------------------------------- | +| `ARRAY` | `ARRAY` | Same as `T` to `U` | +| `ARRAY` | `STRING` | Same as `T` to `STRING` | +| `STRUCT` | `STRUCT` | Compatible if every field cast is compatible. Fields are matched by position | +| `STRUCT` | `STRING` | Compatible if the cast from every field to `STRING` is compatible | +| `MAP` | `MAP` | Compatible if `K` to `K2` and `V` to `V2` are both compatible | +| `MAP` | `STRING` | Unsupported | + +The following casts are exceptions to these rules: + +- `ARRAY` can only be cast natively to `ARRAY` or `ARRAY`. Casts to other array + types are unsupported. +- A cast from a `DATE` struct field or map value to a numeric or boolean type is unsupported. In + Legacy mode these casts always return `NULL` (see [Date to Numeric Types](#date-to-numeric-types)). +- When `spark.sql.legacy.castComplexTypesToString.enabled=true`, casts from arrays, structs, and + maps to `STRING` are unsupported, because Comet's native cast only produces Spark's default + string format. + +In Try mode, a cast between maps fails with an error when a key cannot be cast, where Spark returns +a result ([#5995](https://github.com/apache/datafusion-comet/issues/5995)). This only affects key +casts that Spark accepts under `TRY_CAST`, such as narrowing `BIGINT` to `INT`. Spark rejects a +`STRING` key cast at analysis. + +## Decimal with Negative Scale to Integral Types + +Casting a `DecimalType` with a negative scale, which Spark only creates when +`spark.sql.legacy.allowNegativeScaleOfDecimal=true`, to `ByteType`, `ShortType`, `IntegerType` +or `LongType` is not supported natively in any eval mode. The native kernels divide by +`10^scale`, which overflows for a negative scale, so Comet leaves the cast to Spark. + +## Legacy Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | C | N/A | +| byte | C | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | C | C | - | C | C | C | C | C | C | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | C | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | C | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | C | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## Try Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + + +## ANSI Mode + + + + +| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp | timestamp_ntz | +| ------------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- | ------------- | +| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A | N/A | +| boolean | N/A | - | C | N/A | U | C | C | C | C | C | C | U | N/A | +| byte | U | C | - | N/A | C | C | C | C | C | C | C | C | N/A | +| date | N/A | U | U | - | U | U | U | U | U | U | C | C | C | +| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C | N/A | +| double | N/A | C | C | N/A | C | - | C | C | C | C | C | C | N/A | +| float | N/A | C | C | N/A | C | C | - | C | C | C | C | C | N/A | +| integer | U | C | C | N/A | C | C | C | - | C | C | C | C | N/A | +| long | U | C | C | N/A | C | C | C | C | - | C | C | C | N/A | +| short | U | C | C | N/A | C | C | C | C | C | - | C | C | N/A | +| string | C | C | C | C | C | C | C | C | C | C | - | C | C | +| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - | C | +| timestamp_ntz | N/A | N/A | N/A | C | N/A | N/A | N/A | N/A | N/A | N/A | C | C | - | + +**Notes:** + +- **double -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **double -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **float -> decimal**: Rounding matches Spark on JDK 19 and later. On older JDKs, Spark's own result may differ for values whose shortest decimal form falls exactly on a rounding tie +- **float -> string**: String formatting can differ for floating-point values near precision limits or when scientific notation is used +- **string -> date**: Only supports years between 262143 BC and 262142 AD + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/datetime.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/datetime.md new file mode 100644 index 00000000000..68eb6b4a8a0 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/datetime.md @@ -0,0 +1,166 @@ + + +# Date/Time Expressions + +- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is marked Incompatible and + routes through the JVM codegen dispatcher by default, producing Spark-identical results. The + native path is itself correct for dates within chrono-tz's DST horizon (approximately year 2100; + see "Date and Time Functions" below) and can be enabled by setting + `spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ inputs are handled + correctly regardless of session timezone (timezone-independent truncation). + +## Date and Time Functions + +Comet's native implementation of date and time functions may produce different results than Spark for dates +far in the future (approximately beyond year 2100). This is because Comet uses the chrono-tz library for +timezone calculations, which has limited support for Daylight Saving Time (DST) rules beyond the IANA +time zone database's explicit transitions. + +For dates within a reasonable range (approximately 1970-2100), Comet's date and time functions are compatible +with Spark. For dates beyond this range, functions that involve timezone-aware calculations (such as +`date_trunc` with timezone-aware timestamps) may produce results with incorrect DST offsets. + +If you need to process dates far in the future with accurate timezone handling, consider: + +- Using timezone-naive types (`timestamp_ntz`) when timezone conversion is not required +- Falling back to Spark for these specific operations + +### Timezone Database Versions + +Comet's native code converts between instants and local time with the IANA timezone database that +chrono-tz compiles into the Comet library. Spark uses the JVM's timezone database (`tzdb.dat`), whose +version depends on the JDK build and on whether its timezone data has been updated. When the two versions +have different rules for a timezone, local times that Comet computes natively for that timezone can differ +from Spark's for the affected dates. That covers `hour`, casts between timestamps and strings or dates, +`date_trunc`, and parsing strings as timestamps. Comet logs a warning at startup when the two versions +differ. + + + +## ConvertTimezone + +By default, `ConvertTimezone` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ConvertTimezone.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. +- convert_timezone does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +## DateFormatClass + +The following differences from Spark are always present and do not require any additional configuration: + +- Format strings in a curated allow-list run natively via DataFusion's `to_char` for UTC sessions. Other format strings (including non-literal formats), as well as non-UTC sessions, route through Spark's own `DateFormatClass.doGenCode` via the Arrow-direct codegen dispatcher when `spark.comet.exec.scalaUDF.codegen.enabled=true` (the default). When the codegen dispatcher is disabled the operator falls back to Spark in those cases. + +By default, `DateFormatClass` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.DateFormatClass.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-UTC timezones may produce different results than Spark +- date_format does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +## Days + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DateType` and `TimestampType` inputs are supported. `TimestampNTZType` is not supported. + +## FromUTCTimestamp + +By default, `FromUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## FromUnixTime + +By default, `FromUnixTime` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.FromUnixTime.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's valid timestamp range differs from Spark (https://github.com/apache/datafusion/issues/16594) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the default datetime format pattern `yyyy-MM-dd HH:mm:ss` is supported + +## Hours + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `TimestampType` and `TimestampNTZType` inputs are supported. + +## MakeInterval + +The following differences from Spark are always present and do not require any additional configuration: + +- Both the default JVM codegen-dispatch path and the native path currently limit the elapsed-time component to about 292 years in either direction. This only affects extreme intervals and is tracked in [#5279](https://github.com/apache/datafusion-comet/issues/5279). + +By default, `MakeInterval` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MakeInterval.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- The native implementation converts seconds to `Float64`, which can lose microsecond precision, and stores time in nanoseconds, which overflows for large time components (hours, minutes, seconds) that Spark can represent. + +## NextDay + +By default, `NextDay` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.NextDay.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- next_day does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +## PreciseTimestampConversion + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only reinterprets between TimestampType/TimestampNTZType and LongType are supported. + +## SecondsToTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `DecimalType`, `ByteType`, and `ShortType` inputs + +## ToUTCTimestamp + +By default, `ToUTCTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ToUTCTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed offsets in `+HH:MM` form. Spark also accepts forms such as `GMT+1`, `UTC+1`, or three-letter abbreviations like `PST`; queries using those forms will throw a native parse error at execution time. See https://github.com/apache/datafusion-comet/issues/2013. + +## TruncDate + +By default, `TruncDate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncDate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Non-literal format strings will throw an exception instead of returning NULL +- trunc does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week + +## TruncTimestamp + +By default, `TruncTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.TruncTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Produces incorrect results when used with non-UTC timezones. Compatible when timezone is UTC. (https://github.com/apache/datafusion-comet/issues/2649) +- Non-literal format strings will throw an exception instead of returning NULL +- date_trunc does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Only the following formats are supported: year, yyyy, yy, quarter, mon, month, mm, week, day, dd, hour, minute, second, millisecond, microsecond + +## UnixTimestamp + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- String inputs, including collated strings, have no native implementation. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/map.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/map.md new file mode 100644 index 00000000000..ece843d1a75 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/map.md @@ -0,0 +1,76 @@ + + +# Map Expressions + +## MapSort (Spark 4.0+) + +Spark 4.0 inserts `MapSort` to normalize map values when they appear in shuffle hash partitioning +keys, in `try_element_at`, and in other contexts where map ordering must be deterministic. Comet +runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0. + +When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose +keys contain `Float` or `Double`. Arrow's sort uses +IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for +`NaN` and `-0.0`. + + + +## MapFromArrays + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## MapFromEntries + +The following differences from Spark are always present and do not require any additional configuration: + +- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)). + +By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromEntries.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `BinaryType` is not supported as a map key in `map_from_entries` +- `BinaryType` is not supported as a map value in `map_from_entries` +- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). +- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics. + +## MapSort + +The following incompatibilities cause `MapSort` to fall back to Spark by default. Set `spark.comet.expression.MapSort.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- MapSort on floating-point keys is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- MapSort is unsupported for non-scalar key types (struct, array, map, etc.). + +## StringToMap + +By default, `StringToMap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringToMap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- `spark.sql.legacy.truncateForEmptyRegexSplit` is enabled, so trailing empty split entries may differ from Spark. +- `str_to_map` does not support non-UTF8_BINARY collations on the input string or delimiters. + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/math.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/math.md new file mode 100644 index 00000000000..b0e2ec1d3b1 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/math.md @@ -0,0 +1,79 @@ + + +# Math Expressions + + + +## Abs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- `INTERVAL YEAR TO MONTH` and `INTERVAL DAY TO SECOND` inputs + +## Divide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Greatest + +For applicable cases that are not selected for native execution automatically, `Greatest` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Greatest.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native greatest compares raw bytes + +## IntegralDivide + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Decimal division with negative scale is not supported natively + +## Least + +For applicable cases that are not selected for native execution automatically, `Least` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.Least.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native least compares raw bytes + +## Rand + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Randn + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The `seed` argument must be a literal value + +## Round + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Float and double inputs. Spark rounds them through a BigDecimal built from `java.lang.Double.toString()` rather than from the exact binary value, and that shortened decimal string can round differently than the value it came from +- Negative-scale decimal inputs, which are only creatable with spark.sql.legacy.allowNegativeScaleOfDecimal=true + +## Signum + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DoubleType` is supported + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/misc.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/misc.md new file mode 100644 index 00000000000..f477a93886d --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/misc.md @@ -0,0 +1,92 @@ + + +# Miscellaneous Expressions + + + +## CheckOverflow + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `DecimalType` is supported + +## HllSketchEstimate + +The following incompatibilities cause `HllSketchEstimate` to fall back to Spark by default. Set `spark.comet.expression.HllSketchEstimate.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL estimates may differ slightly from Spark. +- Errors surface as plain Comet execution errors rather than Spark's SparkRuntimeException with condition HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail. +- An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere. + +## HllUnion + +The following incompatibilities cause `HllUnion` to fall back to Spark by default. Set `spark.comet.expression.HllUnion.allowIncompatible=true` to enable Comet acceleration despite these differences. + +- Comet uses a Rust DataSketches port; HLL sketch bytes and estimates may differ slightly from Spark. +- Errors surface as plain Comet execution errors rather than Spark's SparkRuntimeException with condition HLL_UNION_DIFFERENT_LG_K / HLL_INVALID_INPUT_SKETCH_BUFFER (sqlState 22000), so the message and error class differ even though both engines fail. +- An input sketch in the updatable HLL_4 form carrying auxiliary-map entries is rejected with an error, where Spark reads it: the bundled Rust decoder reads the compact auxiliary layout in both forms and would otherwise return a silently wrong estimate. Comet only ever writes HLL_8, so this affects sketch columns produced elsewhere. + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- The allowDifferentLgConfigK argument must be a foldable literal. + +## Literal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for literal values + +## MakeDecimal + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Only `LongType` input is supported + +## ScalarSubquery + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Not all data types are supported for scalar subquery results + +## SortOrder + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- A nested sort key with nondefault root null placement must be deterministic: the wrapper can evaluate the key twice, for isnull(key) and its value, which could disagree + +## StaticInvoke + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 10 is one such value: the result has 19 digits. The other `truncate` input types, and `bucket` on decimals, are unaffected. + +## ToPrettyString + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Falls back to Spark when the input type cannot be cast to string. + +## Uuid + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- uuid requires a resolved random seed + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/string.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/string.md new file mode 100644 index 00000000000..57d39cc2678 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/string.md @@ -0,0 +1,225 @@ + + +# String Expressions + + + +## BitLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## Concat + +By default, `Concat` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Concat.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- concat does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- CONCAT supports only string input parameters + +## ConcatWs + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- all arguments are foldable + +## Contains + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## EndsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## GetJsonObject + +By default, `GetJsonObject` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.GetJsonObject.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Spark allows single-quoted JSON and unescaped control characters which Comet does not support +- Very long numbers near Jackson's 1000-digit limit can depend on Spark's recycled parser buffer, which Comet cannot reproduce from the input alone +- Selected JSON integers outside the 64-bit range and very long numbers can lose precision or fail during Comet's native JSON materialization +- Some selected floating-point values can differ from Spark at decimal parsing or Java-version formatting boundaries +- When a returned object or array contains duplicate keys, Spark preserves them while Comet's native JSON materialization keeps only the last value + +## InitCap + +By default, `InitCap` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.InitCap.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Treats hyphen as a word separator (e.g. `robert rose-smith` produces `Robert Rose-Smith` instead of Spark's `Robert Rose-smith`) (https://github.com/apache/datafusion-comet/issues/1052) + +## Levenshtein + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-default (non-UTF8_BINARY) collated input. The native kernel compares raw bytes, so collation-aware comparison has no native path. + +## Like + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- LIKE with a custom escape character (only `\` is supported natively) +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## Lower + +By default, `Lower` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + +## OctetLength + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `BinaryType` input is not supported + +## RLike + +The following cases use Comet's native implementation by default: + +- A `UTF8_BINARY` literal pattern admitted by the [plan-time compatibility analyzer](../../regex.md#when-the-rust-engine-is-safe) is evaluated natively by default. + +For applicable cases that are not selected for native execution automatically, `RLike` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.RLike.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- For applicable literal patterns outside the automatically admitted subset, the native Rust regex engine may behave differently from Java regex. + +## RandStr + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `length` must be a literal value +- `length` must be non-negative +- `seed` must be a literal value + +## RegExpReplace + +By default, `RegExpReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.RegExpReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regexp pattern may not be compatible with Spark + +## Reverse + +By default, `Reverse` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.Reverse.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- native reverse does not support arrays whose element type contains binary, struct, or map +- reverse does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/2190) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- reverse has no native implementation for binary input + +## StartsWith + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Non-UTF8_BINARY collated operands are routed through the JVM codegen dispatcher (Spark's own `doGenCode`) because native comparison is byte-wise. + +## StringInstr + +For applicable cases that are not selected for native execution automatically, `StringInstr` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringInstr.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native instr compares raw bytes + +## StringLPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRPad + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Scalar values are not supported for the `str` argument. +- Only scalar values are supported for the `pad` argument. + +## StringRepeat + +The following differences from Spark are always present and do not require any additional configuration: + +- A negative argument for the number of times to repeat throws an exception instead of returning an empty string as Spark does + +## StringReplace + +The following cases use Comet's native implementation by default: + +- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default. + +For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast + +## StringSplit + +By default, `StringSplit` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringSplit.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Regex engine differences between Java and Rust + +## StringTranslate + +By default, `StringTranslate` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel + +## StringTrim + +For applicable cases that are not selected for native execution automatically, `StringTrim` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrim.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native trim with a trim string compares raw bytes + +## StringTrimLeft + +For applicable cases that are not selected for native execution automatically, `StringTrimLeft` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimLeft.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native ltrim with a trim string compares raw bytes + +## StringTrimRight + +For applicable cases that are not selected for native execution automatically, `StringTrimRight` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringTrimRight.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native rtrim with a trim string compares raw bytes + +## SubstringIndex + +For applicable cases that are not selected for native execution automatically, `SubstringIndex` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.SubstringIndex.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark: + +- Spark evaluates non-UTF8_BINARY collated string input under its collation, while Comet's native substring_index compares raw bytes + +## UnBase64 + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- unbase64 with failOnError = true uses stricter RFC 4648 validation that is not yet implemented natively +- unbase64 with a non-trivial child expression uses the JVM codegen dispatcher to preserve Spark's short-circuit evaluation (native path is limited to column and literal children) + +## Upper + +By default, `Upper` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.caseConversion.enabled=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Results can vary depending on locale and character set + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/struct.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/struct.md new file mode 100644 index 00000000000..c530b12a1a8 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/struct.md @@ -0,0 +1,52 @@ + + +# Struct Expressions + + + +## CreateNamedStruct + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- `CreateNamedStruct` with duplicate field names is not supported + +## JsonToStructs + +By default, `JsonToStructs` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.JsonToStructs.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Partially implemented and not comprehensively tested + +## StructsToCsv + +By default, `StructsToCsv` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToCsv.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Date, Timestamp, TimestampNTZ, and Binary data types may produce different results (https://github.com/apache/datafusion-comet/issues/3232) + +The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline): + +- Complex types (arrays, maps, structs) in the schema are not supported + +## StructsToJson + +By default, `StructsToJson` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StructsToJson.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark: + +- Does not support `+Infinity` and `-Infinity` for numeric types (float, double). (https://github.com/apache/datafusion-comet/issues/3016) + + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/url.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/url.md new file mode 100644 index 00000000000..765e12941fd --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/url.md @@ -0,0 +1,23 @@ + + +# URL Expressions + + + diff --git a/docs/source/user-guide/latest/configs.md b/docs/source/user-guide/latest/configs.md index 0fb0ec56c49..b7021e7b620 100644 --- a/docs/source/user-guide/latest/configs.md +++ b/docs/source/user-guide/latest/configs.md @@ -24,16 +24,84 @@ Comet provides the following configuration settings. ## Scan Configuration Settings + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.iceberg.ioTimeout` | Timeout for each storage I/O call, such as one read or write, that Comet's native Iceberg scan and write make. Passed to iceberg-rust as `opendal.io-timeout-ms`. | 30000ms | +| `spark.comet.scan.allowDisabledParquetVectorizedReader` | Whether to allow Comet's native scan to replace the Parquet scan when Spark's spark.sql.parquet.enableVectorizedReader is set to false. By default (false), Comet falls back to Spark in that case, because Comet's native readers mirror Spark's vectorized reader semantics rather than Spark's parquet-mr (non-vectorized) semantics, which permit silent overflow / null-on-narrowing that Comet has no equivalent for. For more information, refer to the [Comet Compatibility Guide](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html). | false | +| `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` | The number of Iceberg data files the native Iceberg scan reads concurrently within a single task. Reading several files at once overlaps the latency of opening each one, which speeds up tables with many small files and high-latency storage such as S3, but each file in flight holds its own buffers, so memory use grows with the limit. When it is above 1, the scan returns batches from different files in no particular order, so rows can come back in a different order than with 1. Set it to 1 to read the files one after another, in the order Spark assigned them. The Iceberg Spark test jobs pin it to 1 through the environment variable, because Iceberg's tests expect file order from queries without an ORDER BY. It can be overridden by the environment variable `COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT`. | 2 | +| `spark.comet.scan.icebergNative.enabled` | Whether to enable native Iceberg table scan using iceberg-rust. When enabled, Iceberg tables are read directly through native execution, bypassing Spark's DataSource V2 API for better performance. | true | +| `spark.comet.scan.icebergNative.nestedSchemaPruning.enabled` | Whether the native Iceberg scan reads only the nested fields of a struct, list, or map column that Spark's nested schema pruning keeps. When false, it reads every nested field of a projected column and drops the unused ones after decoding. | true | +| `spark.comet.scan.unsignedSmallIntSafetyCheck` | Parquet files may contain unsigned 8-bit integers (UINT_8) which Spark maps to ShortType. When this config is true (default), Comet falls back to Spark for ShortType columns because we cannot distinguish signed INT16 (safe) from unsigned UINT_8 (may produce different results). Set to false to allow native execution of ShortType columns if you know your data does not contain unsigned UINT_8 columns from improperly encoded Parquet files. For more information, refer to the [Comet Compatibility Guide](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html). | true | +| `spark.hadoop.fs.comet.libhdfs.schemes` | Defines filesystem schemes (e.g., hdfs, webhdfs) that the native side accesses via libhdfs, separated by commas. Valid only when built with hdfs-opendal feature enabled. Read from the Hadoop configuration, so it must be set before the SparkSession is created. A scheme listed only through a reader option is not read natively. libhdfs reads the settings of a listed scheme, such as `fs..impl`, from the executor's `core-site.xml`. | | +| `spark.hadoop.fs.comet.s3Compliant.schemes` | Defines filesystem schemes (e.g., blob, minio, r2) that Comet treats as S3-compliant, separated by commas. Such schemes reuse the `fs.s3a.*` credential surface and accept vendor-style `fs...*` keys. Empty by default, so no alias scheme is claimed unless opted in. Read from the Hadoop configuration, so it must be set before the SparkSession is created. | | + ## Parquet Reader Configuration Settings + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.parquet.prefetchNextFileMetadata.enabled` | When enabled, a native Parquet scan that reads several files in one task loads the next file's footer and page index in the background while it reads the current file, so opening the next file doesn't wait for a storage round trip. This helps most when tasks read many small files from high-latency storage such as S3. Scans of encrypted Parquet files never load ahead. | true | +| `spark.comet.parquet.rowFilterPushdown.enabled` | When enabled, the native Parquet reader evaluates pushed filters during decode and lazily materializes projected columns for surviving rows (DataFusion's pushdown_filters / late-materialization). Format-level pruning (row-group statistics, page index, bloom filters) runs whenever Spark's spark.sql.parquet.filterPushdown is enabled, except that while this flag is enabled, a pushed filter that compares FLOAT or DOUBLE values does not prune, because the reader must evaluate it with Spark's NaN and signed-zero semantics. Disabling this flag still lets format-level pruning work; the per-row eval falls back to the CometFilter operator above the scan. | false | + ## Query Execution Settings + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.caseConversion.enabled` | Java uses locale-specific rules when converting strings to upper or lower case and Rust does not, so we disable upper and lower by default. | false | +| `spark.comet.convert.broadcastBuildSide.enabled` | When enabled, Comet converts a file or V2 scan that it cannot scan natively (for example an ORC source) to Arrow when that scan feeds a broadcast join's build side and every scan on the probe side is already native, so the broadcast join runs natively over the large probe. The build side is usually small (auto-broadcasts are capped by the broadcast threshold), so the row to Arrow copy is usually cheap - though an explicit BROADCAST hint or a selective filter above the scan can make it larger. With dynamic partition pruning the dimension is also read a second time for the pruning subquery. Parquet, JSON, CSV and Text scans are converted only by their own `spark.comet.convert` configs. Set this to false to keep such build sides on Spark. | true | +| `spark.comet.convert.csv.enabled` | When enabled, data from Spark (non-native) CSV v1 and v2 scans will be converted to Arrow format. | false | +| `spark.comet.convert.inMemoryCache.enabled` | When enabled, data from in-memory cached tables that Comet's native cache scan does not read (see `spark.comet.exec.inMemoryCache.enabled`), such as tables cached in Spark's default format, will be converted to Arrow format. | false | +| `spark.comet.convert.json.enabled` | When enabled, data from Spark (non-native) JSON v1 and v2 scans will be converted to Arrow format. | false | +| `spark.comet.convert.oneRowRelation.enabled` | When enabled, the single row that a query without a FROM clause, such as `SELECT 1`, reads will be converted to Arrow format. The row has no columns, so converting it costs almost nothing, and it lets the operators above it run in Comet. | true | +| `spark.comet.convert.parquet.enabled` | When enabled, data from Spark (non-native) Parquet v1 and v2 scans will be converted to Arrow format. | false | +| `spark.comet.convert.range.enabled` | When enabled, the rows of `spark.range` and SQL `range()` that Comet does not generate natively (see `spark.comet.exec.range.enabled`) will be converted to Arrow format. | false | +| `spark.comet.convert.rdd.enabled` | When enabled, data from a DataFrame created from an RDD of rows, for example with `spark.createDataFrame(rdd, schema)`, will be converted to Arrow format. | false | +| `spark.comet.convert.rowDataSource.enabled` | When enabled, data from Data Source V1 relations that are not file-based, such as JDBC tables, will be converted to Arrow format. Spark scans these relations with `RowDataSourceScanExec`. | false | +| `spark.comet.convert.shuffleInput.enabled` | When enabled, a shuffle that would use Comet's JVM columnar shuffle because its input comes from a Spark operator converts that input to Arrow format and uses Comet native shuffle instead. This applies only where native shuffle supports the partitioning and Comet can convert all of the input's columns, which rules out calendar intervals. A shuffle that hashes a string or a value computed from a string stays on the JVM columnar shuffle, so that it partitions rows as Spark does. | false | +| `spark.comet.convert.text.enabled` | When enabled, data from Spark (non-native) Text v1 and v2 scans will be converted to Arrow format. Arrow strings must be valid UTF-8, so Comet replaces bytes that are not valid UTF-8, for example in a Latin-1 file, with U+FFFD, and results that depend on the original bytes can differ from Spark. Enable this only for Text sources that are UTF-8. See the compatibility guide. | false | +| `spark.comet.convert.typedDataset.enabled` | When enabled, the output of typed Dataset operations, such as `map`, `flatMap`, `mapPartitions` and `groupByKey(...).mapGroups`, will be converted to Arrow format so that the operators above them can run natively. The user function still runs in Spark. This pays off when the operators above do enough work, such as an aggregation over many groups, and can be slower when they are cheap, such as an aggregation over a few groups after a selective filter. | false | +| `spark.comet.debug.enabled` | Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose. | false | +| `spark.comet.enabled` | Whether to enable Comet extension for Spark. When this is turned on, Spark will use Comet to read Parquet data source. Note that to enable native vectorized execution, both this config and `spark.comet.exec.enabled` need to be enabled. It can be overridden by the environment variable `ENABLE_COMET`. | true | +| `spark.comet.exec.aggregate.skipPartial.enabled` | Experimental opt-in: let a native partial aggregate stop aggregating once its input looks mostly distinct, and send the rest of the task's rows to the shuffle unaggregated. Only applies to partial aggregates that feed Comet native shuffle and whose aggregate functions, if any, are all single-argument COUNT. The check starts after the first 100,000 input rows of a task, and once aggregation stops it does not resume, so a task whose keys repeat after a mostly distinct start can shuffle many times more rows than it would with this disabled. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | false | +| `spark.comet.exec.broadcastExchange.shareDecoded.enabled` | Whether the tasks of an executor that read a Comet broadcast at the same time share one decoded copy of it. The first of them decodes the broadcast, and the others wait for it instead of decoding it themselves. The copy is freed when the last of them completes. When disabled, every task decodes the whole broadcast for itself. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | true | +| `spark.comet.exec.columnarToRow.direct.enabled` | Experimental: Whether to use the direct columnar to row converter in the JVM columnar to row operator's non-codegen paths (including broadcast relation builds). The direct converter writes values straight from Arrow buffers into UnsafeRow format without per-value object allocation, which is significantly faster for decimal-heavy schemas. Schemas with unsupported data types fall back to the default conversion. Only applies when the JVM columnar to row operator is used (spark.comet.exec.columnarToRow.native.enabled=false). | false | +| `spark.comet.exec.columnarToRow.direct.minBatchSize` | Experimental: Batches with fewer rows than this use the default conversion even when spark.comet.exec.columnarToRow.direct.enabled is true, since the direct converter's per-batch setup does not amortize on very small batches. Only applies when the direct columnar to row converter is enabled. | 128 | +| `spark.comet.exec.columnarToRow.native.enabled` | Whether to enable native columnar to row conversion. When enabled, Comet will use native Rust code to convert Arrow columnar data to Spark UnsafeRow format instead of the JVM implementation. The native conversion carries a fixed JNI cost per batch and is slower than the JVM implementation for small batches. | false | +| `spark.comet.exec.enabled` | Whether to enable Comet native vectorized execution for Spark. This controls whether Spark should convert operators into their Comet counterparts and execute them in native space. Note: each operator is associated with a separate config in the format of `spark.comet.exec..enabled` at the moment, and both the config and this need to be turned on, in order for the operator to be executed in native. | true | +| `spark.comet.exec.forceShuffledHashJoin` | Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance. This feature is not stable yet. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | false | +| `spark.comet.exec.forceShuffledHashJoin.maxBuildSize` | The build side size below which `spark.comet.exec.forceShuffledHashJoin` converts a SortMergeJoin to ShuffledHashJoin. The size is Spark's planning estimate of the build child, or under AQE the materialized shuffle size of the build side. A build side at or over this size, or one with no statistics, keeps the SortMergeJoin. When unset, Spark's own rule applies: `spark.sql.autoBroadcastJoinThreshold` times the initial shuffle partition count (`spark.sql.adaptive.coalescePartitions.initialPartitionNum` when AQE and partition coalescing are both on and it is set, else `spark.sql.shuffle.partitions`). When broadcasts are disabled with a non-positive threshold, Spark's default threshold of 10 MB is used instead. A non-positive value removes the limit. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | | +| `spark.comet.exec.inMemoryCache.compression.codec` | The Arrow IPC compression codec used when Comet's cache serializer writes cached data: zstd, lz4 or none. Unlike spark.io.compression.codec, this compresses each Arrow buffer separately rather than the batch as a whole, which is what lets a projected scan decompress only the columns it selected. zstd is the default for footprint rather than speed: it stores cached data in a fraction of the memory, but makes materializing slower, and a read slower in proportion to how many columns it selects. lz4 decompresses several times faster than zstd, which makes wide reads much faster, at around three times zstd's footprint. It uses lz4-java, the JNI-accelerated library behind spark.io.compression.codec, rather than Arrow's own pure-Java lz4. Set to none when a relation fits in memory uncompressed and is read at close to full width. Only affects newly cached data; the codec a batch was written with is recorded in the batch itself and is what the read path uses. | zstd | +| `spark.comet.exec.inMemoryCache.compression.zstd.level` | The compression level to use when Comet's cache serializer compresses cached data with zstd. Ignored for other codecs. | 1 | +| `spark.comet.exec.inMemoryCache.enabled` | Whether to enable Comet native scans and fused Spark reads of in-memory cached tables. Requires spark.comet.enabled=true. On Spark 3.4, AQE cannot coalesce the shuffle partitions of a union that Comet runs with a cached relation in one branch. At startup, this setting also decides whether CometDriverPlugin installs Comet's cache serializer, which stores cached data in Arrow format. The plugin installs it only if spark.comet.enabled and spark.comet.exec.enabled are also enabled at startup, and only with one of Comet's shuffle managers while Comet shuffle is enabled. Because spark.sql.cache.serializer is a static config, the cached format is fixed for the application, and disabling this or spark.comet.enabled at runtime sends cached scans back to Spark's execution path without the fused reader. Relations whose schema Comet's Arrow writer does not support are always cached in Spark's default format. Each cached batch is stored as one Arrow IPC record batch with per-buffer compression, and a scan copies out only the buffers of the columns it projected, so the unselected ones are never decompressed. Eligible Spark whole-stage codegen consumers read cached vectors directly when vectorized cache reading is enabled; other Spark row consumers use a reusable row buffer. Decoding costs can still make wide numeric reads slower than Spark's default cache. With spark.kryo.registrationRequired=true, the plugin installs it only if Kryo has registered Comet's cached batch, as spark.kryo.registrator=org.apache.comet.CometKryoRegistrator does when set before creating the SparkContext, because Kryo would otherwise reject a cached block as soon as it is serialized, including the disk half of the default MEMORY_AND_DISK storage level. | false | +| `spark.comet.exec.join.dynamicFilter.allowDeterministicFilterPushdown` | Allow native join runtime filters to reach Parquet readers through deterministic residual predicates. Requires spark.comet.exec.join.dynamicFilter.enabled. This can skip expression errors on rows eliminated before the residual runs, including division-by-zero and ANSI cast errors. Errors on retained rows and Parquet schema conversion checks are preserved. Nondeterministic predicates, all standalone projections (including column-only projections), and limits remain reader propagation boundaries. When false, only direct-column IS NOT NULL checks combined with AND can be crossed. | false | +| `spark.comet.exec.join.dynamicFilter.enabled` | Experimental opt-in: use a completed native hash join build's key domain to prune eligible native Parquet row groups and filter probe batches before the hash probe. Supports inner joins with one direct signed integer key and one native partition per input, including both Spark build sides. The probe filter remains active when reader pruning is unavailable. Unsupported joins retain their existing execution path. Filters do not cross Spark exchanges or JVM/Arrow boundaries. | false | +| `spark.comet.exec.nativeArrowPythonUDF.enabled` | Experimental: execute Spark 4.1+ scalar @arrow_udf functions inside the Comet native pipeline via PyO3. Requires a native library built with the python-udf Cargo feature and a compatible Python/PyArrow installation on every executor. | false | +| `spark.comet.exec.pyarrowUDF.enabled` | Experimental: whether to enable optimized execution of PyArrow UDFs (mapInArrow/mapInPandas). When enabled, Comet passes Arrow columnar data directly to Python UDFs without the intermediate Arrow-to-Row-to-Arrow conversion that Spark normally performs. Disabled by default while the feature stabilizes. | false | +| `spark.comet.exec.scalaUDF.codegen.enabled` | Whether to route Spark `ScalaUDF` expressions through Comet's Arrow-direct codegen dispatcher. When enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads and writes Arrow vectors directly from native execution. When disabled, plans containing a ScalaUDF fall back to Spark for the enclosing operator. The same dispatcher backs the regex family (`rlike`, `regexp_replace`, `split`, `regexp_extract`, `regexp_extract_all`, `regexp_instr`) so those route through it by default as well. | true | +| `spark.comet.exec.strictFloatingPoint` | When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as comparing -0.0 and 0.0. `ORDER BY`, window ordering and range partitioning keys are unaffected, including floating-point values nested in arrays and structs, because Comet normalizes those comparison keys to match Spark. `sort_array` is unaffected too, because it follows Spark's ordering. For more information, refer to the [Comet Compatibility Guide](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html). | false | +| `spark.comet.exec.topK.dynamicFilter.enabled` | Experimental opt-in: use a local TopK heap's improving threshold to prune eligible native Parquet input. Requires spark.comet.exec.topK.fusion.enabled. Supports one direct signed integer sort key. Thresholds are local to each native execution and do not cross Spark exchanges or JVM/Arrow boundaries. Unsupported readers retain ordinary TopK execution. | false | +| `spark.comet.exec.topK.fusion.enabled` | Experimental opt-in: run an eligible local TopK in the same native execution as its Parquet scan. Supports one direct signed integer sort key. This changes the local execution pipeline and can reduce scan/TopK overlap, so it may be slower for some workloads. | false | +| `spark.comet.exec.transitionRevert.enabled` | When enabled, Comet reverts a query stage to Spark row-based execution if the number of columnar-to-row (C2R) transitions in the stage exceeds the configured threshold. This avoids the overhead of repeated format conversions in stages where many operators fall back to row-based execution. | false | +| `spark.comet.exec.transitionRevert.maxTransitions` | The maximum number of columnar-to-row (C2R) transitions allowed in a single query stage before Comet reverts the entire stage to Spark row-based execution. When columnar shuffle is enabled, each such C2R typically implies a corresponding row-to-columnar conversion to feed back into the columnar shuffle, so each counted C2R is a useful proxy for the conversion overhead in the stage. Set to 0 to revert any stage with transitions. Only effective when spark.comet.exec.transitionRevert.enabled is true. | 2 | +| `spark.comet.metrics.enabled` | Whether to enable Comet metrics reporting through Spark's external monitoring system. When enabled, Comet exposes metrics such as native operators, Spark operators, queries planned, transitions, and acceleration ratio. These metrics can be visualized through tools like Grafana when a metrics sink (e.g., Prometheus) is configured. Disabled by default because Spark plan traversal adds overhead and metrics require a sink to be useful. This config must be set before the SparkSession is created to take effect. | false | +| `spark.comet.metrics.updateInterval` | The interval in milliseconds to update metrics. If interval is negative, metrics will be updated upon task completion. | 3000 | +| `spark.comet.nativeLoadRequired` | Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted. | false | +| `spark.comet.nativeUdf.allowedPaths` | A comma-separated list of directories that native UDF libraries may be loaded from. When set, a library is loaded only if its path is absolute and, after symlinks and `..` are resolved, lies under one of these directories, which are themselves resolved the same way. A path outside them fails on the driver at registration and again on the executor. When empty, any path is allowed. This limits which files Comet will load, not what the code in them does, and it cannot protect against a file that is replaced after the check, so the directories should be writable only by trusted users. Like any session config, it can be changed by a user who can set Spark configs. | | +| `spark.comet.nativeUdf.enabled` | Whether native UDF libraries may be loaded. A native UDF is a shared library that runs with the full privileges of the JVM process, so an operator who does not want any can turn the feature off. When false, `CometNativeUDF.register` fails on the driver, and a plan that names a native UDF library fails on the executor rather than loading it. | true | +| `spark.comet.operator.DataWritingCommandExec.allowIncompatible` | Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE`. | false | +| `spark.comet.operator.WriteFilesExec.allowIncompatible` | Whether to allow incompatibility for operator: WriteFilesExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE`. | false | +| `spark.comet.sparkToColumnar.enabled` | Whether to convert the output of the leaf operators named in `spark.comet.sparkToColumnar.supportedOperatorList` to Arrow format. Ranges, in-memory cached tables, RDDs and queries without a FROM clause have `spark.comet.convert` configs of their own. When the list is not set, this converts those four instead, which is deprecated and will be removed in a future major release. | false | +| `spark.comet.sparkToColumnar.supportedOperatorList` | A comma-separated list of leaf operators to convert to Arrow format when `spark.comet.sparkToColumnar.enabled` is true, each named by its Spark class name without the `Exec` suffix. It is for operators that have no `spark.comet.convert` config of their own, such as `BatchScan` for a Data Source V2 connector, or `FileSourceScan` for file formats other than Parquet, JSON, CSV and Text. Naming `Range`, `InMemoryTableScan`, `RDDScan`, `OneRowRelation` or `RowDataSourceScan`, or naming `FileSourceScan` or `BatchScan` to convert a Text scan, is deprecated and will stop working in a future major release. Use their `spark.comet.convert` configs instead. | | +| `spark.comet.write.iceberg.enabled` | Whether Comet plans Iceberg writes and writes their data files natively. When enabled, Comet replaces Spark's combined V2 write operator with a file writer (inside AQE) under a committer (outside AQE), and writes the data files of each eligible write with its native (iceberg-rust) writer. Other writes use Iceberg's own writer, and Iceberg commits every write. Merge-on-read writes keep Spark's operator. When disabled, Spark plans its own V2 write operator. | false | + ## Viewing Explain Plan & Fallback Reasons @@ -41,6 +109,17 @@ Comet provides the following configuration settings. These settings can be used to determine which parts of the plan are accelerated by Comet and to see why some parts of the plan could not be supported by Comet. + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.explain.codegen.enabled` | When enabled, Comet annotates the surrounding Comet operator with a `[COMET-INFO: JVM codegen dispatcher: ]` segment listing every expression it routed through the JVM codegen dispatcher. Disabled by default. | false | +| `spark.comet.explain.fallback.enabled` | When this setting is enabled, Comet will provide logging explaining the reason(s) why a query stage cannot be executed natively. Set this to false to reduce the amount of logging. | false | +| `spark.comet.explain.fallback.log.enabled` | When this setting is enabled, Comet will log warnings for all fallback reasons. It can be overridden by the environment variable `ENABLE_COMET_LOG_FALLBACK_REASONS`. | false | +| `spark.comet.explain.format` | Choose extended explain output. The default format of 'verbose' will provide the full query plan annotated with fallback reasons as well as a summary of how much of the plan was accelerated by Comet. The format 'fallback' provides a list of fallback reasons instead. | verbose | +| `spark.comet.explain.native.enabled` | When this setting is enabled, Comet will provide a tree representation of the native query plan before execution and again after execution, with metrics. | false | +| `spark.comet.explain.planOnly.enabled` | When enabled, Comet logs the plan it would have executed, with a coverage summary, to the driver log and then lets Spark execute the query unchanged. Native planning failures are not detected, so the coverage can be optimistic. Requires `spark.comet.exec.enabled=true`. | false | +| `spark.comet.explain.rules` | When this setting is enabled, Comet will log all plan transformations performed in physical optimizer rules. Default: false | false | + ## Shuffle Configuration Settings @@ -51,11 +130,50 @@ The [native remote shuffle tuning reference](tuning/celeborn.md) explains how interact, including encoding workspace and local fallback for oversized rows. + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.shuffle.compression.codec` | The codec of Comet native shuffle used to compress shuffle data. lz4, zstd, and snappy are supported. Compression can be disabled by setting spark.shuffle.compress=false. | lz4 | +| `spark.comet.shuffle.compression.offsetsAsLengths.enabled` | Whether Comet shuffle stores the offsets of string, binary, array and map columns as lengths before compressing each block. Lengths compress much better than offsets, which makes shuffle files with such columns smaller. Only takes effect when spark.comet.shuffle.compression.codec is zstd, and not for Celeborn shuffle or for JVM shuffle blocks with dictionary-encoded columns, which keep their offsets. | true | +| `spark.comet.shuffle.compression.zstd.level` | The compression level to use when compressing shuffle files with zstd. | 1 | +| `spark.comet.shuffle.convertFromSparkPlan.enabled` | When enabled, Comet will convert a Spark `ShuffleExchangeExec` to a Comet columnar shuffle even when its child is a non-Comet (Spark) plan. Disable to leave such shuffles as native Spark shuffles, restricting Comet shuffle to cases where the child is already a Comet plan. | true | +| `spark.comet.shuffle.directRead.enabled` | When enabled, native operators that consume shuffle output will read compressed shuffle blocks directly in native code, bypassing Arrow FFI. Applies to both native shuffle and JVM columnar shuffle. Requires spark.comet.shuffle.enabled to be true. | true | +| `spark.comet.shuffle.enabled` | Whether to enable Comet shuffle. Note that this requires setting `spark.shuffle.manager` to `org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager` or `org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager`. Celeborn requires explicit native mode and compatible application settings. `spark.shuffle.manager` must be set before starting the Spark application and cannot be changed during the application. | true | +| `spark.comet.shuffle.jvm.batchSize` | Batch size when writing out sorted spill files on the native side. A value larger than the batch size (i.e., `spark.comet.batchSize`) is capped at the batch size, so that the native operators after the shuffle do not receive larger batches than expected. | 8192 | +| `spark.comet.shuffle.jvm.maxWritersPerExecutor` | Maximum number of concurrent hash-based shuffle writers per executor. Comet's hash-based (bypass merge sort) columnar shuffle allocates one writer per partition, so `executor cores * numPartitions` writers can be active concurrently. When that product exceeds this cap, Comet falls back to sort-based shuffle to avoid OOM. | 100 | +| `spark.comet.shuffle.jvm.preferDictionary.ratio` | The ratio of total values to distinct values in a string or binary column to decide whether to prefer dictionary encoding when shuffling the column. If the ratio is higher than this config, dictionary encoding will be used when shuffling the column. This config is effective if it is higher than 1.0. Note that this config is only used when `spark.comet.shuffle.mode` is `jvm`. | 10.0 | +| `spark.comet.shuffle.mode` | Select the Comet shuffle implementation: `jvm` for Columnar Shuffle, `native` for Native Shuffle, and `auto` to pick the best supported option (`native` has priority). With CometCelebornShuffleManager, only explicit `native` mode enables Comet shuffle; `auto` and `jvm` retain Spark/Celeborn shuffle. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | auto | +| `spark.comet.shuffle.native.maxBufferBytes` | Maximum number of bytes that the native shuffle writer will buffer in memory before spilling to disk. The default of 0 disables this limit, in which case spilling is triggered only when the memory pool denies an allocation. Setting a limit caps the writer's memory footprint independently of the size of the memory pool, at the cost of more frequent spilling. | 0b | +| `spark.comet.shuffle.native.partitioning.hash.enabled` | Whether to enable hash partitioning for Comet native shuffle. | true | +| `spark.comet.shuffle.native.partitioning.hash.nested.enabled` | Whether to allow nested types (struct, array, map) as hash partitioning keys in Comet native shuffle. Comet's native Murmur3 kernel hashes nested types recursively and shares that kernel, and Spark's seed, with the `hash` expression, so partition assignment matches Spark. A map key additionally requires the `mapsort` normalization that Spark 4.0 and later insert, so maps are rejected on earlier versions. Disabled by default until the performance of the nested hashing paths has been measured: shapes whose leaves are not primitives, such as `array>`, fall back to a per-element code path in the native hasher rather than a vectorized one. | false | +| `spark.comet.shuffle.native.partitioning.range.enabled` | Whether to enable range partitioning for Comet native shuffle. | true | +| `spark.comet.shuffle.native.partitioning.roundrobin.enabled` | Whether to enable round robin partitioning for Comet native shuffle. This is disabled by default because Comet's round-robin produces different partition assignments than Spark. Spark sorts rows by their binary UnsafeRow representation before assigning partitions, but Comet uses Arrow format which has a different binary layout. Instead, Comet implements round-robin as hash partitioning on all columns, which achieves the same goals: even distribution, deterministic output (for fault tolerance), and no semantic grouping. Sorted output will be identical to Spark, but unsorted row ordering may differ. | false | +| `spark.comet.shuffle.native.partitioning.roundrobin.maxHashColumns` | The maximum number of columns to hash for round robin partitioning. When set to 0 (the default), all columns are hashed. When set to a positive value, only the first N columns are used for hashing, which can improve performance for wide tables while still providing reasonable distribution. | 0 | +| `spark.comet.shuffle.native.partitioning.roundrobin.positional.enabled` | When true, Comet's native round-robin shuffle places rows by position rather than by hashing their contents, sending each map task's rows to the output partitions in turn, in contiguous groups. This skips a murmur3 pass over every column of every row and replaces the per-row gather on flush with a bulk copy per group, which is what dominates the shuffle write on wide nested schemas, and it spreads duplicate rows evenly where hashing sends them all to one partition. Placement then depends on the order a map task reads its rows in, so it is only used where Comet can establish from the plan that a retried task reads them in the same order: a native scan under nothing but deterministic projections and filters, and not with the Celeborn shuffle manager. Any other plan keeps content-hash placement. Has no effect unless spark.comet.shuffle.native.partitioning.roundrobin.enabled is also true. | false | +| `spark.comet.shuffle.native.partitioning.roundrobin.positional.groupRows` | Rows per contiguous group under positional round robin. Smaller groups balance better and larger ones are cheaper to copy. Within one map task, output partitions differ by at most this many rows, but a reducer receives groups from every map task, so the stage is only evenly balanced when each task emits many more groups than there are output partitions; a group approaching a task's whole input skews the stage and can leave reducers empty. When set to 0 (the default) Comet derives it from the batch size and the partition count when the query is planned, which keeps each task wrapping around the output partitions about once per batch. The derived group is at least 64 rows, so with more than a sixty-fourth of the batch size in output partitions a task needs several batches to wrap once, and map tasks of only a few batches can still leave reducers empty. Only applies when spark.comet.shuffle.native.partitioning.roundrobin.positional.enabled is true. | 0 | +| `spark.comet.shuffle.native.writeBufferSize` | Size of the write buffer in bytes used by the native shuffle writer when writing shuffle data to disk. Larger values may improve write performance by reducing the number of system calls, but will use more memory. The default is 1MB which provides a good balance between performance and memory usage. | 1048576b | +| `spark.comet.shuffle.readBufferSize` | Size in bytes of each read from a shuffle block stream. Comet reads a shuffle block through a reusable buffer of this size per task thread before decoding it. | 65536b | +| `spark.comet.shuffle.revertRedundantColumnar.enabled` | When enabled, Comet reverts a `CometShuffleExchangeExec` with `CometColumnarShuffle` back to Spark's `ShuffleExchangeExec` when both its parent and child are non-Comet aggregate operators. The same applies to a native shuffle whose input `spark.comet.convert.shuffleInput.enabled` converted. This avoids a redundant row -> Arrow -> shuffle -> Arrow -> row conversion when no Comet operator on either side can consume columnar output. Disable to keep the Comet shuffle even in that case, which preserves Comet's off-heap shuffle memory accounting at the cost of the extra conversion. | true | +| `spark.comet.shuffle.rss.maxFrameBytes` | Maximum encoded size of one complete native shuffle frame sent to a remote shuffle service. Frames are never split across remote push requests. | 67108864b | +| `spark.comet.shuffle.rss.maxInFlightBytes` | Maximum shuffle bytes admitted concurrently by native Comet map attempts sharing an executor-side remote shuffle client. Admission includes native encoding scratch and overlapping native, JNI, and remote shuffle frame copies. A frame must fit its codec and Arrow workspace as well as its encoded bytes; ordinary uncompressed frames need approximately seven times their size plus schema and transport overhead. Compressed frames also reserve workspace for their uncompressed data. Admission is acquired before encoding. Encrypted native RSS is not supported; use ordinary Spark shuffle when spark.io.encryption.enabled is true. | 536870912b | +| `spark.comet.shuffle.sizeInBytesMultiplier` | Comet reports smaller sizes for shuffle due to using Arrow's columnar memory format and this can result in Spark choosing a different join strategy due to the estimated size of the exchange being smaller. Comet will multiple sizeInBytes by this amount to avoid regressions in join strategy. | 1.0 | + ## Memory & Tuning Configuration Settings + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.batchSize` | The columnar batch size, i.e., the maximum number of rows that a batch can contain. | 8192 | +| `spark.comet.exec.memoryPool` | The type of memory pool to be used for Comet native execution when running Spark in off-heap mode. Available pool types are `greedy_unified` and `fair_unified`. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | fair_unified | +| `spark.comet.exec.memoryPool.fraction` | Deprecated: this config will be removed in a future major release. It does not leave room in spark.memory.offHeap.size for native memory that Comet's memory pools do not track, because Spark hands out the whole off-heap pool whatever this is set to. Size spark.executor.memoryOverhead for that memory instead. Only applies to off-heap mode, where the fair_unified pool limits each memory consumer in a task to this fraction of the off-heap size divided by the task's consumers, and the greedy_unified pool ignores it. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | 1.0 | +| `spark.comet.exec.spill.compression.codec` | The codec used to compress the files that native sorts, aggregations, and joins spill to disk. lz4 and zstd are supported, and none disables compression. zstd writes smaller files than lz4 but uses more CPU. Comet's shuffle writers use spark.comet.shuffle.compression.codec instead. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | lz4 | +| `spark.comet.maxTempDirectorySize` | The maximum amount of data (in bytes) stored inside the temporary directories used by native operators when spilling. Applied to each Comet native plan separately, and a Spark task can run more than one native plan at a time, so an executor running N concurrent tasks may use more than N times this value on shared local disks. Once the limit is reached, further spills will fail and the query will error out. | 107374182400b | +| `spark.comet.memory.logInterval` | How often each executor logs its native memory usage at INFO level while Comet native plans are running: the bytes the native allocator has handed out, the bytes reserved in Comet's memory pools, and the Arrow memory Comet holds on the JVM side. The difference between the first two is native memory that the pools are not accounting for. The executor logs one line per interval however many tasks are running, and one more after the last plan finishes. It logs a warning when the native memory looks larger than the executor's container allows. When spark.eventLog.enabled is true and the application runs the Comet plugin, the executor also sends its samples to the driver, which writes the one with the most untracked memory and the last of every minute to the event log, as CometExecutorMemoryUsage events. This is an executor setting, read when an executor starts its first Comet native plan, so it must be set when the application is submitted. An invalid value disables the log with a warning. Set to 0 to disable. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | 10000ms | +| `spark.comet.tracing.enabled` | Enable fine-grained tracing of events and memory usage. For more information, refer to the [Comet Tracing Guide](https://datafusion.apache.org/comet/contributor-guide/tracing.html). | false | + ## Development & Testing Settings @@ -74,19 +192,385 @@ this page does not by itself mean that, though: the per-expression them like any other production setting. + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.debug.memory` | When enabled, log all native memory pool interactions. For more information, refer to the Comet Debugging Guide (https://datafusion.apache.org/comet/contributor-guide/debugging.html). | false | +| `spark.comet.exec.onHeap.enabled` | Whether to allow Comet to run in on-heap mode. Required for running Spark SQL tests. Comet performs no memory accounting in on-heap mode, so its allocations are bounded by nothing; this must not be used in production. It can be overridden by the environment variable `ENABLE_COMET_ONHEAP`. | false | +| `spark.comet.exec.respectDataFusionConfigs` | Development and testing configuration option to allow DataFusion configs set in Spark configuration settings starting with `spark.comet.datafusion.` to be passed into native execution. | false | +| `spark.comet.scan.csv.v2.enabled` | Whether to use the native Comet V2 CSV reader for improved performance. Default: false (uses standard Spark CSV reader) Experimental: Performance benefits are workload-dependent. | false | +| `spark.comet.scan.enabled` | Whether to enable native scans. Intended for use in Comet's own test suites to selectively disable native scans; not intended for production use. | true | +| `spark.comet.testing.strict` | Experimental option to enable strict testing, which will fail tests that could be more comprehensive, such as checking for a specific fallback reason. It can be overridden by the environment variable `ENABLE_COMET_STRICT_TESTING`. | false | +| `spark.comet.write.iceberg.splitOperator.enabled` | Whether to plan Iceberg writes as Comet's file writer and committer even when `spark.comet.write.iceberg.enabled` is false, so that Iceberg's own writer writes every data file inside Comet's plan. It is also what plans merge-on-read writes on Spark 3.5+ that way. Used by tests to compare the two writers under the same plan. | false | +| `spark.comet.write.parquet.enabled` | Whether to enable native Parquet write through Comet. When enabled, Comet will intercept Parquet write operations and execute them natively. This feature is highly experimental and only partially implemented. It should not be used in production. It can be overridden by the environment variable `ENABLE_COMET_WRITE`. | false | + ## Enabling or Disabling Individual Operators + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.exec.aggregate.enabled` | Whether to enable aggregate by default. | true | +| `spark.comet.exec.broadcastExchange.enabled` | Whether to enable broadcastExchange by default. | true | +| `spark.comet.exec.broadcastHashJoin.enabled` | Whether to enable broadcastHashJoin by default. | true | +| `spark.comet.exec.broadcastNestedLoopJoin.enabled` | Whether to enable broadcastNestedLoopJoin by default. | true | +| `spark.comet.exec.coalesce.enabled` | Whether to enable coalesce by default. | true | +| `spark.comet.exec.collectLimit.enabled` | Whether to enable collectLimit by default. | true | +| `spark.comet.exec.emptyRelation.enabled` | Whether to enable emptyRelation by default. | true | +| `spark.comet.exec.existenceJoin.enabled` | Whether to enable native ExistenceJoin, produced when EXISTS / NOT EXISTS / IN is combined with another predicate via OR. Sort-merge joins, residual (non-equi) join conditions, computed (non-column) join keys, and NOT IN combined with OR fall back to Spark. | true | +| `spark.comet.exec.expand.enabled` | Whether to enable expand by default. | true | +| `spark.comet.exec.explode.enabled` | Whether to enable explode by default. | true | +| `spark.comet.exec.filter.enabled` | Whether to enable filter by default. | true | +| `spark.comet.exec.globalLimit.enabled` | Whether to enable globalLimit by default. | true | +| `spark.comet.exec.hashJoin.enabled` | Whether to enable hashJoin by default. | true | +| `spark.comet.exec.localLimit.enabled` | Whether to enable localLimit by default. | true | +| `spark.comet.exec.localTableScan.enabled` | Whether to enable localTableScan by default. | false | +| `spark.comet.exec.mergeRows.enabled` | Whether to enable mergeRows by default. Only takes effect on Spark 3.5 and later. Spark 3.4 has no MergeRowsExec. On Spark 4.1 and later, stock V2 writers retain Spark MergeRowsExec for MergeSummary; native MergeRows is used only where the enclosing Comet write path preserves that contract.. | false | +| `spark.comet.exec.project.enabled` | Whether to enable project by default. | true | +| `spark.comet.exec.range.enabled` | Whether to enable range by default. When enabled, Comet generates the rows of `spark.range` and SQL `range()` in native code, so the operators above them can run natively. It is off by default because it can be slower than Spark when those operators are only cheap expressions, such as a filter, which Spark compiles together with the range into one loop. | false | +| `spark.comet.exec.sample.enabled` | Whether to enable sample by default. | true | +| `spark.comet.exec.sort.enabled` | Whether to enable sort by default. | true | +| `spark.comet.exec.sortMergeJoin.enabled` | Whether to enable sortMergeJoin by default. | true | +| `spark.comet.exec.sortMergeJoinWithJoinFilter.enabled` | Support for Sort Merge Join with filter. Deprecated: this config will be removed in a future major release. | true | +| `spark.comet.exec.takeOrderedAndProject.enabled` | Whether to enable takeOrderedAndProject by default. | true | +| `spark.comet.exec.union.enabled` | Whether to enable union by default. | true | +| `spark.comet.exec.window.enabled` | Whether to enable window by default. | true | +| `spark.comet.exec.windowGroupLimit.enabled` | Whether to enable windowGroupLimit by default. | true | + ## Enabling or Disabling Individual Scalar Expressions + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.expression.Abs.enabled` | Enable Comet acceleration for `Abs` | true | +| `spark.comet.expression.Acos.enabled` | Enable Comet acceleration for `Acos` | true | +| `spark.comet.expression.Acosh.enabled` | Enable Comet acceleration for `Acosh` | true | +| `spark.comet.expression.Add.enabled` | Enable Comet acceleration for `Add` | true | +| `spark.comet.expression.AddMonths.enabled` | Enable Comet acceleration for `AddMonths` | true | +| `spark.comet.expression.Alias.enabled` | Enable Comet acceleration for `Alias` | true | +| `spark.comet.expression.And.enabled` | Enable Comet acceleration for `And` | true | +| `spark.comet.expression.ApplyFunctionExpression.enabled` | Enable Comet acceleration for `ApplyFunctionExpression` | true | +| `spark.comet.expression.ArrayAggregate.enabled` | Enable Comet acceleration for `ArrayAggregate` | true | +| `spark.comet.expression.ArrayAppend.enabled` | Enable Comet acceleration for `ArrayAppend` | true | +| `spark.comet.expression.ArrayContains.enabled` | Enable Comet acceleration for `ArrayContains` | true | +| `spark.comet.expression.ArrayDistinct.enabled` | Enable Comet acceleration for `ArrayDistinct` | true | +| `spark.comet.expression.ArrayExcept.enabled` | Enable Comet acceleration for `ArrayExcept` | true | +| `spark.comet.expression.ArrayExists.enabled` | Enable Comet acceleration for `ArrayExists` | true | +| `spark.comet.expression.ArrayFilter.enabled` | Enable Comet acceleration for `ArrayFilter` | true | +| `spark.comet.expression.ArrayForAll.enabled` | Enable Comet acceleration for `ArrayForAll` | true | +| `spark.comet.expression.ArrayInsert.enabled` | Enable Comet acceleration for `ArrayInsert` | true | +| `spark.comet.expression.ArrayIntersect.enabled` | Enable Comet acceleration for `ArrayIntersect` | true | +| `spark.comet.expression.ArrayJoin.enabled` | Enable Comet acceleration for `ArrayJoin` | true | +| `spark.comet.expression.ArrayMax.enabled` | Enable Comet acceleration for `ArrayMax` | true | +| `spark.comet.expression.ArrayMin.enabled` | Enable Comet acceleration for `ArrayMin` | true | +| `spark.comet.expression.ArrayPosition.enabled` | Enable Comet acceleration for `ArrayPosition` | true | +| `spark.comet.expression.ArrayRemove.enabled` | Enable Comet acceleration for `ArrayRemove` | true | +| `spark.comet.expression.ArrayRepeat.enabled` | Enable Comet acceleration for `ArrayRepeat` | true | +| `spark.comet.expression.ArraySort.enabled` | Enable Comet acceleration for `ArraySort` | true | +| `spark.comet.expression.ArrayTransform.enabled` | Enable Comet acceleration for `ArrayTransform` | true | +| `spark.comet.expression.ArrayUnion.enabled` | Enable Comet acceleration for `ArrayUnion` | true | +| `spark.comet.expression.ArraysOverlap.enabled` | Enable Comet acceleration for `ArraysOverlap` | true | +| `spark.comet.expression.ArraysZip.enabled` | Enable Comet acceleration for `ArraysZip` | true | +| `spark.comet.expression.Ascii.enabled` | Enable Comet acceleration for `Ascii` | true | +| `spark.comet.expression.Asin.enabled` | Enable Comet acceleration for `Asin` | true | +| `spark.comet.expression.Asinh.enabled` | Enable Comet acceleration for `Asinh` | true | +| `spark.comet.expression.AtLeastNNonNulls.enabled` | Enable Comet acceleration for `AtLeastNNonNulls` | true | +| `spark.comet.expression.Atan.enabled` | Enable Comet acceleration for `Atan` | true | +| `spark.comet.expression.Atan2.enabled` | Enable Comet acceleration for `Atan2` | true | +| `spark.comet.expression.Atanh.enabled` | Enable Comet acceleration for `Atanh` | true | +| `spark.comet.expression.AttributeReference.enabled` | Enable Comet acceleration for `AttributeReference` | true | +| `spark.comet.expression.BRound.enabled` | Enable Comet acceleration for `BRound` | true | +| `spark.comet.expression.Base64.enabled` | Enable Comet acceleration for `Base64` | true | +| `spark.comet.expression.Bin.enabled` | Enable Comet acceleration for `Bin` | true | +| `spark.comet.expression.BitLength.enabled` | Enable Comet acceleration for `BitLength` | true | +| `spark.comet.expression.BitwiseAnd.enabled` | Enable Comet acceleration for `BitwiseAnd` | true | +| `spark.comet.expression.BitwiseCount.enabled` | Enable Comet acceleration for `BitwiseCount` | true | +| `spark.comet.expression.BitwiseGet.enabled` | Enable Comet acceleration for `BitwiseGet` | true | +| `spark.comet.expression.BitwiseNot.enabled` | Enable Comet acceleration for `BitwiseNot` | true | +| `spark.comet.expression.BitwiseOr.enabled` | Enable Comet acceleration for `BitwiseOr` | true | +| `spark.comet.expression.BitwiseXor.enabled` | Enable Comet acceleration for `BitwiseXor` | true | +| `spark.comet.expression.BloomFilterMightContain.enabled` | Enable Comet acceleration for `BloomFilterMightContain` | true | +| `spark.comet.expression.CaseWhen.enabled` | Enable Comet acceleration for `CaseWhen` | true | +| `spark.comet.expression.Cast.enabled` | Enable Comet acceleration for `Cast` | true | +| `spark.comet.expression.Cbrt.enabled` | Enable Comet acceleration for `Cbrt` | true | +| `spark.comet.expression.Ceil.enabled` | Enable Comet acceleration for `Ceil` | true | +| `spark.comet.expression.CheckOverflow.enabled` | Enable Comet acceleration for `CheckOverflow` | true | +| `spark.comet.expression.Chr.enabled` | Enable Comet acceleration for `Chr` | true | +| `spark.comet.expression.Coalesce.enabled` | Enable Comet acceleration for `Coalesce` | true | +| `spark.comet.expression.Concat.enabled` | Enable Comet acceleration for `Concat` | true | +| `spark.comet.expression.ConcatWs.enabled` | Enable Comet acceleration for `ConcatWs` | true | +| `spark.comet.expression.Contains.enabled` | Enable Comet acceleration for `Contains` | true | +| `spark.comet.expression.Conv.enabled` | Enable Comet acceleration for `Conv` | true | +| `spark.comet.expression.ConvertTimezone.enabled` | Enable Comet acceleration for `ConvertTimezone` | true | +| `spark.comet.expression.Cos.enabled` | Enable Comet acceleration for `Cos` | true | +| `spark.comet.expression.Cosh.enabled` | Enable Comet acceleration for `Cosh` | true | +| `spark.comet.expression.Cot.enabled` | Enable Comet acceleration for `Cot` | true | +| `spark.comet.expression.Crc32.enabled` | Enable Comet acceleration for `Crc32` | true | +| `spark.comet.expression.CreateArray.enabled` | Enable Comet acceleration for `CreateArray` | true | +| `spark.comet.expression.CreateMap.enabled` | Enable Comet acceleration for `CreateMap` | true | +| `spark.comet.expression.CreateNamedStruct.enabled` | Enable Comet acceleration for `CreateNamedStruct` | true | +| `spark.comet.expression.Csc.enabled` | Enable Comet acceleration for `Csc` | true | +| `spark.comet.expression.CsvToStructs.enabled` | Enable Comet acceleration for `CsvToStructs` | true | +| `spark.comet.expression.DateAdd.enabled` | Enable Comet acceleration for `DateAdd` | true | +| `spark.comet.expression.DateDiff.enabled` | Enable Comet acceleration for `DateDiff` | true | +| `spark.comet.expression.DateFormatClass.enabled` | Enable Comet acceleration for `DateFormatClass` | true | +| `spark.comet.expression.DateFromUnixDate.enabled` | Enable Comet acceleration for `DateFromUnixDate` | true | +| `spark.comet.expression.DateSub.enabled` | Enable Comet acceleration for `DateSub` | true | +| `spark.comet.expression.DayOfMonth.enabled` | Enable Comet acceleration for `DayOfMonth` | true | +| `spark.comet.expression.DayOfWeek.enabled` | Enable Comet acceleration for `DayOfWeek` | true | +| `spark.comet.expression.DayOfYear.enabled` | Enable Comet acceleration for `DayOfYear` | true | +| `spark.comet.expression.Days.enabled` | Enable Comet acceleration for `Days` | true | +| `spark.comet.expression.Divide.enabled` | Enable Comet acceleration for `Divide` | true | +| `spark.comet.expression.DivideDTInterval.enabled` | Enable Comet acceleration for `DivideDTInterval` | true | +| `spark.comet.expression.ElementAt.enabled` | Enable Comet acceleration for `ElementAt` | true | +| `spark.comet.expression.Elt.enabled` | Enable Comet acceleration for `Elt` | true | +| `spark.comet.expression.Empty2Null.enabled` | Enable Comet acceleration for `Empty2Null` | true | +| `spark.comet.expression.EndsWith.enabled` | Enable Comet acceleration for `EndsWith` | true | +| `spark.comet.expression.EqualNullSafe.enabled` | Enable Comet acceleration for `EqualNullSafe` | true | +| `spark.comet.expression.EqualTo.enabled` | Enable Comet acceleration for `EqualTo` | true | +| `spark.comet.expression.Exp.enabled` | Enable Comet acceleration for `Exp` | true | +| `spark.comet.expression.Expm1.enabled` | Enable Comet acceleration for `Expm1` | true | +| `spark.comet.expression.Factorial.enabled` | Enable Comet acceleration for `Factorial` | true | +| `spark.comet.expression.FindInSet.enabled` | Enable Comet acceleration for `FindInSet` | true | +| `spark.comet.expression.Flatten.enabled` | Enable Comet acceleration for `Flatten` | true | +| `spark.comet.expression.Floor.enabled` | Enable Comet acceleration for `Floor` | true | +| `spark.comet.expression.FormatNumber.enabled` | Enable Comet acceleration for `FormatNumber` | true | +| `spark.comet.expression.FormatString.enabled` | Enable Comet acceleration for `FormatString` | true | +| `spark.comet.expression.FromUTCTimestamp.enabled` | Enable Comet acceleration for `FromUTCTimestamp` | true | +| `spark.comet.expression.FromUnixTime.enabled` | Enable Comet acceleration for `FromUnixTime` | true | +| `spark.comet.expression.GetArrayItem.enabled` | Enable Comet acceleration for `GetArrayItem` | true | +| `spark.comet.expression.GetArrayStructFields.enabled` | Enable Comet acceleration for `GetArrayStructFields` | true | +| `spark.comet.expression.GetJsonObject.enabled` | Enable Comet acceleration for `GetJsonObject` | true | +| `spark.comet.expression.GetMapValue.enabled` | Enable Comet acceleration for `GetMapValue` | true | +| `spark.comet.expression.GetStructField.enabled` | Enable Comet acceleration for `GetStructField` | true | +| `spark.comet.expression.GetTimestamp.enabled` | Enable Comet acceleration for `GetTimestamp` | true | +| `spark.comet.expression.GreaterThan.enabled` | Enable Comet acceleration for `GreaterThan` | true | +| `spark.comet.expression.GreaterThanOrEqual.enabled` | Enable Comet acceleration for `GreaterThanOrEqual` | true | +| `spark.comet.expression.Greatest.enabled` | Enable Comet acceleration for `Greatest` | true | +| `spark.comet.expression.Hex.enabled` | Enable Comet acceleration for `Hex` | true | +| `spark.comet.expression.HllSketchEstimate.enabled` | Enable Comet acceleration for `HllSketchEstimate` | true | +| `spark.comet.expression.HllUnion.enabled` | Enable Comet acceleration for `HllUnion` | true | +| `spark.comet.expression.Hour.enabled` | Enable Comet acceleration for `Hour` | true | +| `spark.comet.expression.Hours.enabled` | Enable Comet acceleration for `Hours` | true | +| `spark.comet.expression.Hypot.enabled` | Enable Comet acceleration for `Hypot` | true | +| `spark.comet.expression.If.enabled` | Enable Comet acceleration for `If` | true | +| `spark.comet.expression.In.enabled` | Enable Comet acceleration for `In` | true | +| `spark.comet.expression.InSet.enabled` | Enable Comet acceleration for `InSet` | true | +| `spark.comet.expression.InitCap.enabled` | Enable Comet acceleration for `InitCap` | true | +| `spark.comet.expression.IntegralDivide.enabled` | Enable Comet acceleration for `IntegralDivide` | true | +| `spark.comet.expression.Invoke.enabled` | Enable Comet acceleration for `Invoke` | true | +| `spark.comet.expression.IsNaN.enabled` | Enable Comet acceleration for `IsNaN` | true | +| `spark.comet.expression.IsNotNull.enabled` | Enable Comet acceleration for `IsNotNull` | true | +| `spark.comet.expression.IsNull.enabled` | Enable Comet acceleration for `IsNull` | true | +| `spark.comet.expression.JsonObjectKeys.enabled` | Enable Comet acceleration for `JsonObjectKeys` | true | +| `spark.comet.expression.JsonToStructs.enabled` | Enable Comet acceleration for `JsonToStructs` | true | +| `spark.comet.expression.KnownFloatingPointNormalized.enabled` | Enable Comet acceleration for `KnownFloatingPointNormalized` | true | +| `spark.comet.expression.KnownNotNull.enabled` | Enable Comet acceleration for `KnownNotNull` | true | +| `spark.comet.expression.KnownNullable.enabled` | Enable Comet acceleration for `KnownNullable` | true | +| `spark.comet.expression.LastDay.enabled` | Enable Comet acceleration for `LastDay` | true | +| `spark.comet.expression.Least.enabled` | Enable Comet acceleration for `Least` | true | +| `spark.comet.expression.Left.enabled` | Enable Comet acceleration for `Left` | true | +| `spark.comet.expression.Length.enabled` | Enable Comet acceleration for `Length` | true | +| `spark.comet.expression.LengthOfJsonArray.enabled` | Enable Comet acceleration for `LengthOfJsonArray` | true | +| `spark.comet.expression.LessThan.enabled` | Enable Comet acceleration for `LessThan` | true | +| `spark.comet.expression.LessThanOrEqual.enabled` | Enable Comet acceleration for `LessThanOrEqual` | true | +| `spark.comet.expression.Levenshtein.enabled` | Enable Comet acceleration for `Levenshtein` | true | +| `spark.comet.expression.Like.enabled` | Enable Comet acceleration for `Like` | true | +| `spark.comet.expression.Literal.enabled` | Enable Comet acceleration for `Literal` | true | +| `spark.comet.expression.Log.enabled` | Enable Comet acceleration for `Log` | true | +| `spark.comet.expression.Log10.enabled` | Enable Comet acceleration for `Log10` | true | +| `spark.comet.expression.Log1p.enabled` | Enable Comet acceleration for `Log1p` | true | +| `spark.comet.expression.Log2.enabled` | Enable Comet acceleration for `Log2` | true | +| `spark.comet.expression.Logarithm.enabled` | Enable Comet acceleration for `Logarithm` | true | +| `spark.comet.expression.Lower.enabled` | Enable Comet acceleration for `Lower` | true | +| `spark.comet.expression.MakeDTInterval.enabled` | Enable Comet acceleration for `MakeDTInterval` | true | +| `spark.comet.expression.MakeDate.enabled` | Enable Comet acceleration for `MakeDate` | true | +| `spark.comet.expression.MakeDecimal.enabled` | Enable Comet acceleration for `MakeDecimal` | true | +| `spark.comet.expression.MakeInterval.enabled` | Enable Comet acceleration for `MakeInterval` | true | +| `spark.comet.expression.MakeTimestamp.enabled` | Enable Comet acceleration for `MakeTimestamp` | true | +| `spark.comet.expression.MakeYMInterval.enabled` | Enable Comet acceleration for `MakeYMInterval` | true | +| `spark.comet.expression.MapConcat.enabled` | Enable Comet acceleration for `MapConcat` | true | +| `spark.comet.expression.MapEntries.enabled` | Enable Comet acceleration for `MapEntries` | true | +| `spark.comet.expression.MapFilter.enabled` | Enable Comet acceleration for `MapFilter` | true | +| `spark.comet.expression.MapFromArrays.enabled` | Enable Comet acceleration for `MapFromArrays` | true | +| `spark.comet.expression.MapFromEntries.enabled` | Enable Comet acceleration for `MapFromEntries` | true | +| `spark.comet.expression.MapKeys.enabled` | Enable Comet acceleration for `MapKeys` | true | +| `spark.comet.expression.MapSort.enabled` | Enable Comet acceleration for `MapSort` | true | +| `spark.comet.expression.MapValues.enabled` | Enable Comet acceleration for `MapValues` | true | +| `spark.comet.expression.MapZipWith.enabled` | Enable Comet acceleration for `MapZipWith` | true | +| `spark.comet.expression.Mask.enabled` | Enable Comet acceleration for `Mask` | true | +| `spark.comet.expression.Md5.enabled` | Enable Comet acceleration for `Md5` | true | +| `spark.comet.expression.MicrosToTimestamp.enabled` | Enable Comet acceleration for `MicrosToTimestamp` | true | +| `spark.comet.expression.MillisToTimestamp.enabled` | Enable Comet acceleration for `MillisToTimestamp` | true | +| `spark.comet.expression.Minute.enabled` | Enable Comet acceleration for `Minute` | true | +| `spark.comet.expression.MonotonicallyIncreasingID.enabled` | Enable Comet acceleration for `MonotonicallyIncreasingID` | true | +| `spark.comet.expression.Month.enabled` | Enable Comet acceleration for `Month` | true | +| `spark.comet.expression.MonthsBetween.enabled` | Enable Comet acceleration for `MonthsBetween` | true | +| `spark.comet.expression.Multiply.enabled` | Enable Comet acceleration for `Multiply` | true | +| `spark.comet.expression.MultiplyDTInterval.enabled` | Enable Comet acceleration for `MultiplyDTInterval` | true | +| `spark.comet.expression.MultiplyYMInterval.enabled` | Enable Comet acceleration for `MultiplyYMInterval` | true | +| `spark.comet.expression.Murmur3Hash.enabled` | Enable Comet acceleration for `Murmur3Hash` | true | +| `spark.comet.expression.NaNvl.enabled` | Enable Comet acceleration for `NaNvl` | true | +| `spark.comet.expression.NativeUdfCall.enabled` | Enable Comet acceleration for `NativeUdfCall` | true | +| `spark.comet.expression.NextDay.enabled` | Enable Comet acceleration for `NextDay` | true | +| `spark.comet.expression.Not.enabled` | Enable Comet acceleration for `Not` | true | +| `spark.comet.expression.OctetLength.enabled` | Enable Comet acceleration for `OctetLength` | true | +| `spark.comet.expression.Or.enabled` | Enable Comet acceleration for `Or` | true | +| `spark.comet.expression.Overlay.enabled` | Enable Comet acceleration for `Overlay` | true | +| `spark.comet.expression.ParseUrl.enabled` | Enable Comet acceleration for `ParseUrl` | true | +| `spark.comet.expression.Pi.enabled` | Enable Comet acceleration for `Pi` | true | +| `spark.comet.expression.Pmod.enabled` | Enable Comet acceleration for `Pmod` | true | +| `spark.comet.expression.Pow.enabled` | Enable Comet acceleration for `Pow` | true | +| `spark.comet.expression.PreciseTimestampConversion.enabled` | Enable Comet acceleration for `PreciseTimestampConversion` | true | +| `spark.comet.expression.Quarter.enabled` | Enable Comet acceleration for `Quarter` | true | +| `spark.comet.expression.RLike.enabled` | Enable Comet acceleration for `RLike` | true | +| `spark.comet.expression.Rand.enabled` | Enable Comet acceleration for `Rand` | true | +| `spark.comet.expression.RandStr.enabled` | Enable Comet acceleration for `RandStr` | true | +| `spark.comet.expression.Randn.enabled` | Enable Comet acceleration for `Randn` | true | +| `spark.comet.expression.RegExpExtract.enabled` | Enable Comet acceleration for `RegExpExtract` | true | +| `spark.comet.expression.RegExpExtractAll.enabled` | Enable Comet acceleration for `RegExpExtractAll` | true | +| `spark.comet.expression.RegExpInStr.enabled` | Enable Comet acceleration for `RegExpInStr` | true | +| `spark.comet.expression.RegExpReplace.enabled` | Enable Comet acceleration for `RegExpReplace` | true | +| `spark.comet.expression.Remainder.enabled` | Enable Comet acceleration for `Remainder` | true | +| `spark.comet.expression.Reverse.enabled` | Enable Comet acceleration for `Reverse` | true | +| `spark.comet.expression.Right.enabled` | Enable Comet acceleration for `Right` | true | +| `spark.comet.expression.Rint.enabled` | Enable Comet acceleration for `Rint` | true | +| `spark.comet.expression.Round.enabled` | Enable Comet acceleration for `Round` | true | +| `spark.comet.expression.ScalaUDF.enabled` | Enable Comet acceleration for `ScalaUDF` | true | +| `spark.comet.expression.ScalarSubquery.enabled` | Enable Comet acceleration for `ScalarSubquery` | true | +| `spark.comet.expression.SchemaOfCsv.enabled` | Enable Comet acceleration for `SchemaOfCsv` | true | +| `spark.comet.expression.SchemaOfJson.enabled` | Enable Comet acceleration for `SchemaOfJson` | true | +| `spark.comet.expression.Sec.enabled` | Enable Comet acceleration for `Sec` | true | +| `spark.comet.expression.Second.enabled` | Enable Comet acceleration for `Second` | true | +| `spark.comet.expression.SecondsToTimestamp.enabled` | Enable Comet acceleration for `SecondsToTimestamp` | true | +| `spark.comet.expression.Sequence.enabled` | Enable Comet acceleration for `Sequence` | true | +| `spark.comet.expression.Sha1.enabled` | Enable Comet acceleration for `Sha1` | true | +| `spark.comet.expression.Sha2.enabled` | Enable Comet acceleration for `Sha2` | true | +| `spark.comet.expression.ShiftLeft.enabled` | Enable Comet acceleration for `ShiftLeft` | true | +| `spark.comet.expression.ShiftRight.enabled` | Enable Comet acceleration for `ShiftRight` | true | +| `spark.comet.expression.ShiftRightUnsigned.enabled` | Enable Comet acceleration for `ShiftRightUnsigned` | true | +| `spark.comet.expression.Shuffle.enabled` | Enable Comet acceleration for `Shuffle` | true | +| `spark.comet.expression.Signum.enabled` | Enable Comet acceleration for `Signum` | true | +| `spark.comet.expression.Sin.enabled` | Enable Comet acceleration for `Sin` | true | +| `spark.comet.expression.Sinh.enabled` | Enable Comet acceleration for `Sinh` | true | +| `spark.comet.expression.Size.enabled` | Enable Comet acceleration for `Size` | true | +| `spark.comet.expression.Slice.enabled` | Enable Comet acceleration for `Slice` | true | +| `spark.comet.expression.SortArray.enabled` | Enable Comet acceleration for `SortArray` | true | +| `spark.comet.expression.SortOrder.enabled` | Enable Comet acceleration for `SortOrder` | true | +| `spark.comet.expression.SoundEx.enabled` | Enable Comet acceleration for `SoundEx` | true | +| `spark.comet.expression.SparkPartitionID.enabled` | Enable Comet acceleration for `SparkPartitionID` | true | +| `spark.comet.expression.Sqrt.enabled` | Enable Comet acceleration for `Sqrt` | true | +| `spark.comet.expression.StartsWith.enabled` | Enable Comet acceleration for `StartsWith` | true | +| `spark.comet.expression.StaticInvoke.enabled` | Enable Comet acceleration for `StaticInvoke` | true | +| `spark.comet.expression.StringInstr.enabled` | Enable Comet acceleration for `StringInstr` | true | +| `spark.comet.expression.StringLPad.enabled` | Enable Comet acceleration for `StringLPad` | true | +| `spark.comet.expression.StringLocate.enabled` | Enable Comet acceleration for `StringLocate` | true | +| `spark.comet.expression.StringRPad.enabled` | Enable Comet acceleration for `StringRPad` | true | +| `spark.comet.expression.StringRepeat.enabled` | Enable Comet acceleration for `StringRepeat` | true | +| `spark.comet.expression.StringReplace.enabled` | Enable Comet acceleration for `StringReplace` | true | +| `spark.comet.expression.StringSpace.enabled` | Enable Comet acceleration for `StringSpace` | true | +| `spark.comet.expression.StringSplit.enabled` | Enable Comet acceleration for `StringSplit` | true | +| `spark.comet.expression.StringToMap.enabled` | Enable Comet acceleration for `StringToMap` | true | +| `spark.comet.expression.StringTranslate.enabled` | Enable Comet acceleration for `StringTranslate` | true | +| `spark.comet.expression.StringTrim.enabled` | Enable Comet acceleration for `StringTrim` | true | +| `spark.comet.expression.StringTrimLeft.enabled` | Enable Comet acceleration for `StringTrimLeft` | true | +| `spark.comet.expression.StringTrimRight.enabled` | Enable Comet acceleration for `StringTrimRight` | true | +| `spark.comet.expression.StructsToCsv.enabled` | Enable Comet acceleration for `StructsToCsv` | true | +| `spark.comet.expression.StructsToJson.enabled` | Enable Comet acceleration for `StructsToJson` | true | +| `spark.comet.expression.Substring.enabled` | Enable Comet acceleration for `Substring` | true | +| `spark.comet.expression.SubstringIndex.enabled` | Enable Comet acceleration for `SubstringIndex` | true | +| `spark.comet.expression.Subtract.enabled` | Enable Comet acceleration for `Subtract` | true | +| `spark.comet.expression.Tan.enabled` | Enable Comet acceleration for `Tan` | true | +| `spark.comet.expression.Tanh.enabled` | Enable Comet acceleration for `Tanh` | true | +| `spark.comet.expression.TimestampAdd.enabled` | Enable Comet acceleration for `TimestampAdd` | true | +| `spark.comet.expression.TimestampDiff.enabled` | Enable Comet acceleration for `TimestampDiff` | true | +| `spark.comet.expression.ToCharacter.enabled` | Enable Comet acceleration for `ToCharacter` | true | +| `spark.comet.expression.ToDegrees.enabled` | Enable Comet acceleration for `ToDegrees` | true | +| `spark.comet.expression.ToNumber.enabled` | Enable Comet acceleration for `ToNumber` | true | +| `spark.comet.expression.ToPrettyString.enabled` | Enable Comet acceleration for `ToPrettyString` | true | +| `spark.comet.expression.ToRadians.enabled` | Enable Comet acceleration for `ToRadians` | true | +| `spark.comet.expression.ToUTCTimestamp.enabled` | Enable Comet acceleration for `ToUTCTimestamp` | true | +| `spark.comet.expression.ToUnixTimestamp.enabled` | Enable Comet acceleration for `ToUnixTimestamp` | true | +| `spark.comet.expression.TransformKeys.enabled` | Enable Comet acceleration for `TransformKeys` | true | +| `spark.comet.expression.TransformValues.enabled` | Enable Comet acceleration for `TransformValues` | true | +| `spark.comet.expression.TruncDate.enabled` | Enable Comet acceleration for `TruncDate` | true | +| `spark.comet.expression.TruncTimestamp.enabled` | Enable Comet acceleration for `TruncTimestamp` | true | +| `spark.comet.expression.TryEval.enabled` | Enable Comet acceleration for `TryEval` | true | +| `spark.comet.expression.TryToNumber.enabled` | Enable Comet acceleration for `TryToNumber` | true | +| `spark.comet.expression.UnBase64.enabled` | Enable Comet acceleration for `UnBase64` | true | +| `spark.comet.expression.UnaryMinus.enabled` | Enable Comet acceleration for `UnaryMinus` | true | +| `spark.comet.expression.UnaryPositive.enabled` | Enable Comet acceleration for `UnaryPositive` | true | +| `spark.comet.expression.Unhex.enabled` | Enable Comet acceleration for `Unhex` | true | +| `spark.comet.expression.UnixDate.enabled` | Enable Comet acceleration for `UnixDate` | true | +| `spark.comet.expression.UnixMicros.enabled` | Enable Comet acceleration for `UnixMicros` | true | +| `spark.comet.expression.UnixMillis.enabled` | Enable Comet acceleration for `UnixMillis` | true | +| `spark.comet.expression.UnixSeconds.enabled` | Enable Comet acceleration for `UnixSeconds` | true | +| `spark.comet.expression.UnixTimestamp.enabled` | Enable Comet acceleration for `UnixTimestamp` | true | +| `spark.comet.expression.UnscaledValue.enabled` | Enable Comet acceleration for `UnscaledValue` | true | +| `spark.comet.expression.Upper.enabled` | Enable Comet acceleration for `Upper` | true | +| `spark.comet.expression.Uuid.enabled` | Enable Comet acceleration for `Uuid` | true | +| `spark.comet.expression.WeekDay.enabled` | Enable Comet acceleration for `WeekDay` | true | +| `spark.comet.expression.WeekOfYear.enabled` | Enable Comet acceleration for `WeekOfYear` | true | +| `spark.comet.expression.WidthBucket.enabled` | Enable Comet acceleration for `WidthBucket` | true | +| `spark.comet.expression.XPathBoolean.enabled` | Enable Comet acceleration for `XPathBoolean` | true | +| `spark.comet.expression.XPathDouble.enabled` | Enable Comet acceleration for `XPathDouble` | true | +| `spark.comet.expression.XPathFloat.enabled` | Enable Comet acceleration for `XPathFloat` | true | +| `spark.comet.expression.XPathInt.enabled` | Enable Comet acceleration for `XPathInt` | true | +| `spark.comet.expression.XPathList.enabled` | Enable Comet acceleration for `XPathList` | true | +| `spark.comet.expression.XPathLong.enabled` | Enable Comet acceleration for `XPathLong` | true | +| `spark.comet.expression.XPathShort.enabled` | Enable Comet acceleration for `XPathShort` | true | +| `spark.comet.expression.XPathString.enabled` | Enable Comet acceleration for `XPathString` | true | +| `spark.comet.expression.XxHash64.enabled` | Enable Comet acceleration for `XxHash64` | true | +| `spark.comet.expression.Year.enabled` | Enable Comet acceleration for `Year` | true | +| `spark.comet.expression.ZipWith.enabled` | Enable Comet acceleration for `ZipWith` | true | + ## Enabling or Disabling Individual Aggregate Expressions + +| Config | Description | Default Value | +|--------|-------------|---------------| +| `spark.comet.expression.ApproximatePercentile.enabled` | Enable Comet acceleration for `ApproximatePercentile` | true | +| `spark.comet.expression.Average.enabled` | Enable Comet acceleration for `Average` | true | +| `spark.comet.expression.BitAndAgg.enabled` | Enable Comet acceleration for `BitAndAgg` | true | +| `spark.comet.expression.BitOrAgg.enabled` | Enable Comet acceleration for `BitOrAgg` | true | +| `spark.comet.expression.BitXorAgg.enabled` | Enable Comet acceleration for `BitXorAgg` | true | +| `spark.comet.expression.BloomFilterAggregate.enabled` | Enable Comet acceleration for `BloomFilterAggregate` | true | +| `spark.comet.expression.CollectList.enabled` | Enable Comet acceleration for `CollectList` | true | +| `spark.comet.expression.CollectSet.enabled` | Enable Comet acceleration for `CollectSet` | true | +| `spark.comet.expression.Corr.enabled` | Enable Comet acceleration for `Corr` | true | +| `spark.comet.expression.Count.enabled` | Enable Comet acceleration for `Count` | true | +| `spark.comet.expression.CovPopulation.enabled` | Enable Comet acceleration for `CovPopulation` | true | +| `spark.comet.expression.CovSample.enabled` | Enable Comet acceleration for `CovSample` | true | +| `spark.comet.expression.First.enabled` | Enable Comet acceleration for `First` | true | +| `spark.comet.expression.HllSketchAgg.enabled` | Enable Comet acceleration for `HllSketchAgg` | true | +| `spark.comet.expression.HllUnionAgg.enabled` | Enable Comet acceleration for `HllUnionAgg` | true | +| `spark.comet.expression.HyperLogLogPlusPlus.enabled` | Enable Comet acceleration for `HyperLogLogPlusPlus` | true | +| `spark.comet.expression.Last.enabled` | Enable Comet acceleration for `Last` | true | +| `spark.comet.expression.ListAgg.enabled` | Enable Comet acceleration for `ListAgg` | true | +| `spark.comet.expression.Max.enabled` | Enable Comet acceleration for `Max` | true | +| `spark.comet.expression.MaxBy.enabled` | Enable Comet acceleration for `MaxBy` | true | +| `spark.comet.expression.Min.enabled` | Enable Comet acceleration for `Min` | true | +| `spark.comet.expression.MinBy.enabled` | Enable Comet acceleration for `MinBy` | true | +| `spark.comet.expression.Mode.enabled` | Enable Comet acceleration for `Mode` | true | +| `spark.comet.expression.Percentile.enabled` | Enable Comet acceleration for `Percentile` | true | +| `spark.comet.expression.PivotFirst.enabled` | Enable Comet acceleration for `PivotFirst` | true | +| `spark.comet.expression.RegrIntercept.enabled` | Enable Comet acceleration for `RegrIntercept` | true | +| `spark.comet.expression.RegrR2.enabled` | Enable Comet acceleration for `RegrR2` | true | +| `spark.comet.expression.RegrReplacement.enabled` | Enable Comet acceleration for `RegrReplacement` | true | +| `spark.comet.expression.RegrSXY.enabled` | Enable Comet acceleration for `RegrSXY` | true | +| `spark.comet.expression.RegrSlope.enabled` | Enable Comet acceleration for `RegrSlope` | true | +| `spark.comet.expression.StddevPop.enabled` | Enable Comet acceleration for `StddevPop` | true | +| `spark.comet.expression.StddevSamp.enabled` | Enable Comet acceleration for `StddevSamp` | true | +| `spark.comet.expression.Sum.enabled` | Enable Comet acceleration for `Sum` | true | +| `spark.comet.expression.VariancePop.enabled` | Enable Comet acceleration for `VariancePop` | true | +| `spark.comet.expression.VarianceSamp.enabled` | Enable Comet acceleration for `VarianceSamp` | true | + diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md index f9292b27343..d4f9be61770 100644 --- a/docs/source/user-guide/latest/expressions.md +++ b/docs/source/user-guide/latest/expressions.md @@ -118,21 +118,21 @@ The tables below list every Spark built-in expression with its current status. | `median` | ✅ | — | Rewrites to `percentile(col, 0.5)` and runs natively for supported percentile inputs | | `min` | ✅ | Native | | | `min_by` | ✅ | Native | Value and ordering must be fixed-length types | -| `mode` | ✅ | Native | `mode(col)` only; Spark breaks ties non-deterministically, so Comet returns the smallest tied value and falls back by default, opt-in via allowIncompatible | +| `mode` | ✅ | — | `mode(col)` only; Spark breaks ties non-deterministically, so Comet returns the smallest tied value and falls back by default, opt-in via allowIncompatible | | `percentile` | ✅ | Native | Single literal percentage on numeric input runs natively; array of percentages and a frequency argument fall back to Spark | | `percentile_approx` | ✅ | Native | Alias of `approx_percentile`; same restrictions apply | | `percentile_cont` | ✅ | — | Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`; ascending only runs natively, `DESC` falls back to Spark | | `percentile_disc` | 🔜 | — | Percentile aggregate | -| `pivot_first` | ✅ | Native | Internal aggregate for the optimized `PIVOT` fast path. Value type must be in `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal); other types keep Spark's standard filtered-aggregate path. Binary, array and struct pivot columns (including pivots on more than one column), non-default collations and pivot lists with duplicate values fall back to Spark | +| `pivot_first` | ✅ | — | Internal aggregate for the optimized `PIVOT` fast path. Value type must be in `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal); other types keep Spark's standard filtered-aggregate path. Binary, array and struct pivot columns (including pivots on more than one column), non-default collations and pivot lists with duplicate values fall back to Spark | | `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) | | `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) | | `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) | | `regr_intercept` | ✅ | Native | | | `regr_r2` | ✅ | Native | | | `regr_slope` | ✅ | Native | | -| `regr_sxx` | ✅ | Native | | +| `regr_sxx` | ✅ | — | | | `regr_sxy` | ✅ | Native | | -| `regr_syy` | ✅ | Native | | +| `regr_syy` | ✅ | — | | | `skewness` | 🔜 | — | Not yet implemented natively | | `some` | ✅ | — | | | `std` | ✅ | Native | | @@ -186,9 +186,9 @@ The tables below list every Spark built-in expression with its current status. | Function | Status | Implementation | Notes | | --- | --- | --- | --- | | `&` | ✅ | Native | | -| `<<` | ✅ | — | | -| `>>` | ✅ | — | | -| `>>>` | ✅ | — | Operator alias for `shiftrightunsigned` (Spark 4.0+) | +| `<<` | ✅ | Native | | +| `>>` | ✅ | Native | | +| `>>>` | ✅ | Native | Operator alias for `shiftrightunsigned` (Spark 4.0+) | | `^` | ✅ | Native | | | `bit_count` | ✅ | Native | | | `bit_get` | ✅ | Native | | @@ -235,7 +235,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci | Function | Status | Implementation | Notes | | --- | --- | --- | --- | -| `cast` | ✅ | Native | Casts without a native path (for example boolean to decimal) route through the JVM codegen dispatcher ([details](compatibility/expressions/cast.md)) | +| `cast` | ✅ | Hybrid | Casts without a native path (for example boolean to decimal) route through the JVM codegen dispatcher ([details](compatibility/expressions/cast.md)) | --- @@ -278,28 +278,28 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci | `extract` | ✅ | — | `SECOND FROM TIME` is native for Spark 4.1+ with `Decimal(8,6)` output | | `from_unixtime` | ✅ | Hybrid | | | `from_utc_timestamp` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | -| `hour` | ✅ | Native | | +| `hour` | ✅ | — | | | `last_day` | ✅ | Native | | | `localtimestamp` | ✅ | — | | | `make_date` | ✅ | Native | | | `make_dt_interval` | ✅ | Codegen dispatch | | | `make_interval` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; intervals outside Arrow's nanosecond range are tracked by [#5279](https://github.com/apache/datafusion-comet/issues/5279); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | | `make_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. Runs natively; remaining TIME type work is tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) | -| `make_timestamp` | ✅ | Hybrid | | +| `make_timestamp` | ✅ | — | | | `make_timestamp_ltz` | ✅ | — | The 2-argument `(date, time)` form (Spark 4.1+) runs through codegen dispatch | | `make_timestamp_ntz` | ✅ | — | The 2-argument `(date, time)` form (Spark 4.1+) runs through codegen dispatch | | `make_ym_interval` | ✅ | Codegen dispatch | | -| `minute` | ✅ | Native | | +| `minute` | ✅ | — | | | `month` | ✅ | Native | | | `monthname` | ✅ | — | Abbreviated month name (Spark 4.0+) | | `months_between` | ✅ | Codegen dispatch | | | `next_day` | ✅ | Hybrid | Non-UTF8_BINARY collated `dayOfWeek` routes through the JVM codegen dispatcher; other input runs natively | | `now` | ✅ | — | Constant-folded to a literal (alias of `current_timestamp`) | | `quarter` | ✅ | Native | | -| `second` | ✅ | Native | | +| `second` | ✅ | — | | | `session_window` | 🔜 | — | Batch session-window grouping falls back (`UpdatingSessionsExec` is not yet native); tracked by [#4785](https://github.com/apache/datafusion-comet/issues/4785) | -| `time_diff` | ✅ | Codegen dispatch | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default | -| `time_trunc` | ✅ | Codegen dispatch | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default | +| `time_diff` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default | +| `time_trunc` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default | | `timediff` | ✅ | — | Spark 4.0+ grammar alias that parses to `timestampdiff`; runs through codegen dispatch | | `timestamp_micros` | ✅ | Codegen dispatch | | | `timestamp_millis` | ✅ | Codegen dispatch | | @@ -311,7 +311,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci | `to_timestamp` | ✅ | — | Rewrites to `Cast` (or `GetTimestamp` with a format) before Comet sees the plan | | `to_timestamp_ltz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampType`) | | `to_timestamp_ntz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampNTZType`) | -| `to_unix_timestamp` | ✅ | Hybrid | | +| `to_unix_timestamp` | ✅ | Codegen dispatch | | | `to_utc_timestamp` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | | `trunc` | ✅ | Hybrid | | | `try_make_interval` | ✅ | — | Rewrites to `MakeInterval`; same support as `make_interval` (Spark 4.0+) | @@ -618,15 +618,15 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `substring` | ✅ | Native | | | `substring_index` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively | | `to_binary` | ✅ | — | The hex form runs natively; the base64 and `utf-8` forms route through the JVM codegen dispatcher | -| `to_char` | ✅ | Codegen dispatch | | +| `to_char` | ✅ | — | | | `to_number` | ✅ | Codegen dispatch | | -| `to_varchar` | ✅ | Codegen dispatch | | +| `to_varchar` | ✅ | — | | | `translate` | ✅ | Hybrid | Codegen dispatch by default: DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible | | `trim` | ✅ | Hybrid | Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs natively | | `try_to_binary` | ✅ | — | Rewrites to `try_eval(to_binary(...))`, which routes through the JVM codegen dispatcher | | `try_to_number` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher | | `ucase` | ✅ | Hybrid | | -| `unbase64` | ✅ | Codegen dispatch | Column or literal input runs natively; other child expressions, and the strict form used by `to_binary(str, 'base64')`, route through the JVM codegen dispatcher | +| `unbase64` | ✅ | Hybrid | Column or literal input runs natively; other child expressions, and the strict form used by `to_binary(str, 'base64')`, route through the JVM codegen dispatcher | | `upper` | ✅ | Hybrid | | ---