[nat-openai]

Which issue does this PR close?

This addresses the major incompatibility described in apache/datafusion-comet#4654 for CometFromUTCTimestamp. It does not close that issue.

It's a follow-up to #19879.

Rationale for this change

The timezone spellings accepted by Spark are different than those accepted by Arrow's parser. As a result, from_utc_timestamp in comet isn't safe to use, since it would fail on things like Java short IDs, prefixed offsets with second precision, and SystemV-style IDs. Additionally, Arrow will accept offsets beyond the Spark limit of 18 hours.

What changes are included in this PR?

This change improves compatibility by parsing timezone IDs with the rules implemented by Spark. As part of doing this, I added SparkFromUtcTimestampExpr so that computed timezones evaluated for rows with non-null timestamps, avoiding failing when a timezone is invalid but wouldn't be required anyway. (Full compatibility with spark behaviour seems quite complex and I haven't attempted it here.)

It also adds more documentation recording remaining differences:

• timezone validation and evaluation order can differ from Spark
• chrono accepts a narrower calendar range than Spark implemented range
• chrono-tz doesn't currently perform DST transitions past 2099

This does not achieve full compatibility with Spark due to those issues, but it makes the region of incompatibility a lot smaller.

If this approach isn't the one that folks would prefer, please let me know if there are different avenues you'd rather I explore!

Finally, it's worth mentioning that this change was LLM-assisted.

What is the testing strategy for this PR?

Added unit tests in datafusion/spark/src/function/datetime/timezone.rs covering accepted spelling, the 18 hour boundary, and SystemV custom behaviour around DST transitions and exception.

Added SQL logic tests in datafusion/sqllogictest/test_files/spark/datetime/from_utc_timestamp.slt covering the accepted spellings.

Are there any user-facing changes?

Yes. The existing Spark from_utc_timestamp UDF now accepts additional Spark timezone spellings. It also rejects offsets outside Spark's 18-hour limit.

This won't affect most users since they'd need to have opted into the existing incompatible implementation.

[nat-openai]

@sunchao, if this seems reasonable, can you enable the CI workflows?
Also cc @andygrove, since this is tangentially relevant to apache/datafusion-comet#4654.

[ajsquared]

Reviewed the current changes and feedback; no independently confirmed unresolved P1 remains.

[sunchao]

@nat-openai just enabled the CI run

[sunchao]

LGTM as a partial Spark compatibility improvement. Reviewed 7594f111a214f619c148b98df5b0bc367f75c845 against merge base 3310fd59a743dad59329779451278518ac32d90a; no blocking correctness defect found. The inline suggestions are nonblocking. This does not establish full Spark compatibility.

Summary

• Prior state and problem: The existing UDF delegated timezone parsing to Arrow, rejecting valid Spark spellings such as PST, GMT+1, second-resolution offsets, and SystemV identifiers, while accepting offsets beyond Spark's ±18-hour limit. Base/head component checks confirmed that PST and GMT+1 change from errors to successful conversions.

• Design approach: The private SparkTimeZone enum separates fixed offsets, regional timezones, and Java's legacy SystemV rules. Regional conversion continues using Arrow/Chrono. The separate SparkFromUtcTimestampExpr controls conditional evaluation of timezone expressions, while both entry points share the conversion kernel.

• Correctness: Offset normalization, signs, seconds, ±18-hour boundaries, Java short-ID mappings, and SystemV transitions checked out. Focused checks passed for nulls, scalar/array combinations, sliced validity buffers, empty batches, timestamp precision, and timezone metadata. The shared helper's checked addition turns calendar-overflow panics into errors. No introduced P1/P2 found.

• Compatibility analysis: Important limits remain. The new evaluator deliberately differs from Spark's generated execution: an invalid literal timezone can be skipped when every timestamp is null, and a null timezone does not suppress timestamp evaluation. Chrono's calendar range is narrower. Regional DST rules also remain incorrect for some future dates: in July 2100, this implementation applies UTC−5 to New York while Spark applies UTC−4. The base UDF reproduces the same future-DST limitation. This PR alone therefore does not justify removing Comet's incompatibility guard.

• Key design decisions: Preserving offset seconds is necessary. Mapping EST, HST, and MST to fixed offsets while mapping other short IDs to regions matches Java. Keeping SystemV rules separate avoids substituting modern regional DST rules. The one-entry timezone cache avoids reparsing constants/consecutive repeated values without persistent state.

• Implementation sketch: Evaluate timestamps; skip timezone evaluation for empty/all-null inputs; read literal/column timezones directly or evaluate computed timezones only on selected rows; parse/cache each needed timezone; shift the timestamp while preserving its type and metadata. The UDF shares the conversion steps but receives already-evaluated arguments.

• Performance: The direct literal/column fast path and parsing cache are sensible. For computed timezone expressions with mixed-null timestamps, the default evaluate_selection filters every input column, including unrelated columns; an isolated helper probe confirmed scaling with batch width. The benchmark currently exercises only direct timezone columns. Scalar timezone strings also still expand into full arrays through make_scalar_function: the cache removes repeated parsing but not that allocation. Scalar expansion is pre-existing overhead, not a new regression. Add representative benchmarks before introducing broader caching or specialized SystemV optimizations.

• Design: The approach is solid and reuses the regional database and selection machinery. The important integration boundary is that ordinary SQL still uses the eager UDF; consumers must explicitly construct the new physical expression to obtain conditional evaluation. A runtime-table probe reproduced the difference using a computed timezone with an invalid cast on null timestamp rows. A planner/consumer example would clarify the API's intended use.

• Abstraction & complexity: The three timezone variants represent distinct policies and are appropriately scoped. A custom physical expression is justified because an ordinary UDF cannot control evaluation of arguments it has already received. The small last-value cache is proportionate; a generic cache framework would need workload evidence. The bespoke SystemV rules create maintenance responsibility, but their limited scope and transition coverage make that manageable.

• Behavioral changes worth calling out: More Spark timezone spellings succeed; offsets outside ±18 hours and ROC are rejected; calendar-boundary adjustment returns an error instead of panicking. Conditional evaluation is opt-in through the physical expression. to_utc_timestamp retains its existing parser, so the reverse function does not gain equivalent spelling support here.

• Suggested improvements: Add computed/scalar timezone benchmark cases, including wide batches and sparse nulls. Add a consumer/planner example and integration tests when wiring in the physical expression. As a minor cleanup, have fmt_sql call the children's SQL formatters rather than their Display implementations, which can emit column indices such as timestamp@0.

Validation

• cargo test --locked -p datafusion-spark --lib function::datetime -- --nocapture: 52 passed, including all 15 tests in the changed from_utc_timestamp and new timezone modules.
• cargo test --locked -p datafusion-functions --lib datetime::to_local_time -- --nocapture: 4 passed.
• Independent SQL/physical component probes using unchanged PR source and its compiled dependencies passed. The base-UDF control reused the head's shared helper and dependencies; it was not a full base build.
• 30 actual Spark 4.1.3 SQL probes checked generated and interpreted execution with UTC session timezone and ANSI enabled. Source comparisons also covered Spark 3.5.8, 4.0.4, and 4.2.0; those versions were not runtime-validated.
• Exact extracted Rust offset-parser functions matched Java with Spark normalization on 258,744 cases. Extracted calendar/helpers passed 6,611,852 checks against JDK 21/tzdb 2026b. These are helper-level differential checks, not full-engine tests.
• Full local SLT/workspace tests and release benchmarks were not run. The selection-cost probe used debug dependencies on a shared machine and is not a release or base/head performance benchmark.
• CI snapshot immediately before submission: 36 successful, 5 running, and 3 skipped checks. Approval reflects the reviewed code and focused validation; full CI completion remains outstanding.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 90.30144% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.86%. Comparing base (c52eb97) to head (ff7dd6a).
⚠️ Report is 93 commits behind head on main.

Files with missing lines	Patch %	Lines
.../spark/src/function/datetime/from_utc_timestamp.rs	88.39%	30 Missing and 20 partials ⚠️
datafusion/spark/src/function/datetime/timezone.rs	92.83%	9 Missing and 14 partials ⚠️
datafusion/functions/src/datetime/to_local_time.rs	90.90%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25979      +/-   ##
==========================================
+ Coverage   82.85%   82.86%   +0.01%     
==========================================
  Files        1151     1152       +1     
  Lines      455996   456751     +755     
  Branches   455996   456751     +755     
==========================================
+ Hits       377805   378497     +692     
- Misses      55239    55266      +27     
- Partials    22952    22988      +36     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ajsquared]

Reviewed the current changes and feedback; no independently confirmed unresolved P1 remains.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-functions v55.1.0 (current)
       Built [  39.976s] (current)
     Parsing datafusion-functions v55.1.0 (current)
      Parsed [   0.107s] (current)
    Building datafusion-functions v55.1.0 (baseline)
       Built [  38.888s] (baseline)
     Parsing datafusion-functions v55.1.0 (baseline)
      Parsed [   0.109s] (baseline)
    Checking datafusion-functions v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.425s] 229 checks: 226 pass, 3 fail, 0 warn, 31 skip

--- failure function_missing: pub fn removed or renamed ---

Description:
A publicly-visible function cannot be imported by its prior path. A `pub use` may have been removed, or the function itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/function_missing.ron

Failed in:
  function datafusion_functions::datetime::timezone, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/48c1df5ecb5bf547ee86cf270c64eb5c559919d0/datafusion/functions/src/datetime/mod.rs:51

--- failure module_missing: pub module removed or renamed ---

Description:
A publicly-visible module cannot be imported by its prior path. A `pub use` may have been removed, or the module may have been renamed, removed, or made non-public.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/module_missing.ron

Failed in:
  mod datafusion_functions::datetime::timezone, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/48c1df5ecb5bf547ee86cf270c64eb5c559919d0/datafusion/functions/src/datetime/timezone.rs:18

--- failure struct_missing: pub struct removed or renamed ---

Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/struct_missing.ron

Failed in:
  struct datafusion_functions::datetime::timezone::TimezoneFunc, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/48c1df5ecb5bf547ee86cf270c64eb5c559919d0/datafusion/functions/src/datetime/timezone.rs:87

     Summary semver requires new major version: 3 major and 0 minor checks failed
    Finished [  81.092s] datafusion-functions
    Building datafusion-spark v55.1.0 (current)
       Built [  51.559s] (current)
     Parsing datafusion-spark v55.1.0 (current)
      Parsed [   0.075s] (current)
    Building datafusion-spark v55.1.0 (baseline)
       Built [  51.291s] (baseline)
     Parsing datafusion-spark v55.1.0 (baseline)
      Parsed [   0.076s] (baseline)
    Checking datafusion-spark v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.390s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [ 104.608s] datafusion-spark
    Building datafusion-sqllogictest v55.1.0 (current)
       Built [ 106.211s] (current)
     Parsing datafusion-sqllogictest v55.1.0 (current)
      Parsed [   0.016s] (current)
    Building datafusion-sqllogictest v55.1.0 (baseline)
       Built [ 108.538s] (baseline)
     Parsing datafusion-sqllogictest v55.1.0 (baseline)
      Parsed [   0.017s] (baseline)
    Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.109s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [ 217.781s] datafusion-sqllogictest

[alamb]

I merged up to resolve a conflict as I am trying to clear our PR merge queue

[ajsquared]

[P2] Changing the public timezone parameter from Tz to impl TimeZone + Copy breaks downstream calls that rely on type inference. For example, adjust_to_local_time::<TimestampMicrosecondType>(0, "UTC".parse().unwrap()) previously inferred Tz. The new signature gives parse() no concrete target type, so the same call no longer compiles. Keep the public Tz signature and delegate to a separately named generic helper that the Spark fixed-offset path can also call.

[ajsquared]

[P2] Changing the public timezone parameter from Tz to impl TimeZone + Copy breaks downstream calls that rely on type inference. For example, adjust_to_local_time::<TimestampMicrosecondType>(0, "UTC".parse().unwrap()) previously inferred Tz. The new signature gives parse() no concrete target type, so the same call no longer compiles. Keep the public Tz signature and delegate to a separately named generic helper that the Spark fixed-offset path can also call.

[ajsquared]

Reviewed the current changes and feedback; no independently confirmed unresolved P1 remains.