[ump45nose]

Which issue does this PR close?

Closes #7210.

Rationale for this change

For years outside chrono's NaiveDate range, the native string-to-timestamp path looked up the session timezone offset at 1970-01-01. That returns the zone's standard offset (for example -08:00 for America/Los_Angeles) instead of the local-mean-time offset Spark's ZoneRules use for dates before 1883. Casting -290308-12-21 19:59:05.224192 in an America/Los_Angeles session was therefore 422 seconds off from Spark (-9223372008054775808 vs -9223372008476775808).

1800-06-01 in the same session already matched Spark, because that year is still inside chrono's range and chrono-tz applies LMT there. Only the extreme-year surrogate probe was wrong.

What changes are included in this PR?

In parse_timestamp_to_micros, the extreme-year branch now looks up the offset at the same civil time shifted toward the epoch by whole 400-year Gregorian cycles until the year fits in chrono's range. Leap days are preserved across the shift. Fixed offsets and UTC are unchanged. Far-future DST after chrono-tz's last transition (~2099) is unchanged and remains #6816.

How are these changes tested?

New unit test extreme_year_session_tz_uses_historic_offset in native/spark-expr/src/conversion_funcs/string.rs:

• America/Los_Angeles session cast of -290308-12-21 19:59:05.224192 matches Spark's LMT micros.
• 1800-06-01 00:00:00 in the same zone stays pinned to the known Spark value.
• UTC and +02:00 stay exact at the i64 microsecond bounds.

cargo test -p datafusion-comet-spark-expr extreme_year, cargo test -p datafusion-comet-spark-expr conversion_funcs::string::tests, cargo clippy -p datafusion-comet-spark-expr --all-targets -- -D warnings, and cargo fmt pass.

This PR was prepared with AI assistance and reviewed and tested by the author.

[comphead]

Probing the zone offset at the same civil time shifted by whole 400-year cycles is a sound way to reach the local-mean-time era, and the LMT arithmetic in the description checks out. The Spark path (LocalDateTime.of, ZonedDateTime.of, instantToMicros) is the same in 3.5.9, 4.0.4 and 4.1.3. One comment is about a possible panic. Three comments.

[comphead]

The loop bounds look one year wider than chrono accepts. In chrono 0.4.45, which Cargo.lock pins, MIN_YEAR is (i32::MIN >> 13) + 1 = -262143 and MAX_YEAR is (i32::MAX >> 13) - 1 = 262142, and NaiveDate::MAX is documented as December 31, 262142 CE. A year of 262543 is shifted once to 262143 and the loop stops there, and -262544 is shifted once to -262144 and stops there. from_ymd_opt then returns None and the .expect panics. Both years pass the -290309..=294248 gate and fit in i64 microseconds, so as far as I can tell cast('262543-01-01' as timestamp) returns a value in Spark and on main (the old probe used 1970-01-01), but would panic here, under UTC and fixed-offset sessions too, since this branch runs for every zone. Could the bounds be -262143 and 262142, or could the two loops become arithmetic with rem_euclid(400) and a base year inside the range, which also removes up to about 80 iterations per value? Please add 262543-01-01, -262544-01-01 and their neighbours to the unit test. The (-262144..=262143) check at line 1436 has the same off-by-one and is outside this diff, so years 262143 and -262144 return Ok(None) as invalid dates. I derived the panic by reading and did not run it.

[ump45nose]

Fixed. The probe now uses 1600 + year.rem_euclid(400) so it always lands in 1600..=1999 inside chrono 0.4.45's [-262143, 262142] range (and preserves leap days across the Gregorian cycle), instead of the while-loop that could stop on 262143 / -262144 and panic in from_ymd_opt.expect. The in-range invalid-date check is aligned to the same MIN/MAX. Added extreme_year_chrono_bound_neighbours_do_not_panic covering 262543-01-01, -262544-01-01, and their neighbours.

(AI-assisted reply.)

[comphead]

I could not construct an input that reaches the LocalResult::None or Ambiguous arms for this probe, since it lands either before every transition of the zone or after the last chrono-tz timespan, and neither arm has a test. They also repeat the three-way match from the main path above, so a small helper such as offset_secs_at(tz, naive) -> Option<i64> could serve both. In the gap arm, probe_naive - chrono::Duration::hours(3) panics if the subtraction overflows NaiveDateTime, and the final LocalResult::None => 0 silently treats the zone as UTC. Both are narrow and unverified, so this is a question about whether the arms are needed.

[ump45nose]

Kept the arms but extracted offset_secs_at(tz, naive) -> Option<i64> and routed both the extreme-year probe and the main-path gap arm through it. The 3h subtract is now checked_sub_signed so an extreme NaiveDateTime cannot panic; a second None still falls back to UTC (0) on the extreme path (same as before) and to Ok(None) on the in-range gap path. The arms are hard to hit for this probe for most zones, but a congruent civil time can still land in a spring-forward gap, so the helper keeps that path covered without duplicating the three-way match.

(AI-assisted reply.)

[comphead]

The expected values in extreme_year_session_tz_uses_historic_offset are asserted from the author's reading of Spark and are not compared against it. CometDateTimeUtilsSuite.checkCastToTimestampAsLong already casts to LONG to avoid the toJavaTimestamp limits, and the SQL harness supports -- ConfigMatrix: spark.sql.session.timeZone=.... Could one case there use America/Los_Angeles and a positive-offset zone such as Asia/Tokyo with -290308-12-21 19:59:05.224192 and -262544-01-01 00:00:00, where the second also exercises the point above? A 422 second difference survives the truncation in cast(ts as bigint). The Rust test also repeats the UTC i64::MAX and i64::MIN assertions of extreme_year_boundary_test, and the 1800-06-01 case is outside the extreme branch. I did not verify which session zone the Scala suite runs under.

[ump45nose]

Added extreme-year session TZ uses historic zone offset in CometDateTimeUtilsSuite, which runs checkCastToTimestampAsLong under America/Los_Angeles and Asia/Tokyo for -290308-12-21 19:59:05.224192, -262544-01-01 00:00:00, and chrono-bound neighbours — so expected values come from Spark, and the 422s LMT gap survives the Long cast. Trimmed the Rust session-TZ unit test to drop the UTC i64::MAX/MIN and 1800 cases that duplicated extreme_year_boundary_test / the in-range path; the LA micros pin and the new chrono-bound neighbour unit test remain.

(AI-assisted reply.)

[sunchao]

Reviewed 25408fd6448f7e1a4692a045554d970a99f8a622 across five independent scopes. The revised modulo probe introduces a wrong-offset regression for future January dates and still chooses the wrong era for some far-past years. Details inline.

Validation: compiled a focused probe using the pinned chrono 0.4.45 and chrono-tz 0.10.4, and compared its offsets with Java LocalDateTime.atZone, which Spark uses. Compared the old epoch probe on the release branch as a control. This is a component-level reproduction; I did not run the full Spark/Comet integration suite. Suite registration checks passed. Current-head CI is incomplete (Detect changes is queued).

[sunchao]

Could we choose the offset era separately for far-past and far-future dates? 1600 + year.rem_euclid(400) preserves the calendar but not timezone rules. For 262543-01-01 00:00:00 it probes 1743: the pinned chrono-tz returns Los Angeles offset -07:52:58 while Java/Spark uses -08:00, making the result 422 seconds early. Tokyo similarly returns +09:18:59 instead of +09:00, a 1,139-second error. The previous 1970 probe was correct for both January cases, so this is a new regression, separate from the known future-DST horizon issue #6816. In the other direction, -262500-01-01 probes 1900 and still misses the ancient LMT offset. I reproduced these offsets with the exact Rust dependencies and Java. Please cover both future January and negative-year residues that land after timezone transitions, and run the new Spark comparison test after correcting the probe. A native-cast assertion or expect_native(cast) SQL fixture would also ensure the regression test exercises this Rust path.