[SteNicholas]

Purpose

Linked issue: close #476

Paimon Java accepts TIMESTAMP_LTZ (TIMESTAMP WITH LOCAL TIME ZONE) columns in Lance data files
since apache/paimon#10558. Paimon C++ rejected every Arrow timestamp with a time zone in
SchemaValidation::ValidateLanceDataField() with Lance file format does not support type LOCAL_ZONED_TIMESTAMP, whether Lance was selected by file.format, file-format-per-level or
changelog-file-format.

Removing that check alone is not enough, because the two sides put different time zones on the
Arrow type:

• Java always writes LTZ as Arrow Timestamp(unit, "UTC") (ArrowFieldTypeConversion), and its
  validator relies on that: Lance 0.39 stores the type as timestamp:<unit>:<zone> and cannot
  parse a zone id containing : such as GMT-10:00.
• C++ builds the LTZ Arrow type with DateTimeUtils::GetLocalTimezoneName(), a host-dependent
  zone id that may contain :.

This change:

• Accepts zoned timestamps in ValidateLanceDataField(), including inside ARRAY and ROW, as
  Java's LanceRowTypeVisitor does.
• Makes LanceFormatWriter import each batch with a copy of the writer schema whose zoned
  timestamps, top level and nested, carry the UTC time zone. The values are instants since the
  epoch, so only the Arrow type changes, and the file stores what Java writes whatever the local
  time zone is.
• Relabels the UTC timestamps Lance returns with the read schema's local zone in
  LanceFileBatchReader::AlignBatch(), before AlignArrayToReadType(), which would otherwise
  reject the zone difference. The helper, AlignTimestampTimezones, moves from the Vortex
  reader's anonymous namespace into NestedProjectionUtils and is used by both readers. It takes
  and returns an arrow::Array, matches ROW children by name, and follows ARRAY, fixed-size list
  and MAP keys and values. It only relabels a timestamp when both sides have a zone and the same
  unit.
• Needs no statistics change: LanceStatsExtractor::CreateEmptyStats() already handles
  TIMESTAMP whatever its zone.

Tests

• SchemaValidationTest.TestLanceDataTypes: LTZ at precisions 0, 3, 6 and 9 with UTC,
  Asia/Shanghai, GMT-10:00 and +08:00 zones, in ARRAY and in ROW<ARRAY<...>>, moves from
  the rejected to the accepted fields. An LTZ array is also accepted when Lance is selected
  through file-format-per-level or changelog-file-format.
• NestedProjectionUtilsTest.AlignTimestampTimezonesRelabelsZonedLeavesOfSameUnit: top-level,
  ROW, ARRAY, fixed-size list and MAP key and value leaves are relabelled; identical types, a unit
  change and a timestamp without a zone on either side are left unchanged. It also covers sliced
  input and checks that the input is not modified.
• NestedProjectionUtilsTest.AlignTimestampTimezonesMatchesStructChildrenByName: a ROW child that
  the read type does not name keeps its zone, and the child it names is relabelled although it
  sits at another position, as when the Vortex reader passes a batch it has not pruned.
• LanceFileFormatTest.WriteThenReadSupportedTypes: adds LTZ columns at precisions 0, 3, 6 and 9,
  ARRAY<LTZ> and ROW<LTZ> in the GMT-10:00 zone, reads them back in that zone and checks
  that the file schema stores UTC.
• WriteAndReadInteTest.TestAppendTimestampType and TestPkTimestampType now also run for
  Lance.
• LanceTimestampLtzInteTest.TestWriteAndReadInPinnedTimezone: append and primary-key tables
  under UTC, Asia/Shanghai and Pacific/Honolulu, with nested LTZ, an LTZ column added by
  schema evolution, a reordered projection, and one column changed from TIMESTAMP(6) to
  TIMESTAMP_LTZ(6) and another the other way. Files written before the type change are
  converted in the local time zone, as Java's TimestampToTimestampCastRule does with
  TimeZone.getDefault(); ScanAndReadInteTest.TestCastTimestampType checks the same conversion
  on Java-written Parquet and ORC files.
• ScanAndReadInteTest.TestLanceLtzCompatibility: reads the new lance/append_ltz_java_compat
  fixture (Paimon Java at 535a5c9, JVM time zone GMT-10:00) and
  lance/append_ltz_python_compat fixture (Paimon Python at 63ffd40) under Asia/Shanghai, at
  both snapshots and with a reordered projection. Both fixtures hold the same LTZ(0/3/6/9),
  ARRAY<LTZ(6)> and ROW<number INT, ts LTZ(3)> values, and both writers store every LTZ leaf
  as UTC. The existing append_java_compat and append_python_compat fixtures are unchanged.

API and Format

No API changes in include/. No storage format changes: LTZ is written to Lance as an Arrow
timestamp with the UTC time zone, as in Paimon Java.

Documentation

Yes. docs/source/user_guide/data_types.rst describes how Lance data files store
TIMESTAMP WITH LOCAL TIME ZONE: as an Arrow timestamp with the UTC time zone whatever the
local time zone is, returned in the local time zone on read.

Generative AI tooling

Generated-by: Claude Code 2.1.311 (Claude Opus 5.5)

🤖 Generated with Claude Code

[lxy-9602]

+1