Skip to content

[SPARK-62058][CONNECT][PYTHON] Log pipeline events at the exact times of their timestamps, as PySpark's ToDatetime, and pin the rounding of pre-1970 nanoseconds of createDataFrame and literals - #874

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62058-pipeline-event-timestamps
Closed

HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62058-pipeline-event-timestamps

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This closes the deferred follow-up "pre-1970 timestamps with nanoseconds". The investigation compared pyspark-client 4.3.0 with the drop-in end to end, against a Spark 4.3.0 server, for times before 1970 with digits below the microsecond. One place differed, the log of the events of a pipeline run, and this PR fixes it. Every conversion of data and literals behaved the same, and a new test pins how each of them rounds.

The fix. PySpark's handle_pipeline_events logs each event at event.timestamp.ToDatetime().replace(tzinfo=timezone.utc) (python/pyspark/pipelines/spark_connect_pipeline.py:64). protobuf's Timestamp.ToDatetime is exact: the epoch plus timedelta(seconds=..., microseconds=nanos // 1000) (google/protobuf/internal/well_known_types.py:239-266).

The drop-in's run stream yields the microseconds of the event's Timestamp. Its handle_pipeline_events converted them with datetime.fromtimestamp(timestamp_micros / 1_000_000, tz=timezone.utc), through a float of the seconds. Far from 1970 the spacing of that float exceeds a microsecond:

  • 0001-01-01 00:00:00.999999 was logged as 00:00:01;
  • the last microsecond of 9999 raised ValueError: year 10000 is out of range.

The shim now adds the microseconds to the epoch, datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(microseconds=...). That is PySpark's datetime for every Timestamp of the years 1 to 9999. Times near 1970, before it too, convert as before.

What PySpark does, which the drop-in follows. PySpark drops the digits below the microsecond in two directions, depending on the conversion:

Conversion PySpark 4.3.0's code 1969-12-31 23:59:59.999999999 UTC becomes
local rows of pandas Timestamps, TimestampType and TimestampNTZType PyArrow reads the fields of the datetime, of value.astimezone(utc) for a TimestampType (conversion.py:784) 23:59:59.999999 (down)
a literal of a pandas Timestamp TimestampType().toInternal(value) (connect/expressions.py:303), of the fields 23:59:59.999999 (down)
an object column of pandas Timestamps of a pandas DataFrame PyArrow reads the fields 23:59:59.999999 (down)
a pyarrow timestamp[ns] Table pc.floor_temporal (pandas/types.py:554) 23:59:59.999999 (down)
a datetime64[ns] column of a pandas DataFrame, safe cast off PyArrow's cast (pandas/conversion.py:117) 1970-01-01 00:00:00 (toward 1970)
the same, safe cast on PySparkValueError of the unsafe conversion

A NumPy datetime64[ns] array makes a TIMESTAMP_NTZ(9) column on the server, whose conversion raises UNSUPPORTED_OPERATION in both clients. A NumPy scalar raises UNSUPPORTED_DATA_TYPE as a literal, and the AssertionError of the converter in local rows, in both clients.

A new offline test pins each of these, with two times before 1970. They round as PySpark's already on master, so the test guards them rather than fixing anything.

Docs. The entry of dev/parity/known-differences.md on pandas frames without PyArrow now says how the native row path drops nanoseconds. Like local rows, it rounds them down to the microsecond, where PySpark's unsafe cast of a datetime64[ns] column rounds a time before 1970 toward 1970. PySpark Connect has no session without PyArrow, so this extends the drop-in rather than differing from a setup that PySpark supports.

Why are the changes needed?

To log the events of a pipeline run at the times that PySpark logs them, without a failure for a time that PySpark converts. The test keeps the rounding of pre-1970 nanoseconds of each conversion of createDataFrame and of literals as PySpark's.

Does this PR introduce any user-facing change?

Yes, for the log of a pipeline run whose server sends event times far from 1970: an event at 0001-01-01 00:00:00.999999 is logged at 00:00:00, as in PySpark, instead of 00:00:01, and an event at 9999-12-31 23:59:59.999999 is logged instead of failing with ValueError. A server sends its own clock's times, so the events of a real run are logged as before.

How was this patch tested?

  • New offline tests in python/tests/test_dropin_offline.py:

    • test_pipeline_events_are_logged_at_the_times_of_their_timestamps_as_pysparks: six events, of the years 1, 1033, 1969, 1970, 2025 and 9999, logged as PySpark's ToDatetime of the same Timestamps logs them. It fails on master with ValueError: year 10000 is out of range.
    • test_pre_1970_nanoseconds_round_as_each_conversion_of_pyspark: the microseconds that each conversion in the table above sends to the mock server, of PySpark 4.3.0's values.
  • The official pipeline tests through the drop-in, which pipelines-parity.yml runs (test_cli.py, test_init_cli.py, test_add_pipeline_analysis_context.py, test_block_session_mutations.py, test_spark_connect.py, test_auto_cdc_flow.py), against a Spark 4.3.0 server: 73 passed, with 53 subtests, on master and with this change.

  • Probes of both clients against the same server, compared line by line:

    • every case was identical, 6,067 lines in all, but for the module of the class of the Timestamp of the runtime, pyspark.sql.connect.proto._rust.timestamp_pb2;
    • session time zones: UTC, America/Los_Angeles and Asia/Kolkata;
    • host time zones: UTC, America/Los_Angeles and America/New_York.

    The cases:

    • Local rows. Pandas Timestamps, naive and of zones, datetimes and NumPy scalars, as TimestampType, TimestampNTZType and inferred, and nested in arrays, structs and maps.
    • pandas DataFrames. datetime64[ns] columns, naive and of zones; object columns; the us, ms and s units; frames of more than 1,000 rows, which the fast paths take. With the safe cast on and off.
    • NumPy arrays of one and two dimensions.
    • pyarrow Tables of timestamp[ns], naive, UTC and Los Angeles.
    • Literals, of pandas Timestamps, NumPy scalars, datetimes and lists.
    • Results. collect, toLocalIterator, toPandas and toArrow of these, and of times that the server computes: before 1970, of the years 1, 1000, 1500 and 1677, and at the edges of pandas' nanosecond range. Nested too. Also the nanosecond types, cast down by the server, and the metrics observed by a name, through executionInfo.
    • The Timestamp of the Rust protobuf runtime. Its constructors from the units, from datetimes and from pandas Timestamps; its JSON text; its conversions and arithmetic; its fields as a server could set them; a Timestamp field of a message.
  • The gates, on master 80046f2f:

    • cargo fmt --all --check, cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings, RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core, and cargo test -p pyspark-rs --lib as rust.yml runs it: 15 passed (this PR changes no Rust code);
    • ruff 0.14.0 check and format --check, test_ruff_excludes.py, scripts/audit_no_stubs.sh, gen_type_stubs.py --check, gen_fn_dispatch.py --check and the guards of the test lists;
    • test_type_stubs_offline.py: 13 passed, 3 skipped;
    • the offline suite of coverage.yml: 6496 passed, 77 skipped (pyspark/pipelines/* is outside the Python coverage of .coveragerc);
    • test_dropin_e2e.py against a Spark 4.3.0 server: 333 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

… of their timestamps, as PySpark's ToDatetime, and pin the rounding of pre-1970 nanoseconds of createDataFrame and literals

- handle_pipeline_events of pyspark.pipelines.spark_connect_pipeline converts the microseconds
  of an event's Timestamp exactly, epoch + timedelta, as PySpark's
  event.timestamp.ToDatetime().replace(tzinfo=timezone.utc), instead of datetime.fromtimestamp
  of a float of its seconds, which rounded an event far from 1970 into the next second
  (0001-01-01 00:00:00.999999 logged 00:00:01) and failed with "year 10000 is out of range" for
  the last microsecond of 9999.
- A test pins the rounding of times before 1970 with nanoseconds in each conversion of
  createDataFrame and of a literal, as PySpark 4.3.0's: local rows of pandas Timestamps, an
  object column of them, a pyarrow Table and a literal round down to the microsecond, and a
  datetime64[ns] column of a pandas DataFrame rounds toward 1970 when the safe cast is off and
  raises when it is on.
- known-differences.md says how the row path of pandas frames without PyArrow drops
  nanoseconds: to the microsecond below, as local rows do, where PySpark's unsafe cast of a
  datetime64[ns] column rounds a time before 1970 toward 1970.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants