Repository navigation
[SPARK-62058][CONNECT][PYTHON] Log pipeline events at the exact times of their timestamps, as PySpark's ToDatetime, and pin the rounding of pre-1970 nanoseconds of createDataFrame and literals - #874
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
… of their timestamps, as PySpark's ToDatetime, and pin the rounding of pre-1970 nanoseconds of createDataFrame and literals - handle_pipeline_events of pyspark.pipelines.spark_connect_pipeline converts the microseconds of an event's Timestamp exactly, epoch + timedelta, as PySpark's event.timestamp.ToDatetime().replace(tzinfo=timezone.utc), instead of datetime.fromtimestamp of a float of its seconds, which rounded an event far from 1970 into the next second (0001-01-01 00:00:00.999999 logged 00:00:01) and failed with "year 10000 is out of range" for the last microsecond of 9999. - A test pins the rounding of times before 1970 with nanoseconds in each conversion of createDataFrame and of a literal, as PySpark 4.3.0's: local rows of pandas Timestamps, an object column of them, a pyarrow Table and a literal round down to the microsecond, and a datetime64[ns] column of a pandas DataFrame rounds toward 1970 when the safe cast is off and raises when it is on. - known-differences.md says how the row path of pandas frames without PyArrow drops nanoseconds: to the microsecond below, as local rows do, where PySpark's unsafe cast of a datetime64[ns] column rounds a time before 1970 toward 1970. Co-authored-by: Isaac <no-reply@databricks.com>
dongjoon-hyun
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This closes the deferred follow-up "pre-1970 timestamps with nanoseconds". The investigation compared pyspark-client 4.3.0 with the drop-in end to end, against a Spark 4.3.0 server, for times before 1970 with digits below the microsecond. One place differed, the log of the events of a pipeline run, and this PR fixes it. Every conversion of data and literals behaved the same, and a new test pins how each of them rounds.
The fix. PySpark's
handle_pipeline_eventslogs each event atevent.timestamp.ToDatetime().replace(tzinfo=timezone.utc)(python/pyspark/pipelines/spark_connect_pipeline.py:64). protobuf'sTimestamp.ToDatetimeis exact: the epoch plustimedelta(seconds=..., microseconds=nanos // 1000)(google/protobuf/internal/well_known_types.py:239-266).The drop-in's run stream yields the microseconds of the event's Timestamp. Its
handle_pipeline_eventsconverted them withdatetime.fromtimestamp(timestamp_micros / 1_000_000, tz=timezone.utc), through a float of the seconds. Far from 1970 the spacing of that float exceeds a microsecond:0001-01-01 00:00:00.999999was logged as00:00:01;ValueError: year 10000 is out of range.The shim now adds the microseconds to the epoch,
datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(microseconds=...). That is PySpark's datetime for every Timestamp of the years 1 to 9999. Times near 1970, before it too, convert as before.What PySpark does, which the drop-in follows. PySpark drops the digits below the microsecond in two directions, depending on the conversion:
1969-12-31 23:59:59.999999999 UTCbecomesTimestampTypeandTimestampNTZTypevalue.astimezone(utc)for aTimestampType(conversion.py:784)23:59:59.999999(down)TimestampType().toInternal(value)(connect/expressions.py:303), of the fields23:59:59.999999(down)23:59:59.999999(down)timestamp[ns]Tablepc.floor_temporal(pandas/types.py:554)23:59:59.999999(down)datetime64[ns]column of a pandas DataFrame, safe cast offpandas/conversion.py:117)1970-01-01 00:00:00(toward 1970)PySparkValueErrorof the unsafe conversionA NumPy
datetime64[ns]array makes aTIMESTAMP_NTZ(9)column on the server, whose conversion raisesUNSUPPORTED_OPERATIONin both clients. A NumPy scalar raisesUNSUPPORTED_DATA_TYPEas a literal, and theAssertionErrorof the converter in local rows, in both clients.A new offline test pins each of these, with two times before 1970. They round as PySpark's already on master, so the test guards them rather than fixing anything.
Docs. The entry of
dev/parity/known-differences.mdon pandas frames without PyArrow now says how the native row path drops nanoseconds. Like local rows, it rounds them down to the microsecond, where PySpark's unsafe cast of adatetime64[ns]column rounds a time before 1970 toward 1970. PySpark Connect has no session without PyArrow, so this extends the drop-in rather than differing from a setup that PySpark supports.Why are the changes needed?
To log the events of a pipeline run at the times that PySpark logs them, without a failure for a time that PySpark converts. The test keeps the rounding of pre-1970 nanoseconds of each conversion of
createDataFrameand of literals as PySpark's.Does this PR introduce any user-facing change?
Yes, for the log of a pipeline run whose server sends event times far from 1970: an event at
0001-01-01 00:00:00.999999is logged at00:00:00, as in PySpark, instead of00:00:01, and an event at9999-12-31 23:59:59.999999is logged instead of failing withValueError. A server sends its own clock's times, so the events of a real run are logged as before.How was this patch tested?
New offline tests in
python/tests/test_dropin_offline.py:test_pipeline_events_are_logged_at_the_times_of_their_timestamps_as_pysparks: six events, of the years 1, 1033, 1969, 1970, 2025 and 9999, logged as PySpark'sToDatetimeof the same Timestamps logs them. It fails on master withValueError: year 10000 is out of range.test_pre_1970_nanoseconds_round_as_each_conversion_of_pyspark: the microseconds that each conversion in the table above sends to the mock server, of PySpark 4.3.0's values.The official pipeline tests through the drop-in, which
pipelines-parity.ymlruns (test_cli.py,test_init_cli.py,test_add_pipeline_analysis_context.py,test_block_session_mutations.py,test_spark_connect.py,test_auto_cdc_flow.py), against a Spark 4.3.0 server: 73 passed, with 53 subtests, on master and with this change.Probes of both clients against the same server, compared line by line:
pyspark.sql.connect.proto._rust.timestamp_pb2;The cases:
TimestampType,TimestampNTZTypeand inferred, and nested in arrays, structs and maps.datetime64[ns]columns, naive and of zones; object columns; theus,msandsunits; frames of more than 1,000 rows, which the fast paths take. With the safe cast on and off.timestamp[ns], naive, UTC and Los Angeles.collect,toLocalIterator,toPandasandtoArrowof these, and of times that the server computes: before 1970, of the years 1, 1000, 1500 and 1677, and at the edges of pandas' nanosecond range. Nested too. Also the nanosecond types, cast down by the server, and the metrics observed by a name, throughexecutionInfo.The gates, on master
80046f2f:cargo fmt --all --check,cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings,RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core, andcargo test -p pyspark-rs --libasrust.ymlruns it: 15 passed (this PR changes no Rust code);checkandformat --check,test_ruff_excludes.py,scripts/audit_no_stubs.sh,gen_type_stubs.py --check,gen_fn_dispatch.py --checkand the guards of the test lists;test_type_stubs_offline.py: 13 passed, 3 skipped;coverage.yml: 6496 passed, 77 skipped (pyspark/pipelines/*is outside the Python coverage of.coveragerc);test_dropin_e2e.pyagainst a Spark 4.3.0 server: 333 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.