Skip to content

[SPARK-62047][CONNECT][PYTHON] Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark - #871

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62047-any-pickles-user-context-extensions
Closed

HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62047-any-pickles-user-context-extensions

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Two gaps of the drop-in against PySpark 4.3.0:

  1. The messages of the well-known types of the drop-in's protobuf runtime, Any and Timestamp, did not pickle: their classes are of the runtime's modules pyspark.sql.connect.proto._rust.any_pb2 and timestamp_pb2, which were not in sys.modules, so pickle.dumps raised Can't pickle <class '...any_pb2.Any'>: import of module ... failed, and cloudpickle cannot pickle 'classmethod_descriptor' object. install now puts them in sys.modules, so their pickles load, in another process of the drop-in too, and cloudpickle pickles them by reference.
  2. A user context extension that is not an Any (another message, such as pb2.Relation(), or an object that is not a message) was sent as the bytes of another message parsed as an Any, or raised AttributeError when it was added. PySpark's lists take any object, and each request of an execution, an analysis, a configuration or an interrupt raises upb's TypeError when it is built, of the global extensions first. The drop-in's lists now take it too, and the serialization of the extensions before each call that sends requests (which already re-serializes them, for an Any changed in place) adds each to a UserContext of the runtime, which raises upb's error (or that of the unpacking of an item that is not a pair) before the request is sent, until the extension is removed. The calls whose requests carry no extensions, as PySpark's requests of the artifacts, CloneSession, ReleaseSession and GetStatus carry none (addArtifacts, copyFromLocalToFs, cloneSession, stop, client.release_session and client._get_operation_statuses), no longer serialize them before they are made (interruptible_without_user_context), so they are made, as in PySpark.

Why are the changes needed?

To close these differences from PySpark (the known-differences review).

Does this PR introduce any user-facing change?

Yes: the messages of Any and Timestamp pickle, and a user context extension that is not an Any fails each call as in PySpark rather than being sent as an Any of other bytes. docs/compatibility.md and dev/parity/known-differences.md say that the pickle of the drop-in's Any names the runtime's module, which PySpark does not have, and that the TypeError names the runtime's class of Any.

How was this patch tested?

New tests in python/tests/test_proto_rust_offline.py (pickles of the well-known types, also in a new process and without protobuf) and python/tests/test_dropin_offline.py (extensions that are not an Any, against the mock server: no request is sent, the other thread's requests are, removal; the calls whose requests carry no extensions are made), which fail without the change. Probes of the extensions against a Spark 4.3.0 server give the outputs of PySpark 4.3.0 (upb) line for line, but for the class of Any that the error names (and the grpc error of copyFromLocalToFs that PySpark leaves unconverted, as before).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

This pull request and its description were written by Isaac.

@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62047-any-pickles-user-context-extensions branch from fcb0d2b to c6f9f72 Compare November 19, 2026 00:13
…of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark

Two gaps of the drop-in against PySpark 4.3.0:

1. The messages of the well-known types of the drop-in's protobuf runtime, `Any` and `Timestamp`, did not pickle: their classes are of the runtime's modules `pyspark.sql.connect.proto._rust.any_pb2` and `timestamp_pb2`, which were not in `sys.modules`, so `pickle.dumps` raised `Can't pickle <class '...any_pb2.Any'>: import of module ... failed`, and cloudpickle `cannot pickle 'classmethod_descriptor' object`. `install` now puts them in `sys.modules`, so their pickles load, in another process of the drop-in too, and cloudpickle pickles them by reference.
2. A user context extension that is not an `Any` (another message, such as `pb2.Relation()`, or an object that is not a message) was sent as the bytes of another message parsed as an `Any`, or raised `AttributeError` when it was added. PySpark's lists take any object, and each request of an execution, an analysis, a configuration or an interrupt raises upb's `TypeError` when it is built, of the global extensions first. The drop-in's lists now take it too, and the serialization of the extensions before each call that sends requests (which already re-serializes them, for an `Any` changed in place) adds each to a `UserContext` of the runtime, which raises upb's error (or that of the unpacking of an item that is not a pair) before the request is sent, until the extension is removed. The calls whose requests carry no extensions, as PySpark's requests of the artifacts, `CloneSession`, `ReleaseSession` and `GetStatus` carry none (`addArtifacts`, `copyFromLocalToFs`, `cloneSession`, `stop`, `client.release_session` and `client._get_operation_statuses`), no longer serialize them before they are made (`interruptible_without_user_context`), so they are made, as in PySpark.

To close these differences from PySpark (the known-differences review).

Yes: the messages of `Any` and `Timestamp` pickle, and a user context extension that is not an `Any` fails each call as in PySpark rather than being sent as an `Any` of other bytes. `docs/compatibility.md` and `dev/parity/known-differences.md` say that the pickle of the drop-in's `Any` names the runtime's module, which PySpark does not have, and that the `TypeError` names the runtime's class of `Any`.

New tests in `python/tests/test_proto_rust_offline.py` (pickles of the well-known types, also in a new process and without `protobuf`) and `python/tests/test_dropin_offline.py` (extensions that are not an `Any`, against the mock server: no request is sent, the other thread's requests are, removal; the calls whose requests carry no extensions are made), which fail without the change. Probes of the extensions against a Spark 4.3.0 server give the outputs of PySpark 4.3.0 (upb) line for line, but for the class of `Any` that the error names (and the grpc error of `copyFromLocalToFs` that PySpark leaves unconverted, as before).

Generated-by: Claude Code

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62047-any-pickles-user-context-extensions branch from c6f9f72 to 74db554 Compare November 19, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants