Repository navigation
[SPARK-62047][CONNECT][PYTHON] Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark - #871
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
HyukjinKwon
force-pushed
the
SPARK-62047-any-pickles-user-context-extensions
branch
from
November 19, 2026 00:13
fcb0d2b to
c6f9f72
Compare
…of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark Two gaps of the drop-in against PySpark 4.3.0: 1. The messages of the well-known types of the drop-in's protobuf runtime, `Any` and `Timestamp`, did not pickle: their classes are of the runtime's modules `pyspark.sql.connect.proto._rust.any_pb2` and `timestamp_pb2`, which were not in `sys.modules`, so `pickle.dumps` raised `Can't pickle <class '...any_pb2.Any'>: import of module ... failed`, and cloudpickle `cannot pickle 'classmethod_descriptor' object`. `install` now puts them in `sys.modules`, so their pickles load, in another process of the drop-in too, and cloudpickle pickles them by reference. 2. A user context extension that is not an `Any` (another message, such as `pb2.Relation()`, or an object that is not a message) was sent as the bytes of another message parsed as an `Any`, or raised `AttributeError` when it was added. PySpark's lists take any object, and each request of an execution, an analysis, a configuration or an interrupt raises upb's `TypeError` when it is built, of the global extensions first. The drop-in's lists now take it too, and the serialization of the extensions before each call that sends requests (which already re-serializes them, for an `Any` changed in place) adds each to a `UserContext` of the runtime, which raises upb's error (or that of the unpacking of an item that is not a pair) before the request is sent, until the extension is removed. The calls whose requests carry no extensions, as PySpark's requests of the artifacts, `CloneSession`, `ReleaseSession` and `GetStatus` carry none (`addArtifacts`, `copyFromLocalToFs`, `cloneSession`, `stop`, `client.release_session` and `client._get_operation_statuses`), no longer serialize them before they are made (`interruptible_without_user_context`), so they are made, as in PySpark. To close these differences from PySpark (the known-differences review). Yes: the messages of `Any` and `Timestamp` pickle, and a user context extension that is not an `Any` fails each call as in PySpark rather than being sent as an `Any` of other bytes. `docs/compatibility.md` and `dev/parity/known-differences.md` say that the pickle of the drop-in's `Any` names the runtime's module, which PySpark does not have, and that the `TypeError` names the runtime's class of `Any`. New tests in `python/tests/test_proto_rust_offline.py` (pickles of the well-known types, also in a new process and without `protobuf`) and `python/tests/test_dropin_offline.py` (extensions that are not an `Any`, against the mock server: no request is sent, the other thread's requests are, removal; the calls whose requests carry no extensions are made), which fail without the change. Probes of the extensions against a Spark 4.3.0 server give the outputs of PySpark 4.3.0 (upb) line for line, but for the class of `Any` that the error names (and the grpc error of `copyFromLocalToFs` that PySpark leaves unconverted, as before). Generated-by: Claude Code Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62047-any-pickles-user-context-extensions
branch
from
November 19, 2026 01:05
c6f9f72 to
74db554
Compare
dongjoon-hyun
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Two gaps of the drop-in against PySpark 4.3.0:
AnyandTimestamp, did not pickle: their classes are of the runtime's modulespyspark.sql.connect.proto._rust.any_pb2andtimestamp_pb2, which were not insys.modules, sopickle.dumpsraisedCan't pickle <class '...any_pb2.Any'>: import of module ... failed, and cloudpicklecannot pickle 'classmethod_descriptor' object.installnow puts them insys.modules, so their pickles load, in another process of the drop-in too, and cloudpickle pickles them by reference.Any(another message, such aspb2.Relation(), or an object that is not a message) was sent as the bytes of another message parsed as anAny, or raisedAttributeErrorwhen it was added. PySpark's lists take any object, and each request of an execution, an analysis, a configuration or an interrupt raises upb'sTypeErrorwhen it is built, of the global extensions first. The drop-in's lists now take it too, and the serialization of the extensions before each call that sends requests (which already re-serializes them, for anAnychanged in place) adds each to aUserContextof the runtime, which raises upb's error (or that of the unpacking of an item that is not a pair) before the request is sent, until the extension is removed. The calls whose requests carry no extensions, as PySpark's requests of the artifacts,CloneSession,ReleaseSessionandGetStatuscarry none (addArtifacts,copyFromLocalToFs,cloneSession,stop,client.release_sessionandclient._get_operation_statuses), no longer serialize them before they are made (interruptible_without_user_context), so they are made, as in PySpark.Why are the changes needed?
To close these differences from PySpark (the known-differences review).
Does this PR introduce any user-facing change?
Yes: the messages of
AnyandTimestamppickle, and a user context extension that is not anAnyfails each call as in PySpark rather than being sent as anAnyof other bytes.docs/compatibility.mdanddev/parity/known-differences.mdsay that the pickle of the drop-in'sAnynames the runtime's module, which PySpark does not have, and that theTypeErrornames the runtime's class ofAny.How was this patch tested?
New tests in
python/tests/test_proto_rust_offline.py(pickles of the well-known types, also in a new process and withoutprotobuf) andpython/tests/test_dropin_offline.py(extensions that are not anAny, against the mock server: no request is sent, the other thread's requests are, removal; the calls whose requests carry no extensions are made), which fail without the change. Probes of the extensions against a Spark 4.3.0 server give the outputs of PySpark 4.3.0 (upb) line for line, but for the class ofAnythat the error names (and the grpc error ofcopyFromLocalToFsthat PySpark leaves unconverted, as before).Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
This pull request and its description were written by Isaac.