Repository navigation
[SPARK-62071][CONNECT][PYTHON] Make the classic modules of the objects of a session PySpark's, so that the objects are not instances of the classic classes, as in pyspark-client - #881
Closed
HyukjinKwon wants to merge 1 commit into
Conversation
…s of a session PySpark's, so that the objects are not instances of the classic classes, as in pyspark-client
- pyspark/sql/{catalog,readwriter,udf,udtf,group}.py, pyspark/sql/pandas/group_ops.py and
pyspark/sql/streaming/{readwriter,query}.py are vendored from v4.3.0 as they are, under the
drift gate, the ruff excludes and the coverage omits. pyspark.sql.utils gets PySpark's
ForeachBatchFunction and get_active_spark_context, which they import.
- The drop-in's code of these modules moves to the modules of pyspark.sql.connect of the same
names: the UDF and UDTF classes and registrations, udf, udtf and arrow_udtf, the decoding and
the dispatch of the listener events, and _construct_foreach_function. The classes of the UDFs
and UDTFs and of their registrations get PySpark Connect's docstrings.
- session.rs makes spark.udf and spark.udtf of the Connect modules; streaming.rs gives Trigger,
the listener bus and its event stream the modules of PySpark Connect. The native
SkipRestOfInputTableException, which nothing used, is removed for PySpark's class.
- gen_type_stubs.py no longer stubs the vendored modules, and the stubs of the Connect modules
re-export the names that the drop-in's modules have.
- docs/compatibility.md, dev/parity/known-differences.md and AGENTS.md describe the classes.
Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A follow-up of #775 (SPARK-61852), which made the classic classes of
pyspark.sql.conf,tvf,mergeandobservationPySpark's. The classic names of the other objects of a session still named the drop-in's Connect classes. Soisinstance(spark.read, pyspark.sql.readwriter.DataFrameReader)was True, where pyspark-client gives False. Each row was probed on both clients against a Spark 4.3.0 Connect server:isinstanceagainst the classic class ofspark.catalog,read,readStream,streams,udfandudtf; of a DataFrame'swrite,writeTo,writeStream,groupByandcogroup; of a StreamingQuery and of a UDTFpyspark.sql.Catalog,DataFrameReader,DataFrameWriter,DataFrameWriterV2,GroupedData,PandasCogroupedOps,UDFRegistrationandUDTFRegistration;pyspark.sql.streaming.DataStreamReader,DataStreamWriter,StreamingQueryandStreamingQueryManager;pyspark.sql.functions.UserDefinedFunctionandUserDefinedTableFunctionpyspark.sql.readwriter.to_strandOptionUtils;pyspark.sql.group.dfapianddf_varargs_api; the classicGroupedData's basePandasGroupedOpsMixinpyspark.sql.DataFrameReader(spark),pyspark.sql.Catalog(spark),pyspark.sql.streaming.DataStreamReader(spark)JVM_ATTRIBUTE_NOT_SUPPORTEDof_jsparkSessionpyspark.sql.udf.UDFRegistration(spark).register("f", f)fJVM_ATTRIBUTE_NOT_SUPPORTEDof_jsparkSessionpyspark.sql.udf.UserDefinedFunction(f)(col)ModuleNotFoundError: No module named 'pyspark.sql.classic'pyspark.sql.DataFrameWriter(df)TypeError: cannot create ... instancesJVM_ATTRIBUTE_NOT_SUPPORTEDof_jdfThe names of the eight classic modules are now those of pyspark-client's modules, none missing and none extra. The names of
pyspark.sql,pyspark.sql.streamingandpyspark.sql.functionsare the same classes as pyspark-client's, except those of the first remaining difference below.How:
The classic modules.
python/pyspark/sql/{catalog,readwriter,udf,udtf,group}.py,python/pyspark/sql/pandas/group_ops.pyandpython/pyspark/sql/streaming/{readwriter,query}.pyare vendored from v4.3.0 as they are, under the drift gate, the ruff excludes and the coverage omits.pyspark.sql.utilsgets PySpark'sForeachBatchFunctionandget_active_spark_context, which they import.The Connect modules. The drop-in's code of these modules moves to the modules of
pyspark.sql.connectof the same names, as PySpark Connect's:udf,udtfandarrow_udtf, toconnect/udf.pyandconnect/udtf.py. The registrations ofspark.udfkeep [SPARK-62023][CONNECT][PYTHON] Make SparkSession.client PySpark's SparkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read #863's (SPARK-62023):register,registerJavaFunctionandregisterJavaUDAFregister throughclient.register_udfandclient.register_java, as PySpark Connect's do;connect/streaming/query.py;_construct_foreach_function, toconnect/streaming/readwriter.py;connect/catalog.py,connect/readwriter.pyandconnect/group.pyre-export the native classes.The Connect modules take from the vendored modules what PySpark Connect's take from them: the dataclasses of a UDTF's
analyze,_validate_udtf_handlerand the_check_return_typeof the classicUserDefinedFunction. The drop-in's copies of these, and ofStreamingCheckpointManager, are removed, as the vendored modules have PySpark's. The classes of the UDFs and UDTFs and of their registrations get PySpark Connect's docstrings. To review the move, compare each old shim with its new module, for examplegit diff 3db9f3f2:python/pyspark/sql/udf.py HEAD:python/pyspark/sql/connect/udf.py. Apart from those removals, 18 lines are added inudf.py, 22 inudtf.py, 5 instreaming/query.pyand 9 instreaming/readwriter.py.The native classes.
session.rsmakesspark.udfandspark.udtfofpyspark.sql.connect.udfandpyspark.sql.connect.udtf.streaming.rs:Triggerthe modulepyspark.sql.connect.streaming.readwriter, where it pickles;pyspark.sql.connect.streaming.query, of PySpark Connect'sStreamingQueryListenerBus;_construct_foreach_functionthere.The native
SkipRestOfInputTableException(udtf_analyze.rs), which nothing raised or imported, is removed: the class is PySpark's, of the vendoredpyspark.sql.udtf.The stubs.
scripts/gen_type_stubs.pyno longer stubs the eight vendored modules, which type checkers now read as they do with pyspark-client. The stubs of the Connect modules re-export the names that the drop-in's modules have, and the docstring golden is captured again.pyspark.sql.readwriter.to_strleavesUNRESOLVED_IMPORTS, anddfapi,df_varargs_api,PandasGroupedOpsMixinand the classicOptionUtilsleaveTYPE_ONLY.docs/compatibility.mdanddev/parity/known-differences.mddescribe the classes. Two differences remain, and they are documented:pyspark.sql.Column,DataFrameNaFunctions,DataFrameStatFunctions,WindowSpecandpyspark.sql.table_arg.TableArgstay the Connect classes. PySpark's are the classic bases of those ofpyspark.sql.connect, soisinstanceis True in both clients, but the MRO differs.OptionUtilsofpyspark.sql.connect.readwriter. The drop-in's native classes are not.AGENTS.mdlists the vendored classic modules.Why are the changes needed?
So that the objects of a session are of the classes that pyspark-client gives them, and the classic modules are PySpark's. Code that tells Spark Connect from Spark Classic by the class, such as
isinstance(spark.read, pyspark.sql.readwriter.DataFrameReader), gets pyspark-client's answer. PySpark's helpers of these modules, such aspyspark.sql.readwriter.to_str, are there too.Does this PR introduce any user-facing change?
Yes, as listed above. The classic classes are PySpark's. Code that constructs or subclasses them on Spark Connect now raises as it does with pyspark-client, for example
pyspark.sql.DataFrameReader(spark). Before, it got the drop-in's object.spark.read, the other objects of a session, and the classes ofpyspark.sql.connectare unchanged.How was this patch tested?
New tests that fail on master:
python/tests/test_dropin_class_modules_offline.py, 15 tests:pyspark.sqlorpyspark.sql.streaming, and the object is not an instance of it;to_str,OptionUtils,dfapi,PandasGroupedOpsMixinandStreamingCheckpointManager;udf,pandas_udf,udtfandarrow_udtfmake objects of the Connect classes, which are not instances of the classic ones;Triggerand the listener bus are of the Connect modules;StreamingQueryis PySpark's.test_dropin_udf_args_offline.py,test_the_classic_names_are_pysparks_classic_classes, which replaces the test of the old aliases.test_dropin_e2e.py,test_the_objects_of_a_session_are_not_of_the_classic_classes_as_pysparks, which checks the objects of a live session, a StreamingQuery among them. The same body passes with pyspark-client 4.3.0.Updated tests. 19 tests of
test_dropin_offline.py,test_dropin_udf_args_offline.pyandstreaming_listener_cases.pyreached the drop-in's classes through the classic paths. They now use the Connect paths.Two more tests of the moved code, in
test_dropin_udf_args_offline.py, which pass on master too:udf()andudtf()raise an error of the active session other thanNO_ACTIVE_OR_DEFAULT_SESSION, as PySpark Connect's;PythonUDTFof a UDTF of no return type has noreturn_type.The official suites of
build_python_connect.yml. Master 585cdfe and this branch ran back to back on the same Spark 4.3.0 server, and the results are the same file by file:run_official_tests.py --dropin, 118 files): 97 files pass, the manifest skips 14, and 7 have the same 43 failures (2720 passed, 43 failed, 496 skipped). Master's run came first, under a load average of 30 to 50 from other work on the machine, and 14 more of its files timed out or failed with gRPC errors. Run again back to back on both, those 14 files give the same results: 12 pass, andtest_parity_foreach_batch.pyandtest_parity_streaming.pyhave the same failures as the full runs;test_connect_session.pyandtest_connect_basic.py: 34 files pass and 5 have the same 41 failures (1457 passed, 41 failed, 389 skipped). On the first base of this change, d8011d8, the full transport suite (87 files) was the same on both too.The failures are those of this environment, the same on both. The server ran outside a Spark source tree, so the test data that the tests read by relative paths (
python/test_support/...,data/mllib/...) does not exist there (PATH_NOT_FOUND).The drop-in live-server tests of
build_python_connect.yml(the 31 files,test_dropin_e2e.pyamong them), master and this branch back to back on another server: 1252 and 1253 tests, and the only difference is the new test oftest_dropin_e2e.py. All of the branch's passed. On master,test_window_specs_captured_by_foreach_batch_functions_run_as_pysparkandtest_udfs_captured_by_foreach_batch_functions_run_as_pysparkfailed on the files of their tables in the server's warehouse (LOCATION_ALREADY_EXISTS, and achmodof a_temporarydirectory that was gone). The tables have fixed names and the server was shared with other clients. Run again on both, back to back, they pass.The gates, run again after the rebase onto master 3857dae (the six commits since 585cdfe rebased without conflicts):
scripts/check_vendored_upstream.shagainst thev4.3.0tag from a local clone of apache/spark (177 paths);gen_type_stubs.py --check,gen_fn_dispatch.py --check,scripts/tests(110 passed, 2 skipped) and the guards of SPARK-61851;test_type_stubs_offline.pywith mypy 2.4.1 and pyright 1.1.414 (16 passed);checkandformat --check,test_ruff_excludes.pyandaudit_no_stubs.sh;cargo build -p pyspark-rs --release,cargo fmt --all --check, clippy 1.100.0 with-D warnings, rustdoc with-D warnings, andcargo test -p pyspark-rs --libasrust.ymlruns it (15 passed);coverage.yml: 6523 passed and 77 skipped (68 for "not a public constructor", 8 that needSPARK_REMOTE, and the stub test, which needs mypy and ran above);.coveragercnow omits*/pyspark/sql/connect/streaming/*. The drop-in's streaming code (the listener bus and the functions offoreach) moved there frompyspark/sql/streaming/, and*/pyspark/sql/streaming/*is already omitted by the same policy: streaming has server-backed tests of its own.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.