Skip to content

[SPARK-62071][CONNECT][PYTHON] Make the classic modules of the objects of a session PySpark's, so that the objects are not instances of the classic classes, as in pyspark-client - #881

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62071-classic-modules-verbatim
Closed

HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62071-classic-modules-verbatim

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

A follow-up of #775 (SPARK-61852), which made the classic classes of pyspark.sql.conf, tvf, merge and observation PySpark's. The classic names of the other objects of a session still named the drop-in's Connect classes. So isinstance(spark.read, pyspark.sql.readwriter.DataFrameReader) was True, where pyspark-client gives False. Each row was probed on both clients against a Spark 4.3.0 Connect server:

before PySpark 4.3.0, and after
isinstance against the classic class of spark.catalog, read, readStream, streams, udf and udtf; of a DataFrame's write, writeTo, writeStream, groupBy and cogroup; of a StreamingQuery and of a UDTF True False
pyspark.sql.Catalog, DataFrameReader, DataFrameWriter, DataFrameWriterV2, GroupedData, PandasCogroupedOps, UDFRegistration and UDTFRegistration; pyspark.sql.streaming.DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryManager; pyspark.sql.functions.UserDefinedFunction and UserDefinedTableFunction the Connect classes PySpark's classic classes
pyspark.sql.readwriter.to_str and OptionUtils; pyspark.sql.group.dfapi and df_varargs_api; the classic GroupedData's base PandasGroupedOpsMixin missing PySpark's
pyspark.sql.DataFrameReader(spark), pyspark.sql.Catalog(spark), pyspark.sql.streaming.DataStreamReader(spark) the drop-in's reader or catalog JVM_ATTRIBUTE_NOT_SUPPORTED of _jsparkSession
pyspark.sql.udf.UDFRegistration(spark).register("f", f) registers f JVM_ATTRIBUTE_NOT_SUPPORTED of _jsparkSession
pyspark.sql.udf.UserDefinedFunction(f)(col) a Column ModuleNotFoundError: No module named 'pyspark.sql.classic'
pyspark.sql.DataFrameWriter(df) TypeError: cannot create ... instances JVM_ATTRIBUTE_NOT_SUPPORTED of _jdf

The names of the eight classic modules are now those of pyspark-client's modules, none missing and none extra. The names of pyspark.sql, pyspark.sql.streaming and pyspark.sql.functions are the same classes as pyspark-client's, except those of the first remaining difference below.

How:

  • The classic modules. python/pyspark/sql/{catalog,readwriter,udf,udtf,group}.py, python/pyspark/sql/pandas/group_ops.py and python/pyspark/sql/streaming/{readwriter,query}.py are vendored from v4.3.0 as they are, under the drift gate, the ruff excludes and the coverage omits. pyspark.sql.utils gets PySpark's ForeachBatchFunction and get_active_spark_context, which they import.

  • The Connect modules. The drop-in's code of these modules moves to the modules of pyspark.sql.connect of the same names, as PySpark Connect's:

    The Connect modules take from the vendored modules what PySpark Connect's take from them: the dataclasses of a UDTF's analyze, _validate_udtf_handler and the _check_return_type of the classic UserDefinedFunction. The drop-in's copies of these, and of StreamingCheckpointManager, are removed, as the vendored modules have PySpark's. The classes of the UDFs and UDTFs and of their registrations get PySpark Connect's docstrings. To review the move, compare each old shim with its new module, for example git diff 3db9f3f2:python/pyspark/sql/udf.py HEAD:python/pyspark/sql/connect/udf.py. Apart from those removals, 18 lines are added in udf.py, 22 in udtf.py, 5 in streaming/query.py and 9 in streaming/readwriter.py.

  • The native classes. session.rs makes spark.udf and spark.udtf of pyspark.sql.connect.udf and pyspark.sql.connect.udtf. streaming.rs:

    • gives Trigger the module pyspark.sql.connect.streaming.readwriter, where it pickles;
    • gives the listener bus and its event stream the module pyspark.sql.connect.streaming.query, of PySpark Connect's StreamingQueryListenerBus;
    • calls the listener functions and _construct_foreach_function there.

    The native SkipRestOfInputTableException (udtf_analyze.rs), which nothing raised or imported, is removed: the class is PySpark's, of the vendored pyspark.sql.udtf.

  • The stubs. scripts/gen_type_stubs.py no longer stubs the eight vendored modules, which type checkers now read as they do with pyspark-client. The stubs of the Connect modules re-export the names that the drop-in's modules have, and the docstring golden is captured again. pyspark.sql.readwriter.to_str leaves UNRESOLVED_IMPORTS, and dfapi, df_varargs_api, PandasGroupedOpsMixin and the classic OptionUtils leave TYPE_ONLY.

docs/compatibility.md and dev/parity/known-differences.md describe the classes. Two differences remain, and they are documented:

  • pyspark.sql.Column, DataFrameNaFunctions, DataFrameStatFunctions, WindowSpec and pyspark.sql.table_arg.TableArg stay the Connect classes. PySpark's are the classic bases of those of pyspark.sql.connect, so isinstance is True in both clients, but the MRO differs.
  • PySpark Connect's readers and writers are subclasses of the OptionUtils of pyspark.sql.connect.readwriter. The drop-in's native classes are not.

AGENTS.md lists the vendored classic modules.

Why are the changes needed?

So that the objects of a session are of the classes that pyspark-client gives them, and the classic modules are PySpark's. Code that tells Spark Connect from Spark Classic by the class, such as isinstance(spark.read, pyspark.sql.readwriter.DataFrameReader), gets pyspark-client's answer. PySpark's helpers of these modules, such as pyspark.sql.readwriter.to_str, are there too.

Does this PR introduce any user-facing change?

Yes, as listed above. The classic classes are PySpark's. Code that constructs or subclasses them on Spark Connect now raises as it does with pyspark-client, for example pyspark.sql.DataFrameReader(spark). Before, it got the drop-in's object. spark.read, the other objects of a session, and the classes of pyspark.sql.connect are unchanged.

How was this patch tested?

  • New tests that fail on master:

    • In python/tests/test_dropin_class_modules_offline.py, 15 tests:
      • the classic class of each of the 11 objects of a session above is PySpark's, of its module, named by pyspark.sql or pyspark.sql.streaming, and the object is not an instance of it;
      • the classic modules have PySpark's to_str, OptionUtils, dfapi, PandasGroupedOpsMixin and StreamingCheckpointManager;
      • udf, pandas_udf, udtf and arrow_udtf make objects of the Connect classes, which are not instances of the classic ones;
      • Trigger and the listener bus are of the Connect modules;
      • the classic StreamingQuery is PySpark's.
    • In test_dropin_udf_args_offline.py, test_the_classic_names_are_pysparks_classic_classes, which replaces the test of the old aliases.
    • In test_dropin_e2e.py, test_the_objects_of_a_session_are_not_of_the_classic_classes_as_pysparks, which checks the objects of a live session, a StreamingQuery among them. The same body passes with pyspark-client 4.3.0.
  • Updated tests. 19 tests of test_dropin_offline.py, test_dropin_udf_args_offline.py and streaming_listener_cases.py reached the drop-in's classes through the classic paths. They now use the Connect paths.

  • Two more tests of the moved code, in test_dropin_udf_args_offline.py, which pass on master too:

    • udf() and udtf() raise an error of the active session other than NO_ACTIVE_OR_DEFAULT_SESSION, as PySpark Connect's;
    • the PythonUDTF of a UDTF of no return type has no return_type.
  • The official suites of build_python_connect.yml. Master 585cdfe and this branch ran back to back on the same Spark 4.3.0 server, and the results are the same file by file:

    • the drop-in suite (run_official_tests.py --dropin, 118 files): 97 files pass, the manifest skips 14, and 7 have the same 43 failures (2720 passed, 43 failed, 496 skipped). Master's run came first, under a load average of 30 to 50 from other work on the machine, and 14 more of its files timed out or failed with gRPC errors. Run again back to back on both, those 14 files give the same results: 12 pass, and test_parity_foreach_batch.py and test_parity_streaming.py have the same failures as the full runs;
    • the transport parity suite, of the 39 files of UDFs, UDTFs, registrations, the catalog, the readers and writers, streaming, test_connect_session.py and test_connect_basic.py: 34 files pass and 5 have the same 41 failures (1457 passed, 41 failed, 389 skipped). On the first base of this change, d8011d8, the full transport suite (87 files) was the same on both too.

    The failures are those of this environment, the same on both. The server ran outside a Spark source tree, so the test data that the tests read by relative paths (python/test_support/..., data/mllib/...) does not exist there (PATH_NOT_FOUND).

  • The drop-in live-server tests of build_python_connect.yml (the 31 files, test_dropin_e2e.py among them), master and this branch back to back on another server: 1252 and 1253 tests, and the only difference is the new test of test_dropin_e2e.py. All of the branch's passed. On master, test_window_specs_captured_by_foreach_batch_functions_run_as_pyspark and test_udfs_captured_by_foreach_batch_functions_run_as_pyspark failed on the files of their tables in the server's warehouse (LOCATION_ALREADY_EXISTS, and a chmod of a _temporary directory that was gone). The tables have fixed names and the server was shared with other clients. Run again on both, back to back, they pass.

  • The gates, run again after the rebase onto master 3857dae (the six commits since 585cdfe rebased without conflicts):

    • scripts/check_vendored_upstream.sh against the v4.3.0 tag from a local clone of apache/spark (177 paths);
    • gen_type_stubs.py --check, gen_fn_dispatch.py --check, scripts/tests (110 passed, 2 skipped) and the guards of SPARK-61851;
    • test_type_stubs_offline.py with mypy 2.4.1 and pyright 1.1.414 (16 passed);
    • ruff 0.14.0 check and format --check, test_ruff_excludes.py and audit_no_stubs.sh;
    • cargo build -p pyspark-rs --release, cargo fmt --all --check, clippy 1.100.0 with -D warnings, rustdoc with -D warnings, and cargo test -p pyspark-rs --lib as rust.yml runs it (15 passed);
    • the offline suite of coverage.yml: 6523 passed and 77 skipped (68 for "not a public constructor", 8 that need SPARK_REMOTE, and the stub test, which needs mypy and ran above);
    • its Python coverage: 93.35%, above the floor of 93.0 (master 585cdfe: 93.37%). Part of the code that left the measured shims was fully covered, such as the classic catalog shim and the UDTF dataclasses, and PySpark's vendored modules replace it.
    • .coveragerc now omits */pyspark/sql/connect/streaming/*. The drop-in's streaming code (the listener bus and the functions of foreach) moved there from pyspark/sql/streaming/, and */pyspark/sql/streaming/* is already omitted by the same policy: streaming has server-backed tests of its own.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…s of a session PySpark's, so that the objects are not instances of the classic classes, as in pyspark-client

- pyspark/sql/{catalog,readwriter,udf,udtf,group}.py, pyspark/sql/pandas/group_ops.py and
  pyspark/sql/streaming/{readwriter,query}.py are vendored from v4.3.0 as they are, under the
  drift gate, the ruff excludes and the coverage omits. pyspark.sql.utils gets PySpark's
  ForeachBatchFunction and get_active_spark_context, which they import.
- The drop-in's code of these modules moves to the modules of pyspark.sql.connect of the same
  names: the UDF and UDTF classes and registrations, udf, udtf and arrow_udtf, the decoding and
  the dispatch of the listener events, and _construct_foreach_function. The classes of the UDFs
  and UDTFs and of their registrations get PySpark Connect's docstrings.
- session.rs makes spark.udf and spark.udtf of the Connect modules; streaming.rs gives Trigger,
  the listener bus and its event stream the modules of PySpark Connect. The native
  SkipRestOfInputTableException, which nothing used, is removed for PySpark's class.
- gen_type_stubs.py no longer stubs the vendored modules, and the stubs of the Connect modules
  re-export the names that the drop-in's modules have.
- docs/compatibility.md, dev/parity/known-differences.md and AGENTS.md describe the classes.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant