Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,14 +214,18 @@ history, and update via `git subtree pull`.

Rigorously classify before touching anything (diff our dir vs the version tag):
- **Rust-backed shims — KEEP OURS** (anything importing `pyspark._pyspark`, or an
adapter): `sql/*` top-level, `ml/connect/*`, `sql/streaming/{query,readwriter}.py`,
`errors/*`, `pyspark/__init__.py`, `util.py` (Connect-only trim), `storagelevel.py`.
adapter): most of `sql/*` top-level, `sql/connect/*` (the classes of the objects of a
session, as PySpark Connect's), `ml/connect/*`, `errors/*`, `pyspark/__init__.py`, `util.py`
(Connect-only trim), `storagelevel.py`.
- **Pure upstream — VENDOR via subtree**: `cloudpickle`, `pandas` (see caveat),
`_globals.py`, `loose_version.py`, `serializers.py`, `resource/*` and
`sql/connect/resource/*` (PySpark's resource profiles, whose profile id the connect
`ResourceProfile` creates through `SparkSession.client`), and audit for more. A directory of
shims can hold vendored files too, such as most of `errors/*` and `sql/session.py`:
`VENDORED_FILES` of `scripts/check_vendored_upstream.sh` lists every vendored file.
shims can hold vendored files too, such as most of `errors/*`, `sql/session.py` and the
classic modules of the objects of a session (`sql/{catalog,conf,readwriter,udf,udtf,group}.py`,
`sql/streaming/{query,readwriter}.py`, `sql/pandas/group_ops.py` and others), whose objects on
Spark Connect are of the classes of `sql/connect/*`, as in PySpark: `VENDORED_FILES` of
`scripts/check_vendored_upstream.sh` lists every vendored file.

Subtree conversion (serial, needs a **clean working tree**, one dir at a time):
```
Expand Down
6 changes: 0 additions & 6 deletions crates/pyspark-rs/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ mod transport;
mod tvf;
mod types;
mod udf_command;
mod udtf_analyze;
mod values;
mod window;

Expand Down Expand Up @@ -245,11 +244,6 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {
// Register TVF + Observation
m.add_class::<tvf::PyTableValuedFunction>()?;
m.add_class::<tablearg::PyTableArg>()?;
m.add(
"SkipRestOfInputTableException",
m.py()
.get_type::<udtf_analyze::SkipRestOfInputTableException>(),
)?;

m.add_class::<observation::PyObservation>()?;

Expand Down
10 changes: 6 additions & 4 deletions crates/pyspark-rs/src/session.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3448,11 +3448,13 @@ impl PySparkSession {
}

/// Get the UDF registration API (`spark.udf`). Returns the
/// `pyspark.sql.udf.UDFRegistration` bound to this session, so it can register
/// Python UDFs and Java UDFs/UDAFs (registerJavaFunction / registerJavaUDAF).
/// `pyspark.sql.connect.udf.UDFRegistration` bound to this session, as PySpark Connect's, so
/// it can register Python UDFs and Java UDFs/UDAFs (registerJavaFunction / registerJavaUDAF).
#[getter(udf)]
fn get_udf<'a>(slf: &Bound<'a, Self>, py: Python<'a>) -> PyResult<Bound<'a, PyAny>> {
let cls = py.import("pyspark.sql.udf")?.getattr("UDFRegistration")?;
let cls = py
.import("pyspark.sql.connect.udf")?
.getattr("UDFRegistration")?;
cls.call1((slf,))
}

Expand Down Expand Up @@ -4062,7 +4064,7 @@ impl PySparkSession {
#[getter]
fn udtf<'a>(slf: &Bound<'a, Self>) -> PyResult<Bound<'a, PyAny>> {
slf.py()
.import("pyspark.sql.udtf")?
.import("pyspark.sql.connect.udtf")?
.getattr("UDTFRegistration")?
.call1((slf,))
}
Expand Down
13 changes: 8 additions & 5 deletions crates/pyspark-rs/src/streaming.rs
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ fn post_listener_event<'a>(
event: &StreamingQueryListenerEvent,
started: bool,
) -> std::result::Result<(), ListenerDispatchError> {
let module = py.import("pyspark.sql.streaming.query")?;
let module = py.import("pyspark.sql.connect.streaming.query")?;
let kwargs = PyDict::new(py);
kwargs.set_item("started", started)?;
let decoded = module.call_method(
Expand Down Expand Up @@ -1153,7 +1153,7 @@ impl PyDataStreamReader {
/// Python wrapper for Trigger.
#[pyclass(
name = "Trigger",
module = "pyspark.sql.streaming.readwriter",
module = "pyspark.sql.connect.streaming.readwriter",
from_py_object
)]
#[derive(Clone)]
Expand Down Expand Up @@ -1740,7 +1740,7 @@ impl PyDataStreamWriter {
f: &Bound<'_, PyAny>,
) -> PyResult<Bound<'py, Self>> {
let func = py
.import("pyspark.sql.streaming.readwriter")?
.import("pyspark.sql.connect.streaming.readwriter")?
.getattr("_construct_foreach_function")?
.call1((f,))?;
let serializers = py.import("pyspark.serializers")?;
Expand Down Expand Up @@ -2125,7 +2125,7 @@ fn warn(py: Python<'_>, line: u32, message: String) -> PyResult<()> {
/// which Apache Spark's tests read to remove them.
#[pyclass(
name = "StreamingQueryListenerBus",
module = "pyspark.sql.streaming.query",
module = "pyspark.sql.connect.streaming.query",
frozen
)]
pub struct PyStreamingQueryListenerBus {
Expand Down Expand Up @@ -2381,7 +2381,10 @@ impl PyStreamingQueryManager {

/// Python wrapper for ListenerEventStream.
/// Implements `__iter__` and `__next__` to yield (event_type: i32, event_json: String) tuples.
#[pyclass(name = "ListenerEventStream", module = "pyspark.sql.streaming.query")]
#[pyclass(
name = "ListenerEventStream",
module = "pyspark.sql.connect.streaming.query"
)]
pub struct PyListenerEventStream {
inner: Option<ListenerEventStream>,
/// The session of the stream, whose waits a signal of the process ends.
Expand Down
13 changes: 0 additions & 13 deletions crates/pyspark-rs/src/udtf_analyze.rs

This file was deleted.

21 changes: 12 additions & 9 deletions dev/parity/known-differences.md
Original file line number Diff line number Diff line change
Expand Up @@ -565,10 +565,11 @@ each UDF, UDTF and map or group-map function of its plan when it builds the plan
which their copies and pickles keep as PySpark's do, but not their state, which PySpark's
holds there (`vars(spark.streams)` is empty, where PySpark's has `_session` and `_sqlb`).
- **Subclasses** (intentional; the state is native). PySpark's classes are plain classes that a
user subclasses. The drop-in's DataFrame, `Observation`, `SparkSession`, `Catalog`,
`DataSourceRegistration`, `DataFrameReader`, `DataStreamReader`, `SparkSession.Hook`,
`StorageLevel`, `PythonEvalType` and the data types are subclassed as PySpark's: a subclass
constructs of the arguments of its `__init__`, and its copies and pickles are of the subclass
user subclasses. The drop-in's DataFrame, `Observation`, the `SparkSession`, `Catalog`,
`DataSourceRegistration`, `DataFrameReader` and `DataStreamReader` of `pyspark.sql.connect`,
`SparkSession.Hook`, `StorageLevel`, `PythonEvalType` and the data types are subclassed as
PySpark's: a subclass constructs of the arguments of its `__init__`, and its copies and pickles
are of the subclass
(but those of a session, which are the active session of its ID, as PySpark's), and PySpark's
pickle of a reader, a catalog or a registration, which makes it of its class without
arguments, loads in the drop-in. The `Window` of `pyspark.sql.window` and of
Expand Down Expand Up @@ -694,15 +695,17 @@ each UDF, UDTF and map or group-map function of its plan when it builds the plan
mypy and pyright type an application as with `pyspark-client` (see `docs/compatibility.md`), but:

- **Imports of PySpark's modules** (intentional). A name that a module of PySpark imports, such as
`DataFrame` of `pyspark.sql.catalog`, is an attribute of the module, which mypy takes by default
with `pyspark-client`. The stubs declare it only when the drop-in's module has it at run time, so
that it is an error, as `from pyspark.sql.catalog import DataFrame` fails with the drop-in.
`DataFrame` of `pyspark.sql.connect.catalog`, is an attribute of the module, which mypy takes by
default with `pyspark-client`. The stubs declare it only when the drop-in's module has it at run
time, so that it is an error, as `from pyspark.sql.connect.catalog import DataFrame` fails with
the drop-in.
pyright reports it as a private import with `pyspark-client` too.
- **Names that only PySpark's annotations need** (intentional). `TYPE_ONLY` of
`python/tests/test_type_stubs_offline.py` lists the names that the stubs declare and the drop-in
does not have, with their reasons: the classic-only `DataFrame.rdd`, as type checkers of
`pyspark-client` see it, and the bases, metaclass, decorators and internal classes of PySpark's
classes that the drop-in implements natively.
`pyspark-client` see it, the type variables of the annotations of `pyspark.sql.types`, and the
base and the internal class of PySpark Connect's classes that the drop-in implements natively,
the `OptionUtils` of its readers and writers and its `StreamingQueryListenerBus`.
- **`SparkSession.client`** (intentional; the drop-in's own class). The client has PySpark
Connect's public API, of the class `pyspark.sql.connect.client.SparkConnectClient` rather than
`pyspark.sql.connect.client.core`'s.
Expand Down
69 changes: 40 additions & 29 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -2082,16 +2082,16 @@ differences:
re-exported when the drop-in's module has it too, so that `F.partitioning`, which pyright
reports as a private import with `pyspark-client`, and `F.PandasUDFType`, which mypy with
`--no-implicit-reexport` reports, are not reported. One that the drop-in's module does not
have, such as `pyspark.sql.catalog.DataFrame`, is an error, where mypy takes it with
have, such as `pyspark.sql.connect.catalog.DataFrame`, is an error, where mypy takes it with
`pyspark-client` by default, as it fails at run time. So is the plumbing of PySpark's modules
that the drop-in does not have, such as the py4j converters of `pyspark.sql.types`.
- A few names that PySpark's annotations need are declared, which the drop-in does not have at run
time: `DataFrame.rdd`, which PySpark defines when not `is_remote_only()`, so that type checkers
of `pyspark-client` see it too, and neither client has it; and the bases and the decorators of
PySpark's classes that the drop-in implements natively, such as `OptionUtils` of the readers and
writers, `PandasGroupedOpsMixin` of `GroupedData` and `dfapi` (`TYPE_ONLY` of the test lists
them with their reasons). `pyspark.sql.types` has `DataTypeSingleton`, the metaclass of the
singleton types of the stubs, at run time too, of which the native classes are not. The
of `pyspark-client` see it too, and neither client has it; and the classes of PySpark Connect
that the drop-in implements natively in another way, `OptionUtils`, the base of its readers and
writers, and the `StreamingQueryListenerBus` of its `StreamingQueryManager` (`TYPE_ONLY` of the
test lists them with their reasons). `pyspark.sql.types` has `DataTypeSingleton`, the metaclass
of the singleton types of the stubs, at run time too, of which the native classes are not. The
`SparkContext`, `RDD` and the other names of classic PySpark that PySpark's `pyspark` module
imports of `pyspark.core` when not `is_remote_only()`, which `pyspark-client` does not ship,
the drop-in's module imports so too, in a block that does not run either: mypy takes them as
Expand All @@ -2110,9 +2110,9 @@ differences:
pyright does not see in PySpark's source, as its modules copy them there at run time: a hover
over `col` of `pyspark.sql.connect.functions` or the `builder` of its `SparkSession` shows its
documentation with the drop-in, and none with `pyspark-client`.
- The names that only the drop-in has, such as `Trigger` of `pyspark.sql.streaming.readwriter` and
`FrameBound` of `pyspark.sql.window`, are typed with the drop-in and are errors with
`pyspark-client`, as at run time.
- The names that only the drop-in has, such as `Trigger` of
`pyspark.sql.connect.streaming.readwriter` and `FrameBound` of `pyspark.sql.window`, are typed
with the drop-in and are errors with `pyspark-client`, as at run time.

PySpark's `pyspark.sql.functions` is a package of the modules `builtin` and `partitioning`, where
the drop-in's is a module, with the stubs of the three in the directory `pyspark/sql/functions/`,
Expand Down Expand Up @@ -2767,11 +2767,12 @@ built then, as PySpark's sends the protos that `_build_merge_action` built for t
value that a function reads, changed after the clause, is not that of the merge, also of a copy of
the builder, and a window of no partitions of a clause warns once, at the clause.

The drop-in has one class for each UDF and UDTF, the Connect class of `pyspark.sql.connect.udf`
and `pyspark.sql.connect.udtf`, which the classic names `pyspark.sql.udf.UserDefinedFunction` and
`pyspark.sql.udtf.UserDefinedTableFunction` also name. So `isinstance` of a UDTF that `udtf` builds
is True against either name, where PySpark 4.3.0 gives False for the classic name, whose class
needs a SparkContext. `udf` gives a function in both clients, which is of neither class.
A UDF and a UDTF are of PySpark Connect's classes, `UserDefinedFunction` of
`pyspark.sql.connect.udf` and `UserDefinedTableFunction` of `pyspark.sql.connect.udtf`, as in
PySpark. The classic `pyspark.sql.udf` and `pyspark.sql.udtf` are PySpark's modules, of Apache
Spark as they are, whose classes, which `pyspark.sql.functions` names too, need a SparkContext: so
`isinstance` of a UDTF that `udtf` builds against the classic `UserDefinedTableFunction` is False,
as in PySpark. `udf` gives a function in both clients, which is of neither class.

So it is for the objects of a session: each is of PySpark Connect's class, of its module of
`pyspark.sql.connect`, as `repr`, `type(x).__module__` and pickle show it. These are the
Expand Down Expand Up @@ -2807,26 +2808,36 @@ without a URL or a channel builder, returns the active session from `getOrCreate
`spark.sql.catalogImplementation`.

The classic classes of `pyspark.sql.conf`, `pyspark.sql.tvf`, `pyspark.sql.datasource`,
`pyspark.sql.merge` and `pyspark.sql.observation` are PySpark's, of those modules of Apache Spark
as they are, so `isinstance` of a session's `conf`, `tvf`, `dataSource` and `mergeInto` against
them is False, as in PySpark. `Observation(...)` makes an Observation of
`pyspark.sql.connect.observation`, which is not an instance of `pyspark.sql.Observation`, as
PySpark's `__new__` makes one. `pyspark.sql.connect.window.Window` is a subclass of
`pyspark.sql.window.Window`, as in PySpark. For the other objects of a session, the classic name
names PySpark Connect's class, so `isinstance` against it is True, where PySpark gives False:
`Catalog`, `DataFrameReader`, `DataFrameWriter`, `DataFrameWriterV2`, `DataStreamReader`,
`DataStreamWriter`, `StreamingQueryManager`, `StreamingQuery`, `UDFRegistration`,
`UDTFRegistration`, `GroupedData` and `PandasCogroupedOps`, and the UDTFs above. The classic
`DataFrameNaFunctions`, `DataFrameStatFunctions`, `WindowSpec` and `TableArg` name PySpark
Connect's classes too, which in PySpark are their subclasses: `isinstance` is True in both, and
`pyspark.sql.merge`, `pyspark.sql.observation`, `pyspark.sql.catalog`, `pyspark.sql.readwriter`,
`pyspark.sql.udf`, `pyspark.sql.udtf`, `pyspark.sql.group`, `pyspark.sql.pandas.group_ops`,
`pyspark.sql.streaming.readwriter` and `pyspark.sql.streaming.query` are PySpark's, of those
modules of Apache Spark as they are, which `pyspark.sql` and `pyspark.sql.streaming` name as
PySpark's do. So `isinstance` against them is False, as in PySpark, of a session's `conf`, `tvf`,
`dataSource`, `catalog`, `read`, `readStream`, `streams`, `udf` and `udtf`, of a DataFrame's
`mergeInto`, `write`, `writeTo`, `writeStream`, `groupBy` and `cogroup`, of a StreamingQuery and of
the UDTFs above. Their constructors and methods are those of Spark Classic, as PySpark's:
`pyspark.sql.DataFrameReader(spark)` and `pyspark.sql.Catalog(spark)` raise the
`JVM_ATTRIBUTE_NOT_SUPPORTED` of `_jsparkSession`, and so does the `register` of a
`pyspark.sql.udf.UDFRegistration(spark)`, as in PySpark, where the reader of `spark.read` and
`pyspark.sql.connect.readwriter.DataFrameReader(spark)` are PySpark Connect's.
`Observation(...)` makes an Observation of `pyspark.sql.connect.observation`, which is not an
instance of `pyspark.sql.Observation`, as PySpark's `__new__` makes one.
`pyspark.sql.connect.window.Window` is a subclass of `pyspark.sql.window.Window`, as in PySpark.
The classic `DataFrameNaFunctions`, `DataFrameStatFunctions`, `WindowSpec` and `TableArg` name
PySpark Connect's classes, which in PySpark are their subclasses: `isinstance` is True in both, and
the MRO of such an object is its native class and `object`. The concrete DataFrame and Column are
not subclasses of the public `pyspark.sql.DataFrame` and `pyspark.sql.Column`: `isinstance` holds
by registration, so the MRO of a DataFrame is its native class and `object`, and
`pyspark.sql.Column` names the concrete class.
`pyspark.sql.Column` names the concrete class, where PySpark's is the abstract class of the
concrete one of `pyspark.sql.connect.column`. The readers and writers of PySpark Connect are
subclasses of the `OptionUtils` of `pyspark.sql.connect.readwriter`, a class of private methods
only, of which the drop-in's native readers and writers are not: their MRO is their class and
`object`, and `pyspark.sql.connect.readwriter` has no `OptionUtils`.

PySpark's classes are plain Python classes, which a user subclasses. So are these of the drop-in:
the DataFrame, `Observation`, `SparkSession`, `Catalog`, `DataSourceRegistration`,
`DataFrameReader` and `DataStreamReader`, which construct of their PySpark arguments
the DataFrame, `Observation`, and the `SparkSession`, `Catalog`, `DataSourceRegistration`,
`DataFrameReader` and `DataStreamReader` of `pyspark.sql.connect`, which construct of their PySpark
arguments
(`SparkSession(connection=...)`, `Catalog(spark)`, `DataFrameReader(spark)`), also a subclass of an
`__init__` of its own, and whose setters, copies and pickles are of the subclass (but those of a
session, which are the active session of its ID, as PySpark's);
Expand Down
Loading
Loading