Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 74 additions & 23 deletions crates/pyspark-rs/src/dataframe.rs
Original file line number Diff line number Diff line change
Expand Up @@ -99,18 +99,30 @@ impl PyDataFrame {

/// The schema of this DataFrame, as PySpark's `_cached_schema` is: the object of `to` that is
/// not a DataType, which it keeps (see `schema_object`), or else the schema of the server, as
/// PySpark's client converts it. The schema is cached, so it is asked of the server once.
/// PySpark's client converts it, of the errors of that conversion (see `check_python_schema`).
/// The schema is cached, so it is asked of the server once.
fn own_schema<'py>(&self, py: Python<'py>) -> PyResult<PropertySchema<'py>> {
if let Some(schema) = &self.schema_object {
return Ok(PropertySchema::Object(schema.bind(py).clone()));
}
let schema = interruptible(py, &self.dataframe.spark_session(), || {
self.dataframe.schema()
})?;
check_schema_conversion(py, Some(&schema))?;
check_python_schema(py, &schema)?;
Ok(PropertySchema::Core(schema))
}

/// Raises as PySpark's `schema` of this DataFrame does, which its grouped data reads, of the
/// conversion of the schema that it caches (see `own_schema`): of a copy or an unpickle,
/// whose `schema` is the schema that it requested and converted then (see `copy_schema`),
/// nothing.
pub(crate) fn check_schema_read(&self, py: Python<'_>) -> PyResult<()> {
if self.copy_schema.is_none() {
self.own_schema(py)?;
}
Ok(())
}

/// The schema that the properties `schema`, `columns` and `dtypes` read, as PySpark's read
/// `self._schema`: `own_schema`, or, when the plan fails to build with an `AttributeError`, as
/// of a join on a tuple, the object that Python then gives of `self._schema`, as PySpark's
Expand Down Expand Up @@ -358,7 +370,10 @@ impl PyDataFrame {
.to_pyerr()
.map_err(|err| columns_read_error(py, err))?;
let fresh = DataFrame::from_sent_relation(session.clone(), relation);
interruptible(py, &session, || fresh.schema())
let schema = interruptible(py, &session, || fresh.schema())?;
// As the read of the columns, which converts the schema (see `check_python_schema`).
check_python_schema(py, &schema)?;
Ok(schema)
}

/// The number of rows and the truncation of the eager evaluation, as PySpark's `int` of
Expand Down Expand Up @@ -1009,8 +1024,9 @@ impl PyDataFrame {
},
};
// The schema that this DataFrame cached, of the types of PySpark, when they hold it: the
// `StructType` that it keeps as an object, of `to` or of a map function, as it is now, as
// PySpark's state carries the object.
// `StructType` that it keeps as an object, of `to`, of a map function or of an unpickle,
// as it is now, as PySpark's state carries the object; else that of the server, of new
// objects of its user-defined types, as PySpark's `_schema` made them.
let cached_schema = match &this.schema_object {
Some(schema)
if schema
Expand All @@ -1019,12 +1035,7 @@ impl PyDataFrame {
{
Some(schema.clone_ref(py))
}
_ => this
.dataframe
.known_schema()
.filter(|schema| check_schema_conversion(py, Some(schema)).is_ok())
.map(|schema| crate::types::data_type_to_py(py, &schema))
.transpose()?,
_ => known_schema_object(py, &this.dataframe)?,
};
py.import("pyspark.sql.connect.dataframe")?
.getattr("_reduce_dataframe")?
Expand Down Expand Up @@ -1179,9 +1190,10 @@ impl PyDataFrame {
/// The DataFrame of `relation`, the bytes of a relation of `levels` levels of `session`, as a
/// pickled DataFrame unpickles, with the `_support_repr_html` and the `_cached_schema` of its
/// state. As PySpark's unpickle, whose lookup of the `__setstate__` of the new DataFrame reads
/// its columns, it requests the schema of the relation, whose error it raises, and keeps it
/// (see `copy_schema`), and then the DataFrame has the cached schema of the pickled one,
/// none when it had none, so that its `schema` requests it again then. Without
/// its columns, it requests the schema of the relation, whose error it raises, also that of
/// its conversion (see `check_python_schema`), and keeps it (see `copy_schema`), and then the
/// DataFrame has the cached schema of the pickled one, the objects that it is of, or none
/// when it had none, so that its `schema` requests it again then. Without
/// `request_schema`, of an object that PySpark pickles of the plan of the DataFrame, as a
/// DataStreamWriter, the schema is not requested.
#[staticmethod]
Expand Down Expand Up @@ -1214,12 +1226,18 @@ impl PyDataFrame {
.to_pyerr()?;
let requested = if request_schema {
let schema = interruptible(py, &dataframe.spark_session(), || dataframe.schema())?;
check_schema_conversion(py, Some(&schema))?;
check_python_schema(py, &schema)?;
Some(schema)
} else {
None
};
let mut dataframe = PyDataFrame::new(dataframe.with_known_schema(known_schema));
// The `StructType` of the state as the object that it is, as PySpark's `_cached_schema`,
// which `columns`, `dtypes` and the methods that copy the cached schema read: of the
// objects of `to`, of a map function or of the server that the pickled DataFrame held.
dataframe.schema_object = cached_schema
.filter(|schema| schema.is_instance_of::<crate::types::PyStructType>())
.map(|schema| schema.clone().unbind());
if let Some(requested) = requested {
dataframe.keep_copy_schema(requested);
}
Expand Down Expand Up @@ -7289,13 +7307,51 @@ pub(crate) fn check_schema_conversion(
}
}

/// Raises as PySpark's conversion of `schema`, a schema of the server, raises where its
/// `_schema` makes the type objects of it: the errors of [`check_schema_conversion`], and those of
/// its user-defined types, of which `UserDefinedType.fromJson` makes `UDT()` of the class that
/// the `pyClass` names (see `types::schema_to_py`), so that a class whose `__init__` takes
/// arguments raises its `TypeError`. The properties that read the schema, the column lookups,
/// the copies and the unpickles raise it so.
pub(crate) fn check_python_schema(
py: Python<'_>,
schema: &spark_connect::types::DataType,
) -> PyResult<()> {
check_schema_conversion(py, Some(schema))?;
if crate::types::has_udt(schema) {
crate::types::schema_to_py(py, schema)?;
}
Ok(())
}

/// The `_cached_schema` of PySpark's DataFrame of `dataframe`, of the schema that the core caches,
/// which is of the server: its type objects, as PySpark converts the schema of the server, of a
/// new object of each user-defined type (see `types::schema_to_py`), and not of the objects that
/// the client was given. None when it has none, or when PySpark would not have one, as its
/// conversion of the schema raised (see [`check_python_schema`]).
fn known_schema_object(py: Python<'_>, dataframe: &DataFrame) -> PyResult<Option<Py<PyAny>>> {
let Some(schema) = dataframe.known_schema() else {
return Ok(None);
};
if check_schema_conversion(py, Some(&schema)).is_err() {
return Ok(None);
}
match crate::types::schema_to_py(py, &schema) {
Ok(schema) => Ok(Some(schema)),
Err(err) if err.is_instance_of::<pyo3::exceptions::PyException>(py) => Ok(None),
Err(err) => Err(err),
}
}

/// The result of a set operation of `this` and `other`, `dataframe`, which keeps the schema that
/// PySpark's `_merge_cached_schema` keeps: `this.schema` when the cached schemas of both are
/// equal, else none. The core keeps that of two cached structs; of a schema that a DataFrame
/// keeps as a Python object, the schema of `to` that is not a DataType or the `StructType` of a
/// map function, the objects are compared as they are now, as PySpark compares them with `==`,
/// and the result keeps a deep copy of that of `this`, as `this.schema` gives. Of a copy or an
/// unpickle, `this.schema` is the schema that it requested (see `copy_schema`).
/// with the type objects of the cached schema of the other, which of the server holds new objects
/// of its user-defined types (see `known_schema_object`), and the result keeps a deep copy of
/// that of `this`, as `this.schema` gives. Of a copy or an unpickle, `this.schema` is the schema
/// that it requested (see `copy_schema`).
fn set_operation_result(
py: Python<'_>,
this: &PyDataFrame,
Expand All @@ -7313,12 +7369,7 @@ fn set_operation_result(
let cached = |df: &PyDataFrame| -> PyResult<Option<Py<PyAny>>> {
match &df.schema_object {
Some(schema) => Ok(Some(schema.clone_ref(py))),
None => df
.dataframe
.known_schema()
.filter(|schema| check_schema_conversion(py, Some(schema)).is_ok())
.map(|schema| crate::types::data_type_to_py(py, &schema))
.transpose(),
None => known_schema_object(py, &df.dataframe),
}
};
if let (Some(left), Some(right)) = (cached(this)?, cached(other)?) {
Expand Down
4 changes: 4 additions & 0 deletions crates/pyspark-rs/src/group.rs
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,10 @@ impl PyGroupedData {
};
names.push(name.to_cow().ok().map(|name| name.into_owned()));
}
// As PySpark's `self._df.schema`, which raises the errors of its conversion of the schema
// of the server, such as the `TypeError` of a user-defined type whose `__init__` takes
// arguments.
self.frame.bind(py).borrow().check_schema_read(py)?;
let requested = self.frame.bind(py).borrow().copy_schema();
let of_requested;
let grouped = match requested {
Expand Down
66 changes: 26 additions & 40 deletions crates/pyspark-rs/src/types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2058,46 +2058,12 @@ impl PyDataType {
class_repr(slf.as_any())
}

/// As pyspark's `simpleString()`, the `typeName()` of a type of its class only. An array, map
/// or struct of a user-defined type names it by the `simpleString()` of its Python class,
/// such as `array<vector>`.
fn simpleString(slf: &Bound<'_, Self>) -> PyResult<String> {
let py = slf.py();
// A struct, array or map reads its live type, as its own `simpleString` does: the base
// method is also called on them directly, as `DataType.simpleString(st)`.
if slf.is_instance_of::<PyStructType>()
|| slf.is_instance_of::<PyArrayType>()
|| slf.is_instance_of::<PyMapType>()
{
return match py_to_data_type(slf.as_any()) {
Ok(data_type) => simple_string_with_udts(py, &data_type),
// Of a type that holds one that does not lower now, of the type objects that it
// holds, as PySpark's reads them (see `lenient_type`).
Err(error) if is_exception(py, &error) => {
if let Ok(array) = slf.cast::<PyArrayType>() {
return array.borrow().simple_string(py);
}
if let Ok(map) = slf.cast::<PyMapType>() {
return map.borrow().simple_string(py);
}
slf.cast::<PyStructType>()?.borrow().simple_string(py)
}
Err(error) => Err(error),
};
}
let this = slf.borrow();
let Some(inner) = &this.inner else {
drop(this);
return slf.call_method0(pyo3::intern!(py, "typeName"))?.extract();
};
match inner {
DataType::Array { .. } | DataType::Map { .. } | DataType::Struct { .. }
if has_udt(inner) =>
{
simple_string_with_udts(py, inner)
}
_ => Ok(inner.simple_string()),
}
/// As PySpark's `DataType.simpleString`, `self.typeName()`. The types of a string of their
/// own, such as `bigint`, `decimal(10,2)` or `struct<a:bigint>`, override it, as PySpark's
/// do, so the base method called on one of them directly, as `DataType.simpleString(st)`, is
/// its `typeName()`, `struct`, as in PySpark.
fn simpleString<'py>(slf: &Bound<'py, Self>) -> PyResult<Bound<'py, PyAny>> {
type_name_of(slf.as_any())
}

// Equality/hashing are defined once here and inherited by every concrete type class
Expand Down Expand Up @@ -5945,6 +5911,16 @@ impl PyYearMonthIntervalType {
const YEAR: i32 = 0;
#[classattr]
const MONTH: i32 = 1;
/// As PySpark's `simpleString`, its `_str_repr` of the fields: `interval year to month`, or
/// `interval year` of one field, also of a Python subclass, as it reads no `typeName()`.
#[pyo3(name = "simpleString")]
fn simple_string(&self) -> String {
DataType::YearMonthInterval {
start_field: self.start_field,
end_field: self.end_field,
}
.simple_string()
}
// --- DataType object-model methods (v4.2.0 parity) ---
/// As PySpark's `DataType.json`, of the `jsonValue` of the type, of a Python subclass its own.
#[pyo3(name = "json")]
Expand Down Expand Up @@ -6113,6 +6089,16 @@ impl PyDayTimeIntervalType {
const MINUTE: i32 = 2;
#[classattr]
const SECOND: i32 = 3;
/// As PySpark's `simpleString`, its `_str_repr` of the fields: `interval day to second`, or
/// `interval day` of one field, also of a Python subclass, as it reads no `typeName()`.
#[pyo3(name = "simpleString")]
fn simple_string(&self) -> String {
DataType::DayTimeInterval {
start_field: self.start_field,
end_field: self.end_field,
}
.simple_string()
}
// --- DataType object-model methods (v4.2.0 parity) ---
/// As PySpark's `DataType.json`, of the `jsonValue` of the type, of a Python subclass its own.
#[pyo3(name = "json")]
Expand Down
8 changes: 8 additions & 0 deletions dev/parity/known-differences.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,14 @@ difference is intentional or tracked for a fix.
`TypeError: StructType fields must be StructField objects` there, and an assignment of `fields`
of such an element raises it at once. Of a list of `StructField`s changed in place, the cached
`names` and their uses are PySpark's (see `docs/compatibility.md`).
- **A schema that PySpark does not convert is requested once** (intentional; the errors are the
same). PySpark's `_schema` keeps the schema that it requested only once its conversion of it
succeeded, so of a schema that it does not convert, such as of a user-defined type whose
`__init__` takes arguments or of no Python class, of a nanosecond-precision timestamp, or of a
spatial type of an SRID that it does not take, each read of the schema (`schema`, `columns`,
`dtypes`, `printSchema` and the lookups that read the columns) requests it again, and raises.
This client keeps the schema of the server once it requested it, as of any other schema, and
raises the same error of its conversion at each read, of no other request.
- **The `__repr__` of a Column of a str that UTF-8 cannot encode** (intentional; the expressions
of the core hold UTF-8). Such a str, of a name, a SQL string or a literal, fails the plans of the
Column when they are built, as in PySpark, and its lone surrogates show escaped in the `__repr__`
Expand Down
25 changes: 16 additions & 9 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -2274,9 +2274,16 @@ that hold them, as `UserDefinedType`s of other parameters do, and `toNullable()`
are. The schema of a DataFrame and of its rows, as the server gives it, holds a new object of each
user-defined type, as PySpark's `UserDefinedType.fromJson` makes it of its class, and so does
`fromJson`: not the object that the client was given, nor of its attributes, and a class whose
`__init__` takes arguments raises its `TypeError` there, as in PySpark. A struct, array or map that
the client makes of core types, without the objects of a user, holds the user-defined type of its
JSON that a type was last made of, if one was. `IntegralType()` is one instance, as PySpark's
`__init__` takes arguments raises its `TypeError` wherever PySpark converts that schema, as in
PySpark: the schema, `columns`, `dtypes`, `printSchema`, the rows, the lookups that read the
columns (`df[0]`, `dir(df)`, names of two underscores), the copies and the unpickles, the map
functions and the numeric aggregations of grouped data. The cached schema of a DataFrame holds the
objects that PySpark's `_cached_schema` holds, those of `to` or of a map function, else the new
objects of the schema of the server: a set operation compares them as PySpark's does, by the
`__dict__` of their user-defined types, and a pickle carries them, which the unpickled DataFrame
keeps as its cached schema. `DataType.simpleString(t)`, the method of the base class, is
`t.typeName()`, as PySpark's, which the types of a string of their own, such as `bigint` or
`struct<a:bigint>`, override. `IntegralType()` is one instance, as PySpark's
`DataTypeSingleton` makes it, and so is each Python subclass of it, of an instance of its own, and a
Python subclass of the other classes, such as a `UserDefinedType` of parameters, takes the arguments
of its `__init__`.
Expand Down Expand Up @@ -2881,13 +2888,13 @@ session fails the copy with `SparkConnectException`, as PySpark's does. One diff
PySpark's `DataFrame.__getattr__` answers `getattr(df, "__deepcopy__")` and
`getattr(df, "__copy__")` by requesting the schema, to read the columns, and raises
`ATTRIBUTE_NOT_SUPPORTED`, so `hasattr(df, "__copy__")` is false; the drop-in defines both
itself, which it finds without a request. An unpickled DataFrame requests its
schema once, at the unpickle, which raises the error of the request, as PySpark's pickle looks the
itself, which it finds without a request. An unpickled DataFrame requests its schema once, at the
unpickle, which raises the error of the request or of its conversion, as PySpark's pickle looks the
`__setstate__` of the new DataFrame up through its `__getattr__`. Its cached schema is then that of
the pickled DataFrame, as the state of PySpark's pickle sets it: of none, its `schema` requests it
again. So does the unpickle of an object that PySpark pickles with its DataFrame: a writer, of its
callback, grouped data, `na` and `stat`; a stream writer, which PySpark's pickles of the plan of
the DataFrame, requests nothing.
the pickled DataFrame, as the state of PySpark's pickle sets it, of the objects that it carries: of
none, its `schema` requests it again. So does the unpickle of an object that PySpark pickles with
its DataFrame: a writer, of its callback, grouped data, `na` and `stat`; a stream writer, which
PySpark's pickles of the plan of the DataFrame, requests nothing.

A copy, a deep copy or an unpickle of a DataFrame keeps the schema that it requests apart from the
cached schema of the state, as PySpark's keeps it as the `_cached_schema_serialized` that its
Expand Down
Loading
Loading