Skip to content

[SPARK-62065][CONNECT][PYTHON] Hold PySpark's user-defined type objects in the cached schemas of set operations and pickles, raise the conversion errors of the schema of the server where PySpark reads it, and make DataType.simpleString the typeName - #878

Open
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62065-udt-leftovers

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Three changes that make the drop-in hold the user-defined type (UDT) objects that pyspark-client 4.3.0 holds, and raise where it raises. SPARK-62016 made the schemas of the server new UDT objects; these are the places that still gave the objects of a process-wide registry of the UDTs that the client lowered, or did not convert the schema at all.

1. The cached schemas of set operations and pickles (dataframe.rs). PySpark's _cached_schema of a DataFrame is the object of to or of a map function, or the type objects of its conversion of the schema of the server, of a new object of each UDT (UserDefinedType.fromJson).

  • set_operation_result compared the cached schema of a DataFrame of the schema of the server with data_type_to_py, which gave the UDT object that was lowered last of its JSON, often the user's own. PySpark's _merge_cached_schema compares it with ==, which compares the __dict__ of a UDT, so the object of to of a marked or parameterized UDT differs from the server's and the union caches none (python/pyspark/sql/connect/dataframe.py:1203-1208). Now the known schema is converted with schema_to_py (known_schema_object), and a schema that PySpark could not convert, as of a UDT whose __init__ needs arguments, counts as none, as PySpark's _schema cached none then.
  • __reduce__ put the registry objects into the _cached_schema of the state, and the unpickle (_of_relation) lowered the pickled object into a core type and dropped it. PySpark's state carries the objects (dataframe.py:150-170), which the unpickled DataFrame keeps as its _cached_schema: dtypes, columns and the methods that copy the cached schema, such as filter, read them, while schema is the one that the unpickle requested. Now __reduce__ gives the new objects of the server's schema, and the unpickle keeps the pickled StructType as the schema object of the DataFrame.

2. The reads of the schema of the server convert it (dataframe.rs, group.rs). PySpark's _schema converts the schema of the server, and every read of it raises the errors of that conversion: the TypeError of a UDT whose __init__ needs arguments, a KeyError of a UDT of no Python class, UNSUPPORTED_OPERATION of a nanosecond-precision timestamp (dataframe.py:2024-2038, connect/types.py:320-327, types.py:1986-2004). The drop-in raised them for schema, dtypes and the rows, but not for the reads that need only the names: columns, printSchema, df[0], dir(df), the lookup of a name of two underscores, the columns of mapInPandas and applyInPandas, the copies and unpickles (whose lookup of __setstate__ reads the columns), and the numeric aggregations of grouped data (self._df.schema, group.py:160). Now check_python_schema raises them there: own_schema, fresh_schema_request (the copies), _of_relation (the unpickle) and GroupedData's numeric aggregation.

3. DataType.simpleString is self.typeName() (types.rs). PySpark's base method returns the typeName() (types.py:134-135), which the types of a string of their own override. The drop-in's gave the simple string of the type, of the registry's UDT objects for a struct, array or map, so DataType.simpleString(st) was struct<a:scale2> where PySpark's is struct, and a subclass of another typeName() was not named by it. YearMonthIntervalType and DayTimeIntervalType relied on the base method, and now have their own simpleString, PySpark's _str_repr.

Why are the changes needed?

So that the cached schemas, the pickles and the reads of the schema are PySpark's. Before:

  • to(st).union(server_df).schema kept the object of to of a UDT of other attributes than the server's, and a pickled to(st) lost its objects, so unpickled.dtypes and unpickled.filter(...).schema were of new ones.
  • df.columns, copy.copy(df) and df.groupBy().sum() of a DataFrame of a UDT whose __init__ needs arguments, or of a nanosecond timestamp, did not raise.
  • DataType.simpleString(st) was not typeName().

Does this PR introduce any user-facing change?

Yes, as above, as pyspark-client 4.3.0 does. A schema of the server that PySpark cannot convert is still requested once and its conversion raises at each read, where PySpark's requests it again at each read; this is noted in dev/parity/known-differences.md.

How was this patch tested?

  • New offline tests (test_dropin_offline.py), each failing on master:
    • test_the_base_simple_string_of_a_type_is_its_type_name_as_pysparks and test_the_simple_strings_of_subclasses_of_their_own_type_name_are_pysparks;
    • test_reads_of_a_schema_of_a_udt_of_constructor_arguments_raise_as_pysparks and test_cached_schemas_of_set_operations_and_pickles_are_pysparks_objects, against the mock server, of a schema of a UDT of the test module.
  • test_print_schema_udt_class_name[None-udt] asserted that the schema of a UDT of no Python class prints udt; PySpark's fromJson raises the KeyError of pyClass there, which it asserts now. test_structtype_add_reflected_in_the_base_data_type reads the struct's own simpleString and JSON.
  • New e2e tests (test_dropin_e2e.py), whose bodies pass on pyspark-client 4.3.0 and which fail on master: test_cached_schemas_of_set_operations_and_pickles_hold_the_objects_of_pyspark, test_reads_of_a_schema_of_a_user_defined_type_of_arguments_raise_as_pyspark, and the copies and groupBy().sum() of test_nanosecond_timestamps_are_not_collected_as_pyspark.
  • Probes of both clients against a Spark 4.3.0 server (the union, intersect and except of to and server DataFrames, copies, deep copies and cloudpickles, fromJson, pickles and copies of types, UDF return types, and 28 uses of a DataFrame of a UDT of constructor arguments): every case is the same.
  • The Rust gates:
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;
    • cargo test -p pyspark-rs --lib as rust.yml runs it: 15 passed.
  • The Python gates:
    • ruff 0.14.0 check and format --check, test_ruff_excludes.py and scripts/audit_no_stubs.sh;
    • gen_type_stubs.py --check and gen_fn_dispatch.py --check;
    • test_type_stubs_offline.py: 13 passed, 3 skipped;
    • the guards of SPARK-61851.
  • The offline suite of coverage.yml: 6526 passed, 77 skipped. Under coverage, with its .coveragerc: 93.44%.
  • test_dropin_e2e.py against a Spark 4.3.0 server: 335 passed.
  • The official suites of build_python_connect.yml in drop-in mode (run_official_tests.py --dropin --only ...) of the files that the change reaches, on master and then on this branch, the same on both: test_parity_types.py 70 passed (21 skipped), test_parity_dataframe.py 53 (4), test_parity_group.py 8, test_parity_serde.py 12 (2), test_parity_creation.py 18 (1), test_parity_geometrytype.py 9, test_parity_geographytype.py 10, arrow/test_parity_arrow.py 93 (7) and pandas/test_parity_pandas_map.py 30 (1), of no failures; test_connect_dataframe_property.py, test_connect_creation.py and test_connect_basic.py skip every test in drop-in mode.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…ts in the cached schemas of set operations and pickles, raise the conversion errors of the schema of the server where PySpark reads it, and make DataType.simpleString the typeName

PySpark's _cached_schema of a DataFrame is the object of to() or of a map function, or the type
objects of its conversion of the schema of the server, of a new object of each user-defined type
(UserDefinedType.fromJson). The drop-in still gave the objects of its registry of lowered UDTs,
by their JSON, in three places, or did not convert the schema:
- set_operation_result compared a cached schema of the server's with data_type_to_py, the UDT
  that was lowered last, often the user's own; PySpark's _merge_cached_schema compares the new
  objects with ==, of their __dict__, so a union of to() of a marked UDT and a DataFrame of the
  server caches none. Now the known schema converts with schema_to_py (known_schema_object), and
  one that PySpark could not convert counts as none.
- __reduce__ pickled the registry's objects, and the unpickle (_of_relation) lowered the pickled
  StructType and dropped it. Now the state carries the new objects of the server's schema, and
  the unpickled DataFrame keeps the pickled StructType as its schema object, which dtypes,
  columns and filter() read, while schema is the one that its unpickle requested.
- The reads of the schema that need only the names (columns, printSchema, df[0], dir(), a lookup
  of a name of two underscores, the columns of mapInPandas and applyInPandas, the copies and
  unpickles, and the numeric aggregations of grouped data) did not convert it, so a UDT whose
  __init__ needs arguments, a UDT of no Python class or a nanosecond timestamp did not raise
  there. Now check_python_schema raises PySpark's errors in own_schema, fresh_schema_request,
  _of_relation and GroupedData's numeric aggregation.

DataType.simpleString is self.typeName(), as PySpark's base method, which the types of a string
of their own override; YearMonthIntervalType and DayTimeIntervalType have their own, PySpark's
_str_repr. The drop-in's gave the simple string of the type, of the registry's UDTs.

A schema that PySpark cannot convert is still requested once (dev/parity/known-differences.md).

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62065-udt-leftovers branch from 7da3e56 to b1cb34c Compare November 19, 2026 05:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants