Repository navigation
[SPARK-62065][CONNECT][PYTHON] Hold PySpark's user-defined type objects in the cached schemas of set operations and pickles, raise the conversion errors of the schema of the server where PySpark reads it, and make DataType.simpleString the typeName - #878
Open
HyukjinKwon wants to merge 1 commit into
Open
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
dongjoon-hyun
approved these changes
Nov 19, 2026
…ts in the cached schemas of set operations and pickles, raise the conversion errors of the schema of the server where PySpark reads it, and make DataType.simpleString the typeName PySpark's _cached_schema of a DataFrame is the object of to() or of a map function, or the type objects of its conversion of the schema of the server, of a new object of each user-defined type (UserDefinedType.fromJson). The drop-in still gave the objects of its registry of lowered UDTs, by their JSON, in three places, or did not convert the schema: - set_operation_result compared a cached schema of the server's with data_type_to_py, the UDT that was lowered last, often the user's own; PySpark's _merge_cached_schema compares the new objects with ==, of their __dict__, so a union of to() of a marked UDT and a DataFrame of the server caches none. Now the known schema converts with schema_to_py (known_schema_object), and one that PySpark could not convert counts as none. - __reduce__ pickled the registry's objects, and the unpickle (_of_relation) lowered the pickled StructType and dropped it. Now the state carries the new objects of the server's schema, and the unpickled DataFrame keeps the pickled StructType as its schema object, which dtypes, columns and filter() read, while schema is the one that its unpickle requested. - The reads of the schema that need only the names (columns, printSchema, df[0], dir(), a lookup of a name of two underscores, the columns of mapInPandas and applyInPandas, the copies and unpickles, and the numeric aggregations of grouped data) did not convert it, so a UDT whose __init__ needs arguments, a UDT of no Python class or a nanosecond timestamp did not raise there. Now check_python_schema raises PySpark's errors in own_schema, fresh_schema_request, _of_relation and GroupedData's numeric aggregation. DataType.simpleString is self.typeName(), as PySpark's base method, which the types of a string of their own override; YearMonthIntervalType and DayTimeIntervalType have their own, PySpark's _str_repr. The drop-in's gave the simple string of the type, of the registry's UDTs. A schema that PySpark cannot convert is still requested once (dev/parity/known-differences.md). Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62065-udt-leftovers
branch
from
November 19, 2026 05:58
7da3e56 to
b1cb34c
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Three changes that make the drop-in hold the user-defined type (UDT) objects that pyspark-client 4.3.0 holds, and raise where it raises. SPARK-62016 made the schemas of the server new UDT objects; these are the places that still gave the objects of a process-wide registry of the UDTs that the client lowered, or did not convert the schema at all.
1. The cached schemas of set operations and pickles (
dataframe.rs). PySpark's_cached_schemaof a DataFrame is the object oftoor of a map function, or the type objects of its conversion of the schema of the server, of a new object of each UDT (UserDefinedType.fromJson).set_operation_resultcompared the cached schema of a DataFrame of the schema of the server withdata_type_to_py, which gave the UDT object that was lowered last of its JSON, often the user's own. PySpark's_merge_cached_schemacompares it with==, which compares the__dict__of a UDT, so the object oftoof a marked or parameterized UDT differs from the server's and the union caches none (python/pyspark/sql/connect/dataframe.py:1203-1208). Now the known schema is converted withschema_to_py(known_schema_object), and a schema that PySpark could not convert, as of a UDT whose__init__needs arguments, counts as none, as PySpark's_schemacached none then.__reduce__put the registry objects into the_cached_schemaof the state, and the unpickle (_of_relation) lowered the pickled object into a core type and dropped it. PySpark's state carries the objects (dataframe.py:150-170), which the unpickled DataFrame keeps as its_cached_schema:dtypes,columnsand the methods that copy the cached schema, such asfilter, read them, whileschemais the one that the unpickle requested. Now__reduce__gives the new objects of the server's schema, and the unpickle keeps the pickledStructTypeas the schema object of the DataFrame.2. The reads of the schema of the server convert it (
dataframe.rs,group.rs). PySpark's_schemaconverts the schema of the server, and every read of it raises the errors of that conversion: theTypeErrorof a UDT whose__init__needs arguments, aKeyErrorof a UDT of no Python class,UNSUPPORTED_OPERATIONof a nanosecond-precision timestamp (dataframe.py:2024-2038,connect/types.py:320-327,types.py:1986-2004). The drop-in raised them forschema,dtypesand the rows, but not for the reads that need only the names:columns,printSchema,df[0],dir(df), the lookup of a name of two underscores, the columns ofmapInPandasandapplyInPandas, the copies and unpickles (whose lookup of__setstate__reads the columns), and the numeric aggregations of grouped data (self._df.schema,group.py:160). Nowcheck_python_schemaraises them there:own_schema,fresh_schema_request(the copies),_of_relation(the unpickle) andGroupedData's numeric aggregation.3.
DataType.simpleStringisself.typeName()(types.rs). PySpark's base method returns thetypeName()(types.py:134-135), which the types of a string of their own override. The drop-in's gave the simple string of the type, of the registry's UDT objects for a struct, array or map, soDataType.simpleString(st)wasstruct<a:scale2>where PySpark's isstruct, and a subclass of anothertypeName()was not named by it.YearMonthIntervalTypeandDayTimeIntervalTyperelied on the base method, and now have their ownsimpleString, PySpark's_str_repr.Why are the changes needed?
So that the cached schemas, the pickles and the reads of the schema are PySpark's. Before:
to(st).union(server_df).schemakept the object oftoof a UDT of other attributes than the server's, and a pickledto(st)lost its objects, sounpickled.dtypesandunpickled.filter(...).schemawere of new ones.df.columns,copy.copy(df)anddf.groupBy().sum()of a DataFrame of a UDT whose__init__needs arguments, or of a nanosecond timestamp, did not raise.DataType.simpleString(st)was nottypeName().Does this PR introduce any user-facing change?
Yes, as above, as pyspark-client 4.3.0 does. A schema of the server that PySpark cannot convert is still requested once and its conversion raises at each read, where PySpark's requests it again at each read; this is noted in
dev/parity/known-differences.md.How was this patch tested?
test_dropin_offline.py), each failing on master:test_the_base_simple_string_of_a_type_is_its_type_name_as_pysparksandtest_the_simple_strings_of_subclasses_of_their_own_type_name_are_pysparks;test_reads_of_a_schema_of_a_udt_of_constructor_arguments_raise_as_pysparksandtest_cached_schemas_of_set_operations_and_pickles_are_pysparks_objects, against the mock server, of a schema of a UDT of the test module.test_print_schema_udt_class_name[None-udt]asserted that the schema of a UDT of no Python class printsudt; PySpark'sfromJsonraises theKeyErrorofpyClassthere, which it asserts now.test_structtype_add_reflected_in_the_base_data_typereads the struct's ownsimpleStringand JSON.test_dropin_e2e.py), whose bodies pass on pyspark-client 4.3.0 and which fail on master:test_cached_schemas_of_set_operations_and_pickles_hold_the_objects_of_pyspark,test_reads_of_a_schema_of_a_user_defined_type_of_arguments_raise_as_pyspark, and the copies andgroupBy().sum()oftest_nanosecond_timestamps_are_not_collected_as_pyspark.toand server DataFrames, copies, deep copies and cloudpickles,fromJson, pickles and copies of types, UDF return types, and 28 uses of a DataFrame of a UDT of constructor arguments): every case is the same.cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;cargo test -p pyspark-rs --libasrust.ymlruns it: 15 passed.checkandformat --check,test_ruff_excludes.pyandscripts/audit_no_stubs.sh;gen_type_stubs.py --checkandgen_fn_dispatch.py --check;test_type_stubs_offline.py: 13 passed, 3 skipped;coverage.yml: 6526 passed, 77 skipped. Under coverage, with its.coveragerc: 93.44%.test_dropin_e2e.pyagainst a Spark 4.3.0 server: 335 passed.build_python_connect.ymlin drop-in mode (run_official_tests.py --dropin --only ...) of the files that the change reaches, on master and then on this branch, the same on both:test_parity_types.py70 passed (21 skipped),test_parity_dataframe.py53 (4),test_parity_group.py8,test_parity_serde.py12 (2),test_parity_creation.py18 (1),test_parity_geometrytype.py9,test_parity_geographytype.py10,arrow/test_parity_arrow.py93 (7) andpandas/test_parity_pandas_map.py30 (1), of no failures;test_connect_dataframe_property.py,test_connect_creation.pyandtest_connect_basic.pyskip every test in drop-in mode.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.