Repository navigation
[SPARK-62066][CONNECT][PYTHON] Take local rows apart by the cached names of a StructType changed in place, raise PySpark's errors of the types that proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED conditions of no parameters - #879
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…mes of a StructType changed in place, raise PySpark's errors of the types that proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED conditions of no parameters, as PySpark
PySpark's StructType caches its names as it is made and as add() adds a field, so an in-place
change of its fields list, such as a StructField appended, popped or replaced, leaves names and
fieldNames() as they were. Its conversion of local rows takes each row, and each struct value
below the top level, apart by fieldNames(): the arity of a row and the keys of a dict are of the
cached names, and the values convert by position with the fields as they are, so a field past them
has no value (IndexError of row[i]); below the top level, a value past the fields has no
converter, and fewer names than fields fail _deduplicate_field_names, both IndexError. A pandas
DataFrame is named with toDF(*schema.fieldNames()). The drop-in took them apart by the names of the
fields as they are. Now:
- row_values and the struct converter take the cached names (infer::cached_names), which
TypeObjects keeps of each struct below the top level, and convert by position;
- infer::dedup_names_error raises the IndexError of fewer names than fields once the values are
converted, before the Arrow schema;
- the integer fast path of createDataFrame declines such a schema;
- the pandas path renames the table by as many of the names as the fields, and applies toDF of
fieldNames().
An element that is not a StructField still raises the drop-in's TypeError where the struct is
lowered (dev/parity/known-differences.md).
proto_schema_to_pyspark_data_type of a type of no kind, such as pb2.DataType() or an element type
that an array does not set, raised a PySparkRuntimeError of the core ("DataType kind not set"),
and of an unparsed type gave an UnparsedDataType. PySpark's last branch raises
UNSUPPORTED_OPERATION of f"data type {schema}", of the proto text of the type, also of a
nanosecond-precision timestamp, in the order of its conversion, of which a spatial type of an
SRID that it does not take raises ST_INVALID_SRID_VALUE; it makes a user-defined type of its
Python class, of no conversion of its SQL type. Now _data_type_of_proto finds the first such type
in that order (connect_bridge::unconverted) and raises PySpark's error, of the text of the message
of the drop-in's protobuf runtime, and drops the SQL type of a user-defined type that holds a type
of no kind.
MALFORMED_VARIANT, MALFORMED_GEOMETRY and MALFORMED_GEOGRAPHY of the conversion of local rows are
raised of the condition alone (errors::raise_pyspark_condition), as PySpark's
PySparkValueError(errorClass=...) of its converters, whose getMessageParameters() is None, where
the drop-in's was {}.
Co-authored-by: Isaac <no-reply@databricks.com>
dongjoon-hyun
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Three changes that make the drop-in raise and convert as pyspark-client 4.3.0 does.
1. A StructType whose
fieldslist was changed in place (infer.rs,session.rs,_pandas_input.py). PySpark'sStructTypecachesnamesas it is made and asaddadds a field (python/pyspark/sql/types.py:1471,1541,1557), andfieldNames()islist(self.names)(types.py:1761). So an in-place change offields, such as aStructFieldappended, popped or replaced, leaves the names as they were. Its conversion of local rows reads both:to_rowtakes each row apart byschema.fieldNames(): the arity of a tuple, a list or aRow, and the keys of a dict or of an object's__dict__(python/pyspark/sql/conversion.py:941-975);[conv(row[i]) for row in rows](conversion.py:978-1003), so a field past the names has no value (IndexError: tuple index out of range) and a value past the fields is not read;fieldNames()too, offield_convs[i]by position (conversion.py:605-660), so a value past the fields raisesIndexError: list index out of range, and the Arrow schema of fewer names than fields fails_deduplicate_field_names(python/pyspark/sql/pandas/types.py:1630-1653) with the same error, once every value is converted;toDF(*schema.fieldNames())(python/pyspark/sql/connect/session.py:608,793), of which the server raisesASSIGNMENT_ARITY_MISMATCHfor another number of names than columns.The drop-in took the rows apart by the names of the fields as they are. Now:
row_valuesand the struct converter take the cached names when they differ from the fields (infer::cached_names), whichTypeObjectskeeps of each struct below the top level, and convert by position;infer::dedup_names_errorraises theIndexErrorof fewer names than fields after the values convert, before the Arrow schema;createDataFramedeclines such a schema;toDFoffieldNames().An element of
fieldsthat is not aStructField, such as a str appended, still raises the drop-in'sTypeErrorwherever the native type lowers its fields. PySpark's Python methods raise theAttributeErrorof the first attribute that they read. This is inherent to the native type, whose uses lower its fields into the core's struct type, and is noted indev/parity/known-differences.md.2.
proto_schema_to_pyspark_data_typeof a type that it does not convert (connect_bridge.rs). PySpark's last branch raisesPySparkValueErrorUNSUPPORTED_OPERATIONoff"data type {schema}", the proto text of the type (python/pyspark/sql/connect/types.py:218-333). It does so for a type of no kind, such aspb2.DataType()or the element type that an array does not set, and for a kind that it has no branch of: an unparsed type or a nanosecond-precision timestamp. The first such type in the order of its conversion decides, and a spatial type of an SRID that it does not take raisesST_INVALID_SRID_VALUE. A user-defined type is made of its Python class, of no conversion of its SQL type.The drop-in raised the core's
PySparkRuntimeError("DataType kind not set", "Array element_type is missing") for a type of no kind, gave anUnparsedDataTypeof an unparsed type, and raised for a nanosecond timestamp in the SQL type of a user-defined type. Now_data_type_of_protofinds the first such type (unconverted) and raises PySpark's error, of the text of the message of the drop-in's protobuf runtime. It drops the SQL type of a user-defined type that holds a type of no kind, which the core does not read.3. The MALFORMED conditions of the conversion of local rows (
infer.rs,errors.rs). PySpark's converters raisePySparkValueError(errorClass="MALFORMED_VARIANT")(conversion.py:889), andMALFORMED_GEOGRAPHY(903) andMALFORMED_GEOMETRY(917), withoutmessageParameters, sogetMessageParameters()is None. The drop-in's were{}. Now they are raised of the condition alone (errors::raise_pyspark_condition). An audit of the other conditions that the drop-in raises of no parameters finds PySpark passingmessageParameters={}for each of them.Why are the changes needed?
So that
createDataFrame,proto_schema_to_pyspark_data_typeand these errors are PySpark's. Before:createDataFrame([(1, "q")], st)of aStructTypethat a field was appended to in place made a DataFrame, where PySpark raisesAXIS_LENGTH_MISMATCH, and a dict row of a replaced field was null.proto_schema_to_pyspark_data_type(pb2.DataType())raised aPySparkRuntimeErrorof no condition.e.getMessageParameters()ofMALFORMED_VARIANTwas{}.Does this PR introduce any user-facing change?
Yes, as above, as pyspark-client 4.3.0 does.
How was this patch tested?
test_local_rows_of_a_struct_changed_in_place_are_taken_apart_by_its_cached_namesandtest_pandas_frames_of_a_struct_changed_in_place_are_named_by_its_cached_names(test_dropin_offline.py), against the mock server;test_proto_schema_conversion_raises_pysparks_errors_of_types_that_it_does_not_convert(test_dropin_types_columns_offline.py);test_variant_spatial_and_user_defined_values_are_sent_as_pysparkchecks the parameters, also nested.test_create_dataframe_of_a_struct_changed_in_place_reads_its_cached_names_as_pyspark(test_dropin_e2e.py). Its body passes on pyspark-client 4.3.0, and it fails on master.StructTypes;Every case is now the same, but for those of an element that is not a
StructField(above) and of a pandas DataFrame of another width than the fields, for which the drop-in raisesAXIS_LENGTH_MISMATCH(known, SPARK-60902).cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;cargo test -p pyspark-rs --libasrust.ymlruns it: 15 passed.checkandformat --check,test_ruff_excludes.pyandscripts/audit_no_stubs.sh;gen_type_stubs.py --checkandgen_fn_dispatch.py --check;test_type_stubs_offline.py: 13 passed, 3 skipped;coverage.yml: 6503 passed, 77 skipped. Under coverage, with its.coveragerc: 93.44%.test_dropin_e2e.pyagainst a Spark 4.3.0 server: 334 passed.build_python_connect.ymlin drop-in mode (run_official_tests.py --dropin --only ...) of the files that the change reaches, on master and then on this branch, the same on both:test_parity_types.py70 passed (21 skipped),test_parity_creation.py18 (1),arrow/test_parity_arrow.py93 (7),test_parity_geometrytype.py9,test_parity_geographytype.py10,test_parity_dataframe.py53 (4),test_parity_group.py8,test_parity_serde.py12 (2) andpandas/test_parity_pandas_map.py30 (1), of no failures;test_connect_creation.py,test_connect_basic.pyandtest_connect_dataframe_property.pyskip every test in drop-in mode.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.