Skip to content

[SPARK-62066][CONNECT][PYTHON] Take local rows apart by the cached names of a StructType changed in place, raise PySpark's errors of the types that proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED conditions of no parameters - #879

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62066-types-leftovers

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Three changes that make the drop-in raise and convert as pyspark-client 4.3.0 does.

1. A StructType whose fields list was changed in place (infer.rs, session.rs, _pandas_input.py). PySpark's StructType caches names as it is made and as add adds a field (python/pyspark/sql/types.py:1471, 1541, 1557), and fieldNames() is list(self.names) (types.py:1761). So an in-place change of fields, such as a StructField appended, popped or replaced, leaves the names as they were. Its conversion of local rows reads both:

  • to_row takes each row apart by schema.fieldNames(): the arity of a tuple, a list or a Row, and the keys of a dict or of an object's __dict__ (python/pyspark/sql/conversion.py:941-975);
  • the values convert by position with the fields as they are, [conv(row[i]) for row in rows] (conversion.py:978-1003), so a field past the names has no value (IndexError: tuple index out of range) and a value past the fields is not read;
  • below the top level, the converter of a struct takes its values apart by the struct's fieldNames() too, of field_convs[i] by position (conversion.py:605-660), so a value past the fields raises IndexError: list index out of range, and the Arrow schema of fewer names than fields fails _deduplicate_field_names (python/pyspark/sql/pandas/types.py:1630-1653) with the same error, once every value is converted;
  • a pandas DataFrame is named with toDF(*schema.fieldNames()) (python/pyspark/sql/connect/session.py:608, 793), of which the server raises ASSIGNMENT_ARITY_MISMATCH for another number of names than columns.

The drop-in took the rows apart by the names of the fields as they are. Now:

  • row_values and the struct converter take the cached names when they differ from the fields (infer::cached_names), which TypeObjects keeps of each struct below the top level, and convert by position;
  • infer::dedup_names_error raises the IndexError of fewer names than fields after the values convert, before the Arrow schema;
  • the integer fast path of createDataFrame declines such a schema;
  • the pandas path renames the table by as many of the names as the fields, and applies toDF of fieldNames().

An element of fields that is not a StructField, such as a str appended, still raises the drop-in's TypeError wherever the native type lowers its fields. PySpark's Python methods raise the AttributeError of the first attribute that they read. This is inherent to the native type, whose uses lower its fields into the core's struct type, and is noted in dev/parity/known-differences.md.

2. proto_schema_to_pyspark_data_type of a type that it does not convert (connect_bridge.rs). PySpark's last branch raises PySparkValueError UNSUPPORTED_OPERATION of f"data type {schema}", the proto text of the type (python/pyspark/sql/connect/types.py:218-333). It does so for a type of no kind, such as pb2.DataType() or the element type that an array does not set, and for a kind that it has no branch of: an unparsed type or a nanosecond-precision timestamp. The first such type in the order of its conversion decides, and a spatial type of an SRID that it does not take raises ST_INVALID_SRID_VALUE. A user-defined type is made of its Python class, of no conversion of its SQL type.

The drop-in raised the core's PySparkRuntimeError ("DataType kind not set", "Array element_type is missing") for a type of no kind, gave an UnparsedDataType of an unparsed type, and raised for a nanosecond timestamp in the SQL type of a user-defined type. Now _data_type_of_proto finds the first such type (unconverted) and raises PySpark's error, of the text of the message of the drop-in's protobuf runtime. It drops the SQL type of a user-defined type that holds a type of no kind, which the core does not read.

3. The MALFORMED conditions of the conversion of local rows (infer.rs, errors.rs). PySpark's converters raise PySparkValueError(errorClass="MALFORMED_VARIANT") (conversion.py:889), and MALFORMED_GEOGRAPHY (903) and MALFORMED_GEOMETRY (917), without messageParameters, so getMessageParameters() is None. The drop-in's were {}. Now they are raised of the condition alone (errors::raise_pyspark_condition). An audit of the other conditions that the drop-in raises of no parameters finds PySpark passing messageParameters={} for each of them.

Why are the changes needed?

So that createDataFrame, proto_schema_to_pyspark_data_type and these errors are PySpark's. Before:

  • createDataFrame([(1, "q")], st) of a StructType that a field was appended to in place made a DataFrame, where PySpark raises AXIS_LENGTH_MISMATCH, and a dict row of a replaced field was null.
  • proto_schema_to_pyspark_data_type(pb2.DataType()) raised a PySparkRuntimeError of no condition.
  • e.getMessageParameters() of MALFORMED_VARIANT was {}.

Does this PR introduce any user-facing change?

Yes, as above, as pyspark-client 4.3.0 does.

How was this patch tested?

  • New offline tests, each failing on master:
    • test_local_rows_of_a_struct_changed_in_place_are_taken_apart_by_its_cached_names and test_pandas_frames_of_a_struct_changed_in_place_are_named_by_its_cached_names (test_dropin_offline.py), against the mock server;
    • test_proto_schema_conversion_raises_pysparks_errors_of_types_that_it_does_not_convert (test_dropin_types_columns_offline.py);
    • test_variant_spatial_and_user_defined_values_are_sent_as_pyspark checks the parameters, also nested.
  • A new e2e test, test_create_dataframe_of_a_struct_changed_in_place_reads_its_cached_names_as_pyspark (test_dropin_e2e.py). Its body passes on pyspark-client 4.3.0, and it fails on master.
  • Probes of both clients against a Spark 4.3.0 server, run on master and then on this branch:
    • 132 uses of four kinds of changed StructTypes;
    • 24 cases of nested structs and other kinds of rows;
    • 11 cases of pandas DataFrames, pyarrow Tables, NumPy arrays and empty data;
    • 24 cases of proto types;
    • 17 cases of malformed values.
      Every case is now the same, but for those of an element that is not a StructField (above) and of a pandas DataFrame of another width than the fields, for which the drop-in raises AXIS_LENGTH_MISMATCH (known, SPARK-60902).
  • The Rust gates:
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;
    • cargo test -p pyspark-rs --lib as rust.yml runs it: 15 passed.
  • The Python gates:
    • ruff 0.14.0 check and format --check, test_ruff_excludes.py and scripts/audit_no_stubs.sh;
    • gen_type_stubs.py --check and gen_fn_dispatch.py --check;
    • test_type_stubs_offline.py: 13 passed, 3 skipped;
    • the guards of SPARK-61851.
  • The offline suite of coverage.yml: 6503 passed, 77 skipped. Under coverage, with its .coveragerc: 93.44%.
  • test_dropin_e2e.py against a Spark 4.3.0 server: 334 passed.
  • The official suites of build_python_connect.yml in drop-in mode (run_official_tests.py --dropin --only ...) of the files that the change reaches, on master and then on this branch, the same on both: test_parity_types.py 70 passed (21 skipped), test_parity_creation.py 18 (1), arrow/test_parity_arrow.py 93 (7), test_parity_geometrytype.py 9, test_parity_geographytype.py 10, test_parity_dataframe.py 53 (4), test_parity_group.py 8, test_parity_serde.py 12 (2) and pandas/test_parity_pandas_map.py 30 (1), of no failures; test_connect_creation.py, test_connect_basic.py and test_connect_dataframe_property.py skip every test in drop-in mode.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…mes of a StructType changed in place, raise PySpark's errors of the types that proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED conditions of no parameters, as PySpark

PySpark's StructType caches its names as it is made and as add() adds a field, so an in-place
change of its fields list, such as a StructField appended, popped or replaced, leaves names and
fieldNames() as they were. Its conversion of local rows takes each row, and each struct value
below the top level, apart by fieldNames(): the arity of a row and the keys of a dict are of the
cached names, and the values convert by position with the fields as they are, so a field past them
has no value (IndexError of row[i]); below the top level, a value past the fields has no
converter, and fewer names than fields fail _deduplicate_field_names, both IndexError. A pandas
DataFrame is named with toDF(*schema.fieldNames()). The drop-in took them apart by the names of the
fields as they are. Now:
- row_values and the struct converter take the cached names (infer::cached_names), which
  TypeObjects keeps of each struct below the top level, and convert by position;
- infer::dedup_names_error raises the IndexError of fewer names than fields once the values are
  converted, before the Arrow schema;
- the integer fast path of createDataFrame declines such a schema;
- the pandas path renames the table by as many of the names as the fields, and applies toDF of
  fieldNames().
An element that is not a StructField still raises the drop-in's TypeError where the struct is
lowered (dev/parity/known-differences.md).

proto_schema_to_pyspark_data_type of a type of no kind, such as pb2.DataType() or an element type
that an array does not set, raised a PySparkRuntimeError of the core ("DataType kind not set"),
and of an unparsed type gave an UnparsedDataType. PySpark's last branch raises
UNSUPPORTED_OPERATION of f"data type {schema}", of the proto text of the type, also of a
nanosecond-precision timestamp, in the order of its conversion, of which a spatial type of an
SRID that it does not take raises ST_INVALID_SRID_VALUE; it makes a user-defined type of its
Python class, of no conversion of its SQL type. Now _data_type_of_proto finds the first such type
in that order (connect_bridge::unconverted) and raises PySpark's error, of the text of the message
of the drop-in's protobuf runtime, and drops the SQL type of a user-defined type that holds a type
of no kind.

MALFORMED_VARIANT, MALFORMED_GEOMETRY and MALFORMED_GEOGRAPHY of the conversion of local rows are
raised of the condition alone (errors::raise_pyspark_condition), as PySpark's
PySparkValueError(errorClass=...) of its converters, whose getMessageParameters() is None, where
the drop-in's was {}.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants