Skip to content

[SPARK-62061][CONNECT][PYTHON] Read a path of addArtifacts that is not a str as PySpark's ArtifactManager, and define the client's results in pyspark.sql.connect.client.core with PySpark's import * of its package - #875

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62061-client-leftovers

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Two follow-ups of SPARK-62023 (#863), each matching pyspark-client 4.3.0. Details are in jira-SPARK-62061-client-leftovers.txt.

1. A path of addArtifacts that is not a str. PySpark's client reads, checks and uploads its paths one after the other (connect/client/core.py:2461-2470). Its ArtifactManager._parse_artifacts parses each path with urlparse (artifact.py:184-203), after SparkSession.addArtifacts has checked its flags. The drop-in took the paths as a list of str, so any other object raised PyO3's TypeError. That happened before the flags were checked and before any path was uploaded. Now:

  • the binding reads each path of addArtifacts, addArtifact and client.add_artifacts as PySpark's ArtifactManager does:
    • a str is taken as it is;
    • a Windows absolute path of another object becomes the file URI that PureWindowsPath and Path.resolve make of it;
    • any other object raises the error of urlparse itself, such as the AttributeError of a pathlib.Path or an int, or UNSUPPORTED_OPERATION of a scheme of bytes, for bytes or a falsy object;
  • each path is uploaded before the next is read.

2. The names of pyspark.sql.connect.client. PySpark's package is the import * of its core plus getLogLevel, with no __all__. The drop-in's package is the module of its client. Now:

  • AnalyzeResult, ConfigResult and PlanObservedMetrics are defined in pyspark.sql.connect.client.core, as PySpark's are, and not in the package. The package imports them under private names.
  • The core module also has the metrics classes of PySpark's core.
  • The package's __all__ gives what PySpark's import * of it gives: the four classes, core and getLogLevel. The gRPC modules are the exception: artifact and reattach are replaced by the transport, and retries imports grpc, so it is imported only when read.
  • The package sets the four classes in its core once it has defined them. The binding reads PlanObservedMetrics there.
  • The rest of the package's namespace is the module of the client until SparkConnectClient moves into client/core.py, which a later change does.

Also checked, with no change needed. Both clients behave the same on master for:

docs/compatibility.md now notes one consequence of the native singleton types. An error that names the type of such a class, as the NOT_EXPECTED_TYPE of udf(f, IntegerType), names type, where PySpark's names DataTypeSingleton.

Why are the changes needed?

So that the paths of addArtifacts and the names of pyspark.sql.connect.client are PySpark Connect's. A program that passes a pathlib.Path gets PySpark's error, in PySpark's order, after the paths before it are uploaded. A program that imports from the package gets PySpark's names, and the result classes only from core.

Does this PR introduce any user-facing change?

Yes, as pyspark-client 4.3.0:

  • spark.addArtifacts(pathlib.Path(...)) raises AttributeError: 'PosixPath' object has no attribute 'decode'.
  • Bytes, None and 0 raise UNSUPPORTED_OPERATION of b'' scheme.
  • The flags are checked first, and the paths before a bad one are uploaded.
  • from pyspark.sql.connect.client import AnalyzeResult raises ImportError.
  • from pyspark.sql.connect.client import * gives getLogLevel and core too.

How was this patch tested?

  • New and changed tests in test_dropin_offline.py, both of which fail on master:
    • test_add_artifacts_reads_a_path_that_is_not_a_str_as_pyspark, on the mock server, which now records the first request of each AddArtifacts;
    • test_the_classes_of_the_client_are_of_its_core_module_as_pyspark.
  • Probes of both clients against a Spark 4.3.0 server:
    • p12_artifact_paths.py: identical, 15 of 15 lines.
    • p13_client_names.py: the result classes, import *, core's names and getLogLevel are identical. The rest of the package's namespace and its __all__ differ, as described above.
  • The official test_client, test_artifact, test_parity_observation, test_connect_session and test_connect_clone_session were run against the drop-in (run_official_tests.py --dropin --only). The results are the same on master and on this branch.
  • The gates:
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps of the four crates --features apache-spark-connect/wasm-udf;
    • cargo test -p pyspark-rs --lib: 15 passed;
    • ruff 0.14.0, the stub and dispatch generators, the guards and test_type_stubs_offline.py (13 passed, 3 skipped);
    • the offline suite of coverage.yml: 6495 passed, 77 skipped;
    • the coverage step of coverage.yml, with its commands and its .coveragerc: 93.43% (floor 93.00%), with no new line of the change uncovered;
    • test_dropin_e2e.py against a Spark 4.3.0 server: 333 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…t a str as PySpark's ArtifactManager, and define the client's results in pyspark.sql.connect.client.core with PySpark's import * of its package

PySpark's SparkConnectClient.add_artifacts (connect/client/core.py:2461-2470) reads, checks and uploads its paths
one after the other, and its ArtifactManager parses each with urlparse (artifact.py:184-203), after
SparkSession.addArtifacts has checked its flags (connect/session.py:1110-1121). So a path that is not a str raises
urlparse's error: the AttributeError of the `decode` that a pathlib.Path or an int does not have, or, of bytes or of
a falsy object, which urlparse parses into a scheme of bytes, UNSUPPORTED_OPERATION of that scheme. The drop-in's
addArtifacts, addArtifact and client.add_artifacts took the paths as a list of str, so any other object raised
PyO3's TypeError before the flags were checked and before the paths before it were uploaded.

PySpark's pyspark.sql.connect.client package is `from pyspark.sql.connect.client.core import *` and its getLogLevel,
with no `__all__`: its `import *` gives the four classes of the `__all__` of its core, getLogLevel and its modules
core, artifact, reattach and retries. AnalyzeResult, ConfigResult and PlanObservedMetrics are of its core only. The
drop-in's package, which is the module of its client, defined those three classes and exported only the four.

- The binding reads each path of addArtifacts and _add_artifacts_of_flags as PySpark's ArtifactManager does: a str
  as it is, a Windows absolute path of another object as the file URI that PureWindowsPath and Path.resolve make of
  it, and urlparse's error, or UNSUPPORTED_OPERATION of a scheme of bytes, otherwise. Each path is uploaded before
  the next is read.
- AnalyzeResult, ConfigResult and PlanObservedMetrics are defined in pyspark.sql.connect.client.core, which has the
  metrics classes of PySpark's core too, and the package imports them as private names. Its `__all__` is the names
  of PySpark's `import *` of the package but its gRPC modules: the four classes, core and getLogLevel. The package
  sets the four classes in core once it has defined them; the binding reads PlanObservedMetrics there.
- The mock server of the offline suite records the first request of each AddArtifacts.
- docs/compatibility.md describes the modules, and that an error that names the type of a singleton type class,
  as the NOT_EXPECTED_TYPE of udf(f, IntegerType), names `type`, the metaclass of the native class.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62061-client-leftovers branch from 081681d to 8517f59 Compare November 19, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant