Repository navigation
[SPARK-62061][CONNECT][PYTHON] Read a path of addArtifacts that is not a str as PySpark's ArtifactManager, and define the client's results in pyspark.sql.connect.client.core with PySpark's import * of its package - #875
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…t a str as PySpark's ArtifactManager, and define the client's results in pyspark.sql.connect.client.core with PySpark's import * of its package PySpark's SparkConnectClient.add_artifacts (connect/client/core.py:2461-2470) reads, checks and uploads its paths one after the other, and its ArtifactManager parses each with urlparse (artifact.py:184-203), after SparkSession.addArtifacts has checked its flags (connect/session.py:1110-1121). So a path that is not a str raises urlparse's error: the AttributeError of the `decode` that a pathlib.Path or an int does not have, or, of bytes or of a falsy object, which urlparse parses into a scheme of bytes, UNSUPPORTED_OPERATION of that scheme. The drop-in's addArtifacts, addArtifact and client.add_artifacts took the paths as a list of str, so any other object raised PyO3's TypeError before the flags were checked and before the paths before it were uploaded. PySpark's pyspark.sql.connect.client package is `from pyspark.sql.connect.client.core import *` and its getLogLevel, with no `__all__`: its `import *` gives the four classes of the `__all__` of its core, getLogLevel and its modules core, artifact, reattach and retries. AnalyzeResult, ConfigResult and PlanObservedMetrics are of its core only. The drop-in's package, which is the module of its client, defined those three classes and exported only the four. - The binding reads each path of addArtifacts and _add_artifacts_of_flags as PySpark's ArtifactManager does: a str as it is, a Windows absolute path of another object as the file URI that PureWindowsPath and Path.resolve make of it, and urlparse's error, or UNSUPPORTED_OPERATION of a scheme of bytes, otherwise. Each path is uploaded before the next is read. - AnalyzeResult, ConfigResult and PlanObservedMetrics are defined in pyspark.sql.connect.client.core, which has the metrics classes of PySpark's core too, and the package imports them as private names. Its `__all__` is the names of PySpark's `import *` of the package but its gRPC modules: the four classes, core and getLogLevel. The package sets the four classes in core once it has defined them; the binding reads PlanObservedMetrics there. - The mock server of the offline suite records the first request of each AddArtifacts. - docs/compatibility.md describes the modules, and that an error that names the type of a singleton type class, as the NOT_EXPECTED_TYPE of udf(f, IntegerType), names `type`, the metaclass of the native class. Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62061-client-leftovers
branch
from
November 19, 2026 02:08
081681d to
8517f59
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Two follow-ups of SPARK-62023 (#863), each matching pyspark-client 4.3.0. Details are in
jira-SPARK-62061-client-leftovers.txt.1. A path of
addArtifactsthat is not a str. PySpark's client reads, checks and uploads its paths one after the other (connect/client/core.py:2461-2470). ItsArtifactManager._parse_artifactsparses each path withurlparse(artifact.py:184-203), afterSparkSession.addArtifactshas checked its flags. The drop-in took the paths as a list of str, so any other object raised PyO3'sTypeError. That happened before the flags were checked and before any path was uploaded. Now:addArtifacts,addArtifactandclient.add_artifactsas PySpark'sArtifactManagerdoes:PureWindowsPathandPath.resolvemake of it;urlparseitself, such as theAttributeErrorof apathlib.Pathor an int, orUNSUPPORTED_OPERATIONof a scheme of bytes, for bytes or a falsy object;2. The names of
pyspark.sql.connect.client. PySpark's package is theimport *of its core plusgetLogLevel, with no__all__. The drop-in's package is the module of its client. Now:AnalyzeResult,ConfigResultandPlanObservedMetricsare defined inpyspark.sql.connect.client.core, as PySpark's are, and not in the package. The package imports them under private names.__all__gives what PySpark'simport *of it gives: the four classes,coreandgetLogLevel. The gRPC modules are the exception:artifactandreattachare replaced by the transport, andretriesimports grpc, so it is imported only when read.PlanObservedMetricsthere.SparkConnectClientmoves intoclient/core.py, which a later change does.Also checked, with no change needed. Both clients behave the same on master for:
registerJavaFunction,registerJavaUDAF,udtf.registerandudf.register: protobuf'sTypeError(SPARK-61911, SPARK-62023);docs/compatibility.mdnow notes one consequence of the native singleton types. An error that names the type of such a class, as theNOT_EXPECTED_TYPEofudf(f, IntegerType), namestype, where PySpark's namesDataTypeSingleton.Why are the changes needed?
So that the paths of
addArtifactsand the names ofpyspark.sql.connect.clientare PySpark Connect's. A program that passes apathlib.Pathgets PySpark's error, in PySpark's order, after the paths before it are uploaded. A program that imports from the package gets PySpark's names, and the result classes only fromcore.Does this PR introduce any user-facing change?
Yes, as pyspark-client 4.3.0:
spark.addArtifacts(pathlib.Path(...))raisesAttributeError: 'PosixPath' object has no attribute 'decode'.Noneand0raiseUNSUPPORTED_OPERATIONofb'' scheme.from pyspark.sql.connect.client import AnalyzeResultraisesImportError.from pyspark.sql.connect.client import *givesgetLogLevelandcoretoo.How was this patch tested?
test_dropin_offline.py, both of which fail on master:test_add_artifacts_reads_a_path_that_is_not_a_str_as_pyspark, on the mock server, which now records the first request of each AddArtifacts;test_the_classes_of_the_client_are_of_its_core_module_as_pyspark.p12_artifact_paths.py: identical, 15 of 15 lines.p13_client_names.py: the result classes,import *, core's names andgetLogLevelare identical. The rest of the package's namespace and its__all__differ, as described above.test_client,test_artifact,test_parity_observation,test_connect_sessionandtest_connect_clone_sessionwere run against the drop-in (run_official_tests.py --dropin --only). The results are the same on master and on this branch.cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-depsof the four crates--features apache-spark-connect/wasm-udf;cargo test -p pyspark-rs --lib: 15 passed;test_type_stubs_offline.py(13 passed, 3 skipped);coverage.yml: 6495 passed, 77 skipped;coverage.yml, with its commands and its.coveragerc: 93.43% (floor 93.00%), with no new line of the change uncovered;test_dropin_e2e.pyagainst a Spark 4.3.0 server: 333 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.