Repository navigation
[SPARK-62064][CONNECT][PYTHON] Send the relations of DataFrames that the extensions of expressions hold, and with the changes a plan makes to them, and resolve the messages of extensions in a time of their length, as PySpark sends its plans - #877
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…the extensions of expressions hold, and with the changes a plan makes to them, and resolve the messages of extensions in a time of their length, as PySpark sends its plans - The extensions of the expressions of a relation or a command built in Python are resolved: of each such relation of a kind that holds expressions, the core encodes the fields other than its inputs, and reads them for a placeholder only when the encoding holds its type URL (plan::resolve_own_fields). - A placeholder whose common a plan changes (relation.common.plan_id = ...) is sent with that common merged into the relation's, as a relation of a plan and in the message of an extension, where the core found it by its encoding and sent it as a placeholder. - The messages of extensions are resolved in a time of their length, however deep they nest: the offsets of the type URL are found once, a field is read only when it holds one, and the resolved encoding is written once (plan::resolve_encoded). Co-authored-by: Isaac <no-reply@databricks.com>
dongjoon-hyun
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A follow-up of #862 (SPARK-62021) and #785 (SPARK-61867), from the follow-up list of the review of the last 100 commits against pyspark-client 4.3.0.
df._plan.plan(session)gives a placeholder that holds the relation of the DataFrame. The core replaces it with that relation wherever a plan built in Python holds it (plan::resolve_relation,resolve_command). PySpark'splan(session)is the relation itself, which a plan or a plugin can hold anywhere and change. #862 documented two gaps, and #785 left a third:Expression.extension) was sent as a placeholder, where PySpark's plan carries the relation.relation.common.plan_id = ...:CANNOT_RESOLVE_DATAFRAME_COLUMN;The rows of the plan ID were checked on both clients against a Spark 4.3.0 server. A server has no handler of the plugin's messages, so the drop-in's were checked with the recording mock server; PySpark's hold the relation, which
plan(session)returns:spark.range(3)._plan.plan(client)input.common.plan_id = 424242, of a column of that plan ID{'id': [0, 1, 2]}CANNOT_RESOLVE_DATAFRAME_COLUMN{'id': [0, 1, 2]}{'id': [0, 1, 2]}CANNOT_RESOLVE_DATAFRAME_COLUMN{'id': [0, 1, 2]}ExamplePluginRelation(input=...)packed into a project's expression, also under an aliascommon.plan_idchangedThe extensions of expressions (
plan::resolve_own_fields).holds_expressions) names them.resolve_encoded) and decode it back. Then it puts the inputs back, which it resolves as before.stacker::maybe_grow), as the other deep reads do.A placeholder that a plan changes.
common(1) and anextension(998) of the placeholder's type URL.common, when set, is merged into thecommonof its relation (prost'sMessage::merge, as protobuf'sMergeFromsets the fields). This holds both as a relation of a plan and in the message of an extension, so the plan ID is the one the plan set, as on PySpark's changed relation.A time of the length of the messages (
plan::resolve_encoded).What a plan of no extension costs. The encoding of the fields of its relations that hold expressions, which the core then searches for the type URL. Only the plans built outside the core pay it: those of
DataFrame(plan, session), of the client'sto_table,execute_commandand analyses, and ofpyspark.ml, and the requests that execute-plan hooks return. The plans of the DataFrame API, without hooks, are not resolved at all.DataFrame(plan, session)of a plan built in Python over a placeholder, in a release build on a loaded machine. Each time is the best of 5 rounds of 7 runs, with the rounds of master and of this PR alternating:docs/compatibility.md,dev/parity/known-differences.mdandAGENTS.mdnow say that the core resolves placeholders in the extensions of expressions and sends a changed placeholder with its change, where they said it did not.Why are the changes needed?
So that a plan built in Python sends the relation of a DataFrame wherever it holds it, and with the changes it makes to it, as PySpark's plans do. And so that a crafted deep message is sent in a time of its length.
Does this PR introduce any user-facing change?
Yes:
df._plan.plan(session)in the extension of an expression is sent as the relation of the DataFrame;commona plan or a plugin's message changes is sent as the relation, with that change;How was this patch tested?
plan.rs'splaceholder_tests, all of which fail on master and pass here:placeholders_resolve_in_the_extensions_of_expressions: a plugin's expression, also under an alias;a_placeholder_that_a_plan_changes_is_sent_with_the_change: as the input of an ML transform, and under a subquery alias in the message of an extension;deep_messages_of_extensions_resolve_in_a_time_of_their_length: 30,000 levels, with a placeholder at the bottom and without, within 10 s in a debug build. Master took 69 s on a loaded machine (41 s before).python/tests/test_dropin_offline.py, with the mock server:test_the_relations_of_dataframes_in_expressions_and_changed_ones_are_sent_as_pyspark. It coversExamplePluginRelationin expressions, a changed plan ID in an extension, and one as a relation of a plan. It fails on master and passes here.python/tests/test_dropin_e2e.py:test_the_relation_of_a_dataframes_plan_is_sent_with_its_changed_plan_id_as_pyspark. It fails on master and passes here.InvalidPlanInputfor the crafted extension.cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps;cargo test -p pyspark-rs --lib: 15 passed;cargo test -p apache-spark-connect -p apache-spark-connect-corein a debug build, also against a Spark 4.3.0 server: 1,729 passed, 0 failed, 18 ignored. One timing test ofapache-spark-connect-core, which this PR does not touch, hit its 100 ms deadline against the mock server once on the loaded machine. It passed in each of 3 reruns, and so did the whole crate.test_ruff_excludes.pyandscripts/audit_no_stubs.sh;test_type_stubs_offline.py: 13 passed, 3 skipped.coverage.yml, under coverage with its.coveragerc: 6,501 passed, 77 skipped, and Python coverage 93.38% (the floor is 93%).test_dropin_e2e.pyagainst a Spark 4.3.0 server: 334 passed.test_ml_e2e.py, its transform, pipeline, summary, evaluator and hook tests: 17 passed.pyspark.ml.tests.connect.test_parity_pipeline,test_parity_featureandtest_parity_tuningthroughrun_official_tests.py --dropin, on master and here: the same on both, with no unexpected failures:test_parity_pipeline6 passed and 4 skipped,test_parity_feature51 passed,test_parity_tuning7 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.