Skip to content

[SPARK-62064][CONNECT][PYTHON] Send the relations of DataFrames that the extensions of expressions hold, and with the changes a plan makes to them, and resolve the messages of extensions in a time of their length, as PySpark sends its plans - #877

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62064-placeholder-leftovers

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

A follow-up of #862 (SPARK-62021) and #785 (SPARK-61867), from the follow-up list of the review of the last 100 commits against pyspark-client 4.3.0.

df._plan.plan(session) gives a placeholder that holds the relation of the DataFrame. The core replaces it with that relation wherever a plan built in Python holds it (plan::resolve_relation, resolve_command). PySpark's plan(session) is the relation itself, which a plan or a plugin can hold anywhere and change. #862 documented two gaps, and #785 left a third:

  1. The extensions of expressions. The core did not look into the expressions of a plan built in Python. So a placeholder in the message of a plugin's expression (Expression.extension) was sent as a placeholder, where PySpark's plan carries the relation.
  2. A placeholder that a plan changes, such as relation.common.plan_id = ...:
    • as a relation of a plan, the core replaced the placeholder with its relation and dropped the change, so a column of the changed plan ID failed with CANNOT_RESOLVE_DATAFRAME_COLUMN;
    • in the message of an extension, the core found a placeholder by its exact encoding, which the change alters, so it sent the placeholder, which the server refuses.
  3. The time of the messages of extensions. The core read the message of an extension level by level, and at each level searched all of its bytes for the type URL of a placeholder and copied them again. A crafted message that nests many messages took a time of its length times its depth. PySpark sends the bytes as they are.

The rows of the plan ID were checked on both clients against a Spark 4.3.0 server. A server has no handler of the plugin's messages, so the drop-in's were checked with the recording mock server; PySpark's hold the relation, which plan(session) returns:

a plan built in Python that holds spark.range(3)._plan.plan(client) pyspark-client 4.3.0 drop-in before drop-in now
as the input of a project, with input.common.plan_id = 424242, of a column of that plan ID {'id': [0, 1, 2]} CANNOT_RESOLVE_DATAFRAME_COLUMN {'id': [0, 1, 2]}
the same, with the plan ID changed after the copy into the plan {'id': [0, 1, 2]} CANNOT_RESOLVE_DATAFRAME_COLUMN {'id': [0, 1, 2]}
in ExamplePluginRelation(input=...) packed into a project's expression, also under an alias the relation the placeholder the relation
in a relation extension's message, with its common.plan_id changed the relation, of that plan ID the placeholder the relation, of that plan ID
at the bottom of a crafted extension that nests 64,000 messages the bytes as they are 20.9 s 0.077 s

The extensions of expressions (plan::resolve_own_fields).

  • The core does not walk the expressions. Instead, for each relation built in Python of a kind that holds expressions, it takes the inputs out and encodes the other fields. These kinds are a projection, a filter, a join, a sort, an aggregation, SQL with arguments and the like, and an exhaustive match of the relation types (holds_expressions) names them.
  • Only when that encoding holds the type URL of a placeholder does the core resolve it (resolve_encoded) and decode it back. Then it puts the inputs back, which it resolves as before.
  • The commands built in Python get the same, after the relation they hold, but for the extensions of commands and the ML commands, which the core resolved already.
  • The encoding holds every expression of the relation, however it nests, so no walk of the expression types has to keep up with the protos. The relations that the core builds, such as that of a placeholder, are not encoded.
  • The encoding and the decoding run on a grown stack (stacker::maybe_grow), as the other deep reads do.

A placeholder that a plan changes.

  • A placeholder is now found by its structure: a relation whose only fields are common (1) and an extension (998) of the placeholder's type URL.
  • Its common, when set, is merged into the common of its relation (prost's Message::merge, as protobuf's MergeFrom sets the fields). This holds both as a relation of a plan and in the message of an extension, so the plan ID is the one the plan set, as on PySpark's changed relation.

A time of the length of the messages (plan::resolve_encoded).

  • The core finds the offsets of the type URL in the encoding once.
  • It reads a field as a message only when its range holds one of them, found by a binary search.
  • The changes make a tree whose lengths it computes first. Then it writes the resolved encoding once.

What a plan of no extension costs. The encoding of the fields of its relations that hold expressions, which the core then searches for the type URL. Only the plans built outside the core pay it: those of DataFrame(plan, session), of the client's to_table, execute_command and analyses, and of pyspark.ml, and the requests that execute-plan hooks return. The plans of the DataFrame API, without hooks, are not resolved at all. DataFrame(plan, session) of a plan built in Python over a placeholder, in a release build on a loaded machine. Each time is the best of 5 rounds of 7 runs, with the rounds of master and of this PR alternating:

plan master here
a project of 10 expressions 0.099 ms 0.094 ms
a project of 1,000 expressions 0.322 ms 0.338 ms
a project of 10,000 expressions 2.370 ms 2.716 ms
a project of 50,000 expressions 13.885 ms 17.653 ms
a chain of 400 limits, which hold no expressions and are not encoded 0.381 ms 0.375 ms

docs/compatibility.md, dev/parity/known-differences.md and AGENTS.md now say that the core resolves placeholders in the extensions of expressions and sends a changed placeholder with its change, where they said it did not.

Why are the changes needed?

So that a plan built in Python sends the relation of a DataFrame wherever it holds it, and with the changes it makes to it, as PySpark's plans do. And so that a crafted deep message is sent in a time of its length.

Does this PR introduce any user-facing change?

Yes:

  • a placeholder of df._plan.plan(session) in the extension of an expression is sent as the relation of the DataFrame;
  • a placeholder whose common a plan or a plugin's message changes is sent as the relation, with that change;
  • the messages of extensions that nest deeply are resolved in a time of their length.

How was this patch tested?

  • New Rust unit tests in plan.rs's placeholder_tests, all of which fail on master and pass here:
    • placeholders_resolve_in_the_extensions_of_expressions: a plugin's expression, also under an alias;
    • a_placeholder_that_a_plan_changes_is_sent_with_the_change: as the input of an ML transform, and under a subquery alias in the message of an extension;
    • deep_messages_of_extensions_resolve_in_a_time_of_their_length: 30,000 levels, with a placeholder at the bottom and without, within 10 s in a debug build. Master took 69 s on a loaded machine (41 s before).
  • New offline test in python/tests/test_dropin_offline.py, with the mock server: test_the_relations_of_dataframes_in_expressions_and_changed_ones_are_sent_as_pyspark. It covers ExamplePluginRelation in expressions, a changed plan ID in an extension, and one as a relation of a plan. It fails on master and passes here.
  • New e2e test in python/tests/test_dropin_e2e.py: test_the_relation_of_a_dataframes_plan_is_sent_with_its_changed_plan_id_as_pyspark. It fails on master and passes here.
  • Probes of both clients against a Spark 4.3.0 server: the first two rows of the table. Also the time of the crafted nesting, of 2,000 to 64,000 levels: 0.030, 0.286, 4.36 and 20.9 s on master, and 0.008, 0.010, 0.037 and 0.077 s here, until the server's InvalidPlanInput for the crafted extension.
  • The Rust gates:
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps;
    • cargo test -p pyspark-rs --lib: 15 passed;
    • cargo test -p apache-spark-connect -p apache-spark-connect-core in a debug build, also against a Spark 4.3.0 server: 1,729 passed, 0 failed, 18 ignored. One timing test of apache-spark-connect-core, which this PR does not touch, hit its 100 ms deadline against the mock server once on the loaded machine. It passed in each of 3 reruns, and so did the whole crate.
  • The Python gates:
    • ruff 0.14.0;
    • test_ruff_excludes.py and scripts/audit_no_stubs.sh;
    • the stub and dispatch generators;
    • the test-list guards;
    • test_type_stubs_offline.py: 13 passed, 3 skipped.
  • The offline suite of coverage.yml, under coverage with its .coveragerc: 6,501 passed, 77 skipped, and Python coverage 93.38% (the floor is 93%).
  • test_dropin_e2e.py against a Spark 4.3.0 server: 334 passed. test_ml_e2e.py, its transform, pipeline, summary, evaluator and hook tests: 17 passed.
  • Official suites pyspark.ml.tests.connect.test_parity_pipeline, test_parity_feature and test_parity_tuning through run_official_tests.py --dropin, on master and here: the same on both, with no unexpected failures: test_parity_pipeline 6 passed and 4 skipped, test_parity_feature 51 passed, test_parity_tuning 7 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…the extensions of expressions hold, and with the changes a plan makes to them, and resolve the messages of extensions in a time of their length, as PySpark sends its plans

- The extensions of the expressions of a relation or a command built in Python are resolved: of
  each such relation of a kind that holds expressions, the core encodes the fields other than its
  inputs, and reads them for a placeholder only when the encoding holds its type URL
  (plan::resolve_own_fields).
- A placeholder whose common a plan changes (relation.common.plan_id = ...) is sent with that
  common merged into the relation's, as a relation of a plan and in the message of an extension,
  where the core found it by its encoding and sent it as a placeholder.
- The messages of extensions are resolved in a time of their length, however deep they nest: the
  offsets of the type URL are found once, a field is read only when it holds one, and the
  resolved encoding is written once (plan::resolve_encoded).

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants