Skip to content

[SPARK-62063][CONNECT][PYTHON] Report the option keys and values that are objects, and the arguments that the writers keep, to Python's garbage collector in the readers, writers and session builder, also of their shallow copies - #876

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62063-gc-option-objects

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

A follow-up of #838 (SPARK-61983), from the follow-up list of the review of the last 100 commits against pyspark-client 4.3.0.

PySpark's readers, writers and Connect session builder are plain Python objects, which keep some arguments as they are, in their __dict__:

  • option dicts keep a key that is not a str as it is: DataFrameReader.option, DataStreamReader.option, DataFrameWriter.option, DataFrameWriterV2.option and tableProperty, and Builder.config. A table property's value is kept as it is too;
  • the writers keep a format, a save mode, partitioning and clustering columns and a provider that are not strs until the write.

Python's garbage collector sees these objects through the objects' dicts, also when a shallow copy (copy.copy) shares them.

The drop-in kept them in native state, which the shallow copies share as PySpark's copies share the dict, and which the collector did not see:

  • UnencodableOptions of the readers and of the V2 writer;
  • the writers' UnsentArgs and kept columns;
  • the builder's BuilderOptions.

So a reference cycle through one was never collected, such as a key of an option that keeps its reader. The V1 and V2 writers reported a few of their kept arguments only when no copy shared them, so a cycle through a writer and a copy of it was never collected either. Checked on both clients against a Spark 4.3.0 server:

a cycle through pyspark-client 4.3.0 drop-in before
a key of spark.read.option, options, readStream.option, df.write.option, writeTo(...).option, tableProperty (key or value), builder.config collected not collected
the same, while a shallow copy is in the cycle too (reader, writer, builder) collected not collected
df.write.format(obj) and writeTo(...).using(obj), with a shallow copy collected not collected
df.write.format, mode, partitionBy, clusterBy, writeTo(...).using, clusterBy, alone, and spark.read.format, also with a shallow copy collected collected
df.writeStream.option(obj, ...), bucketBy/sortBy of an object refused at the call the same

A Python object holds each state that the copies share.

  • crate::options::SharedObjects is the state's view of its Python objects.
  • PySharedObjects (_SharedObjects) holds the state. It reports the state's objects in its __traverse__, and drops them in its __clear__, which runs when every holder of the state is unreachable.
  • Each reader, stream reader, V1 and V2 writer and builder holds the object of its state, and its shallow copies hold the same object. Each reports it in its __traverse__.

So each Python object of the state is reported once, by that object, and is freed with the last copy that holds the state, as PySpark's dict is.

  • The objects of UnencodableOptions, which its request copies share, are reported only when the state alone holds them.
  • The writers report all of their kept arguments through the object: also the sorting and bucketing columns, the options and the refused bucket count, which they did not report before. They no longer report any directly.

The new class is one of the native classes outside the module that get their bare C name (classes_outside_module). docs/compatibility.md and dev/parity/known-differences.md describe this.

Why are the changes needed?

So that the objects that these keep alive, and everything that holds them, are freed as in PySpark, and long-running programs do not leak them.

Does this PR introduce any user-facing change?

Yes: reference cycles through the option keys and values that are objects, and through the arguments that the writers keep, of the readers, the writers and the Connect builder and of their shallow copies, are collected.

How was this patch tested?

  • New offline test in python/tests/test_dropin_offline.py, with the mock server: test_cycles_through_the_option_keys_and_kept_arguments_are_collected_as_pyspark. It fails on master and passes here. It covers:
    • 13 holders, alone and with a shallow copy in the cycle;
    • a copy that stays keeps the options of the copy that is collected, until it goes too.
  • test_every_native_class_has_its_bare_name_as_its_c_name covers the new class. The creation-crash test covers the readers, writers, builders and their copies.
  • Probes of both clients against a Spark 4.3.0 server: the 24 cases of the table. On master, 13 of them leaked where PySpark collected them. Here, 21 are collected on both and 3 are refused at the call on both.
  • The Rust gates:
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps;
    • cargo test -p pyspark-rs --lib: 15 passed;
    • cargo test -p apache-spark-connect -p apache-spark-connect-core in a debug build, also against a Spark 4.3.0 server: 1,726 passed, 0 failed, 18 ignored, of 118 test binaries and doc-test runs.
  • The Python gates:
    • ruff 0.14.0;
    • test_ruff_excludes.py and scripts/audit_no_stubs.sh;
    • the stub and dispatch generators;
    • the test-list guards;
    • test_type_stubs_offline.py: 13 passed, 3 skipped.
  • The offline suite of coverage.yml, under coverage with its .coveragerc: 6,500 passed, 77 skipped, and Python coverage 93.37% (the floor is 93%).
  • test_dropin_e2e.py against a Spark 4.3.0 server: 333 passed. test_ml_e2e.py, its transform, pipeline, summary, evaluator and hook tests: 17 passed.
  • Official suites test_connect_readwriter, test_parity_readwriter and test_connect_session through run_official_tests.py --dropin, on master and here: the same on both, with no unexpected failures: test_connect_readwriter 0 passed and 38 skipped, test_parity_readwriter 22 passed, test_connect_session 10 passed and 17 skipped.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

… are objects, and the arguments that the writers keep, to Python's garbage collector in the readers, writers and session builder, also of their shallow copies

PySpark's readers, writers and Connect builder keep a key of an option that is not a str, and a
table property's value, as they are in their dicts, and the writers a format, a mode, columns and
a provider that are not strs; the collector sees them, also when a shallow copy shares the dict.
The drop-in kept them in native state that the copies share, which the collector did not see, so a
cycle through one was never collected.

- A native object holds each such state (crate::options::SharedObjects, PySharedObjects): each
  copy holds it and reports it, and it reports the Python objects of the state, each once, and
  clears them in its __clear__.
- The readers, the stream reader, the V1 and V2 writers and the builder hold one; the writers
  report all of their kept arguments through it.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant