Repository navigation
[SPARK-62063][CONNECT][PYTHON] Report the option keys and values that are objects, and the arguments that the writers keep, to Python's garbage collector in the readers, writers and session builder, also of their shallow copies - #876
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
… are objects, and the arguments that the writers keep, to Python's garbage collector in the readers, writers and session builder, also of their shallow copies PySpark's readers, writers and Connect builder keep a key of an option that is not a str, and a table property's value, as they are in their dicts, and the writers a format, a mode, columns and a provider that are not strs; the collector sees them, also when a shallow copy shares the dict. The drop-in kept them in native state that the copies share, which the collector did not see, so a cycle through one was never collected. - A native object holds each such state (crate::options::SharedObjects, PySharedObjects): each copy holds it and reports it, and it reports the Python objects of the state, each once, and clears them in its __clear__. - The readers, the stream reader, the V1 and V2 writers and the builder hold one; the writers report all of their kept arguments through it. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A follow-up of #838 (SPARK-61983), from the follow-up list of the review of the last 100 commits against pyspark-client 4.3.0.
PySpark's readers, writers and Connect session builder are plain Python objects, which keep some arguments as they are, in their
__dict__:DataFrameReader.option,DataStreamReader.option,DataFrameWriter.option,DataFrameWriterV2.optionandtableProperty, andBuilder.config. A table property's value is kept as it is too;Python's garbage collector sees these objects through the objects' dicts, also when a shallow copy (
copy.copy) shares them.The drop-in kept them in native state, which the shallow copies share as PySpark's copies share the dict, and which the collector did not see:
UnencodableOptionsof the readers and of the V2 writer;UnsentArgsand kept columns;BuilderOptions.So a reference cycle through one was never collected, such as a key of an option that keeps its reader. The V1 and V2 writers reported a few of their kept arguments only when no copy shared them, so a cycle through a writer and a copy of it was never collected either. Checked on both clients against a Spark 4.3.0 server:
spark.read.option,options,readStream.option,df.write.option,writeTo(...).option,tableProperty(key or value),builder.configdf.write.format(obj)andwriteTo(...).using(obj), with a shallow copydf.write.format,mode,partitionBy,clusterBy,writeTo(...).using,clusterBy, alone, andspark.read.format, also with a shallow copydf.writeStream.option(obj, ...),bucketBy/sortByof an objectA Python object holds each state that the copies share.
crate::options::SharedObjectsis the state's view of its Python objects.PySharedObjects(_SharedObjects) holds the state. It reports the state's objects in its__traverse__, and drops them in its__clear__, which runs when every holder of the state is unreachable.__traverse__.So each Python object of the state is reported once, by that object, and is freed with the last copy that holds the state, as PySpark's dict is.
UnencodableOptions, which its request copies share, are reported only when the state alone holds them.The new class is one of the native classes outside the module that get their bare C name (
classes_outside_module).docs/compatibility.mdanddev/parity/known-differences.mddescribe this.Why are the changes needed?
So that the objects that these keep alive, and everything that holds them, are freed as in PySpark, and long-running programs do not leak them.
Does this PR introduce any user-facing change?
Yes: reference cycles through the option keys and values that are objects, and through the arguments that the writers keep, of the readers, the writers and the Connect builder and of their shallow copies, are collected.
How was this patch tested?
python/tests/test_dropin_offline.py, with the mock server:test_cycles_through_the_option_keys_and_kept_arguments_are_collected_as_pyspark. It fails on master and passes here. It covers:test_every_native_class_has_its_bare_name_as_its_c_namecovers the new class. The creation-crash test covers the readers, writers, builders and their copies.cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps;cargo test -p pyspark-rs --lib: 15 passed;cargo test -p apache-spark-connect -p apache-spark-connect-corein a debug build, also against a Spark 4.3.0 server: 1,726 passed, 0 failed, 18 ignored, of 118 test binaries and doc-test runs.test_ruff_excludes.pyandscripts/audit_no_stubs.sh;test_type_stubs_offline.py: 13 passed, 3 skipped.coverage.yml, under coverage with its.coveragerc: 6,500 passed, 77 skipped, and Python coverage 93.37% (the floor is 93%).test_dropin_e2e.pyagainst a Spark 4.3.0 server: 333 passed.test_ml_e2e.py, its transform, pipeline, summary, evaluator and hook tests: 17 passed.test_connect_readwriter,test_parity_readwriterandtest_connect_sessionthroughrun_official_tests.py --dropin, on master and here: the same on both, with no unexpected failures:test_connect_readwriter0 passed and 38 skipped,test_parity_readwriter22 passed,test_connect_session10 passed and 17 skipped.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.