Repository navigation
[MINOR][TESTS] Keep the debugging cache of each test to itself, read only pytest's summary line in run_official_tests.py, and document how to install a package that requires pyspark over the drop-in - #872
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…only pytest's summary line in run_official_tests.py, and document how to install a package that requires pyspark over the drop-in Three tests of test_dropin_offline.py failed after test_dropin_types_columns_offline.py, as their expressions carried call-site origins: PySpark's is_debugging_enabled (vendored verbatim) keeps its first answer of spark.python.sql.dataFrameDebugging.enabled for the interpreter, and the first Column of test_the_abstract_types_have_no_proto_nor_arrow_type_as_pysparks, made with a mock session active, set it to True before test_dropin_offline.py read it without a session. An autouse fixture of python/tests/conftest.py now leaves the cache, after each test, as the test found it. run_official_tests.py's parse_counts took the last line with "passed", "failed", "error" or "skipped" for pytest's summary, so the "The error is:" line of PySpark's listener-bus warning, on stderr after the summary, gave zero counts. It now matches pytest's summary line only, with unit tests. docs/installation.md says what to do when a package requires pyspark or pyspark-client, which pip installs over the drop-in: --no-deps, the order of the installs, or uv's overrides. Co-authored-by: Isaac <no-reply@databricks.com>
dongjoon-hyun
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
1. The order of the offline test files no longer changes their results.
Three tests of
test_dropin_offline.pyfailed whentest_dropin_types_columns_offline.pyran before them. CI's order passes. The failing order:The tests compare the expressions of a pickled Column with those of the Column it was pickled from, which are built on another line. With call-site origins, the
file:lineof each expression differs.The leaking state is
pyspark.errors.utils._enable_debugging_cache. That module is PySpark's, vendored verbatim. Itsis_debugging_enabledkeeps its first answer ofspark.python.sql.dataFrameDebugging.enabledfor the interpreter. It reads the conf of the active session, or answers False without one.test_dropin_offline.pyreads it without a session in a session-scoped fixture (_debugging_read_without_a_session), so its tests run without origins.test_the_abstract_types_have_no_proto_nor_arrow_type_as_pysparksoftest_dropin_types_columns_offline.pymakes the first Column of the process (F.col("a")) with a mock session active. The mock answers the conf's default, so the cache became True.test_dropin_offline.pythen read the cached True, and every later expression carried an origin.This is not a bug of the drop-in. PySpark's own
is_debugging_enabledcaches the same way.The fix: a new autouse fixture of
python/tests/conftest.pyleaves the cache, after each test, as the test found it. A test can no longer decide the origins of the tests after it, whatever the order of the files.2.
scripts/run_official_tests.py'sparse_countsreads pytest's summary line only.connect/streaming/query.py:380-383). Its word "error" ended the search with zero counts, so the counts of a file that ran were lost.=rule unless-qdrops it. ANSI colors are removed first.3.
docs/installation.mdsays how to install a package that requirespysparkorpyspark-client.pyspark-client-rustprovides the samepysparkpackage. pip has no option to drop one requirement, and a constraints file cannot do it.pip install --no-depsand then the package's other requirements;--overridewith a marker that never applies.Why are the changes needed?
Does this PR introduce any user-facing change?
No code of the drop-in changes.
docs/installation.mdhas the new section. The test fixture and the parity runner are for contributors.How was this patch tested?
The test order:
test_dropin_types_columns_offline.pywithtest_dropin_offline.py -k "pickle or origin or debugging or call_site", in both file orders: 352 passed and 365 passed.coverage.yml, in CI's order: 6494 passed, 77 skipped.test_dropin_e2e.pyagainst a Spark 4.3.0 server: 333 passed.The runner: the new
ParseCountsTestsofscripts/tests/test_run_official_tests.pycheck:-q,=rules, durations, colors, subtests, xfail and xpass, and no tests;The first test fails without the change.
python3 -m unittest discover -s scripts/tests: 112 tests OK.The installation advice: checked with pip 26.3 and uv 0.13.5 in scratch virtual environments. They installed the published
pyspark-client-rust4.3.0, the publishedpyspark-client4.3.0, and a package that requirespyspark-clientandsix. A script compared the drop-in's installed files with the sha256 of itsRECORD:pip installof the package replaced 91 of the drop-in's files withpyspark-client's;--no-depsand thensixkept all of them, upgrades with--no-depstoo, andpip checkreports the missingpyspark-client;pip uninstall -y pyspark pyspark-clientandpip install pyspark-client-rust, kept them. A laterpip install --force-reinstallof the package replaced them again;pyspark-client==0.0.0failed withResolutionImpossible;uv pip install --overridewithpyspark; sys_platform == "never"andpyspark-client; sys_platform == "never"kept the drop-in and installedsix.Gates:
cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;cargo test -p pyspark-rs --lib: 15 passed;scripts/audit_no_stubs.sh, the stub and dispatch generators'--check;test_type_stubs_offline.py: 13 passed, 3 skipped;Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.