Skip to content

[MINOR][TESTS] Keep the debugging cache of each test to itself, read only pytest's summary line in run_official_tests.py, and document how to install a package that requires pyspark over the drop-in - #872

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:MINOR-tests-order-and-runner

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

1. The order of the offline test files no longer changes their results.

  • Three tests of test_dropin_offline.py failed when test_dropin_types_columns_offline.py ran before them. CI's order passes. The failing order:

    pytest tests/test_dropin_types_columns_offline.py \
      tests/test_dropin_offline.py::test_pickled_sort_columns_are_sort_orders_as_pyspark \
      tests/test_dropin_offline.py::test_columns_pickle_as_their_expressions_as_pyspark \
      tests/test_dropin_offline.py::test_window_specs_pickle_as_pyspark
    

    The tests compare the expressions of a pickled Column with those of the Column it was pickled from, which are built on another line. With call-site origins, the file:line of each expression differs.

  • The leaking state is pyspark.errors.utils._enable_debugging_cache. That module is PySpark's, vendored verbatim. Its is_debugging_enabled keeps its first answer of spark.python.sql.dataFrameDebugging.enabled for the interpreter. It reads the conf of the active session, or answers False without one.

    • test_dropin_offline.py reads it without a session in a session-scoped fixture (_debugging_read_without_a_session), so its tests run without origins.
    • test_the_abstract_types_have_no_proto_nor_arrow_type_as_pysparks of test_dropin_types_columns_offline.py makes the first Column of the process (F.col("a")) with a mock session active. The mock answers the conf's default, so the cache became True.
    • The fixture of test_dropin_offline.py then read the cached True, and every later expression carried an origin.
  • This is not a bug of the drop-in. PySpark's own is_debugging_enabled caches the same way.

  • The fix: a new autouse fixture of python/tests/conftest.py leaves the cache, after each test, as the test found it. A test can no longer decide the origins of the tests after it, whatever the order of the files.

2. scripts/run_official_tests.py's parse_counts reads pytest's summary line only.

  • It took the last line with " passed", " failed", " error" or " skipped" for the summary.
  • The warning of PySpark's listener bus comes after pytest's summary on stderr ("... handler thread is terminated. The error is: ...", connect/streaming/query.py:380-383). Its word "error" ended the search with zero counts, so the counts of a file that ran were lost.
  • It now matches the summary line itself: counts such as "1 failed, 2 passed, 3 warnings" or "no tests ran", then "in s", an optional "(h:mm:ss)", and the = rule unless -q drops it. ANSI colors are removed first.

3. docs/installation.md says how to install a package that requires pyspark or pyspark-client.

  • pip installs that distribution over the drop-in's files, as pip does not know that pyspark-client-rust provides the same pyspark package. pip has no option to drop one requirement, and a constraints file cannot do it.
  • The new section gives three ways:
    • pip install --no-deps and then the package's other requirements;
    • the package first, then the drop-in in place of the client, again after each upgrade of the package;
    • uv's --override with a marker that never applies.

Why are the changes needed?

  • So that the offline suites pass in any order of their files.
  • So that the parity gate counts the tests of a file whose output has PySpark's listener warning.
  • So that a user whose project depends on a library that requires PySpark keeps the drop-in.

Does this PR introduce any user-facing change?

No code of the drop-in changes. docs/installation.md has the new section. The test fixture and the parity runner are for contributors.

How was this patch tested?

  • The test order:

    • The order above: 3 failed, 178 passed before; 181 passed after.
    • test_dropin_types_columns_offline.py with test_dropin_offline.py -k "pickle or origin or debugging or call_site", in both file orders: 352 passed and 365 passed.
    • A probe plugin that reports each change of the cache found the test that set it.
    • The offline suite of coverage.yml, in CI's order: 6494 passed, 77 skipped.
    • test_dropin_e2e.py against a Spark 4.3.0 server: 333 passed.
  • The runner: the new ParseCountsTests of scripts/tests/test_run_official_tests.py check:

    • a summary followed by the listener warning;
    • the summary forms: -q, = rules, durations, colors, subtests, xfail and xpass, and no tests;
    • output with no summary line.

    The first test fails without the change. python3 -m unittest discover -s scripts/tests: 112 tests OK.

  • The installation advice: checked with pip 26.3 and uv 0.13.5 in scratch virtual environments. They installed the published pyspark-client-rust 4.3.0, the published pyspark-client 4.3.0, and a package that requires pyspark-client and six. A script compared the drop-in's installed files with the sha256 of its RECORD:

    • plain pip install of the package replaced 91 of the drop-in's files with pyspark-client's;
    • --no-deps and then six kept all of them, upgrades with --no-deps too, and pip check reports the missing pyspark-client;
    • the package first, then pip uninstall -y pyspark pyspark-client and pip install pyspark-client-rust, kept them. A later pip install --force-reinstall of the package replaced them again;
    • a constraints file of pyspark-client==0.0.0 failed with ResolutionImpossible;
    • uv pip install --override with pyspark; sys_platform == "never" and pyspark-client; sys_platform == "never" kept the drop-in and installed six.
  • Gates:

    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;
    • cargo test -p pyspark-rs --lib: 15 passed;
    • ruff 0.14.0, scripts/audit_no_stubs.sh, the stub and dispatch generators' --check;
    • test_type_stubs_offline.py: 13 passed, 3 skipped;
    • the test-list guards.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…only pytest's summary line in run_official_tests.py, and document how to install a package that requires pyspark over the drop-in

Three tests of test_dropin_offline.py failed after test_dropin_types_columns_offline.py, as
their expressions carried call-site origins: PySpark's is_debugging_enabled (vendored verbatim)
keeps its first answer of spark.python.sql.dataFrameDebugging.enabled for the interpreter, and
the first Column of test_the_abstract_types_have_no_proto_nor_arrow_type_as_pysparks, made
with a mock session active, set it to True before test_dropin_offline.py read it without a
session. An autouse fixture of python/tests/conftest.py now leaves the cache, after each test,
as the test found it.

run_official_tests.py's parse_counts took the last line with "passed", "failed", "error" or
"skipped" for pytest's summary, so the "The error is:" line of PySpark's listener-bus warning,
on stderr after the summary, gave zero counts. It now matches pytest's summary line only, with
unit tests.

docs/installation.md says what to do when a package requires pyspark or pyspark-client, which
pip installs over the drop-in: --no-deps, the order of the installs, or uv's overrides.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants