Skip to content

[SPARK-62072][CONNECT][PYTHON] Keep the _name, _result and _error of an Observation in its __dict__, build its plan of its _name each time and set its metrics at the action, and let the plan compression lookup catch a signal handler's Exception, as PySpark - #880

Open
HyukjinKwon wants to merge 2 commits into
apache:masterfrom
HyukjinKwon:SPARK-62072-observation-and-exit-leftovers

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Follow-ups of SPARK-62009 (#853) and SPARK-62005 (#850), each matching pyspark-client 4.3.0, in two commits: the first has items 1 and 2, the second item 3. Details are in jira-SPARK-62072-observation-and-exit-leftovers.txt.

1. The attributes of an Observation. PySpark Connect's Observation is a plain class. Its _name, _result and _error are attributes of its __dict__, which its client sets at each action (connect/client/core.py:1766-1799). The drop-in had them as read-only getters of its native state. Now:

  • The attributes are kept in the __dict__, so vars() shows them, and a program sets and deletes them. __init__ sets them.
  • get raises _error or gives _result, as PySpark's.
  • _on checks _result, so an Observation whose _result was set to None is used again. It names an unnamed Observation by a random UUID, sets _result to a new dict, and observes by a new state of the core.
  • _set_error sets _error.
  • Copies, deep copies and pickles are of the __dict__.
  • At the action, the core's mirror of the Observation, which holds it weakly, reads its _result, converts the metrics and updates _result in place. It gives the error of the server to _set_error. The errors of these steps are the action's, as PySpark's:
    • the AssertionError of a _result that is None;
    • the ValueError of a timestamp of the year 1, which no longer fails get instead;
    • the AttributeError of a deleted _result.
  • _error is one object on every read. A reference cycle through the result dict is freed.
  • An Observation that the client is given for an execution is set by its attributes too.
  • DataFrame.observe gives what the _on of the Observation gives, of a subclass too, as PySpark's does.
  • The core's MetricsMirror now says whether the observation is used. Its update and its error are fallible, and the routing of an execution's observed metrics returns those errors. A new mirror replaces the one before.

2. A signal handler's Exception while a call waits. PySpark's lookup of the plan compression settings (core.py:2714-2740) and its read of the stack-trace settings catch Exception, including one that a SIGINT handler raises while they wait. Its execution interrupts the operation for a KeyboardInterrupt only (core.py:1943-1950). Now:

  • the core runtime has catching, a part of a wait whose caller catches such an exception, and catch_signal, which ends that part;
  • the two lookups run in one. A caught lookup disables the compression for the client, as PySpark's does;
  • the binding's watch of the signals catches a handler's Exception during such a part. For any handler exception other than a KeyboardInterrupt, it ends the wait at once without an interrupt.

So the clean_cache of the exit hook ends after the same number of signals as PySpark's: two when the first arrives during the lookup. The first action of such a session raises the same signal's exception as PySpark.

3. The name of an Observation at each build of its plan. PySpark's CollectMetrics (connect/plan.py:1817-1837) sets str(self._observation._name) in its message each time it builds the plan, after its input. Its observations name the Observation by that str, and the client sets the metrics by it. Observation._on does not read the name. The drop-in's plan, which the core builds, was of the name at observe. Now:

  • The core's MetricsMirror can give the name of its binding's observation as it is now (name, None by default). Observation::name_of_plan asks the mirror of a used observation for it.
  • The build of the CollectMetrics of an Observe node is of that name, else of the node's name. A failed name is the error of the request (record_unbuildable), so the request is not sent, as for the other values that a plan does not take.
  • LogicalPlan::observations names the Observations by that name too. So does the rebuild of a deep copy (udf.rs, PicklerRebuild::observation), which finds the copy of an Observation by it.
  • The binding's mirror reads _name of the Python Observation and takes its str(). A str that UTF-8 cannot encode gives protobuf's UnicodeEncodeError. An error of the read or of __str__ is raised as it is.
  • observe reads no str of the name. The core's state is of the text of a str name, which a plan uses only without the mirror.

So a renamed Observation is sent, explained and set by its new name. 5, None and b"n" are sent as "5", "None" and "b'n'". A raising __str__ or a lone surrogate fails each request until the Observation is renamed again. The number of __str__ calls per request is PySpark's too: 2 for an action and 1 for a schema request.

Docs.

  • docs/compatibility.md describes the attributes of an Observation, the name of each build of its plan, and the signal handling of the waits.
  • dev/parity/known-differences.md: the __dict__ entry says that an Observation keeps its attributes in its __dict__. The first commit listed the renamed Observation as a difference; the second removes that entry.

Why are the changes needed?

So that an Observation behaves as PySpark's. Code that reads vars(o), sets o._result = None to observe again, renames an Observation, or catches the action's error of a metric gets PySpark's behavior. A program whose SIGINT handler raises an Exception sees PySpark's waits.

Does this PR introduce any user-facing change?

Yes, as pyspark-client 4.3.0:

  • vars(Observation("o")) is {'_name': 'o', '_result': None, '_error': None}, and the attributes can be set and deleted;
  • _error is one object;
  • an action raises the error of a metric that does not convert, and get is then {};
  • each request of an observed DataFrame is of str(o._name) as it is then, by which the metrics are set. A raising __str__ or a str that UTF-8 cannot encode fails the request, not observe;
  • the plan compression lookup catches the Exception of a signal handler, and only a KeyboardInterrupt interrupts an operation.

How was this patch tested?

  • New and changed tests, which fail on master:
    • in test_dropin_offline.py:
      • test_the_attributes_of_an_observation_are_those_of_its_dict_as_pyspark;
      • test_observe_gives_the_dataframe_of_the_on_of_an_observation_as_pyspark;
      • test_an_observation_in_a_cycle_through_its_result_is_collected_as_pyspark;
      • test_an_action_converts_the_metrics_of_its_observations_as_pyspark;
      • test_the_plan_compression_lookup_catches_the_exception_of_a_signal_handler_as_pyspark (collect and toLocalIterator);
      • test_the_stack_trace_settings_of_an_error_catch_the_exception_of_a_handler_as_pyspark;
      • test_exception_of_a_signal_handler_is_raised_by_collect_as_pyspark, whose Exception case now asserts no interrupt. A probe of PySpark showed that interrupt_operation is called for a KeyboardInterrupt only;
      • test_the_plan_of_an_observation_is_built_of_its_name_as_it_is_then_as_pyspark (second commit). It covers a rename and the metrics set by the new name, 5, None and bytes as their str, a lone surrogate and a raising __str__ (no request sent), an observe of a raising name, and a deep copy that holds a copy of the Observation by its name. It fails on master, where _name is not writable, and on the first commit ('before' == 'after'). Without the udf.rs change it fails at the deep copy, which held the original Observation;
    • in test_exit_cleanup_offline.py: test_f_the_plan_compression_lookup_of_the_exit_hook_catches_the_exception_of_a_handler;
    • core tests:
      • test_a_catching_scope_ends_for_a_caught_signal_and_the_wait_goes_on;
      • a_signal_that_the_plan_compression_lookup_catches_ends_it_and_the_execution_goes_on;
      • a_mirror_is_told_of_each_update_and_error_and_says_whether_it_is_used;
      • a_plan_of_an_observation_is_built_of_the_name_that_its_mirror_gives_then (second commit), which fails without the plan.rs change ("first" where "now" is expected).
  • Probes of both clients against a Spark 4.3.0 server:
    • p15_observation_attrs.py: 21 of 21 lines identical;
    • agent G's p_ts_observe.py in UTC and Asia/Kolkata: identical;
    • p14_exit_parent.py: both exit hooks end after 2 signals;
    • p14_action.py: both raise the second signal's exception after 2 signals;
    • second commit:
      • p15_rename.py and p15_rename_deepcopy.py: identical, including the plan's name, the metrics, and a deep copy's own Observation;
      • p15_bad_names.py and p15_bad_names2.py: identical for 5, None, bytes, a str subclass, a float, a tuple, "" (the server's Name must not be empty), a lone surrogate, a raising __str__, a deleted _name, and an observe of a raising name;
      • p15_bad_name_requests.py: 36 of 36 lines identical, for 18 request kinds of a lone surrogate and of a raising __str__, with the number of __str__ calls;
      • p15_name_reads.py: identical numbers of __str__ calls for 7 request kinds;
      • p15_reentrant_name.py: identical for a __str__ that runs a request of the session, reads the Observation's get, or observes and collects another DataFrame while the plan is built.
  • Official suites against the drop-in (run_official_tests.py --dropin --only of test_parity_observation, test_connect_session, test_connect_dataframe_property and test_parity_job_cancellation): 13, 10 (17 skipped), 0 (12 skipped) and 8 (1 skipped) passed, the same on master and on both commits.
  • The gates, of both commits on master 3857dae (SPARK-62064 and SPARK-62066 included; the rebase had no conflicts):
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps of the four crates --features apache-spark-connect/wasm-udf;
    • in debug, as the coverage job builds them:
      • all of cargo test -p apache-spark-connect-core: lib 258, server_session 12, reattach_cleanup 19, proxy 20, deadlines 17, artifact_uploads 7, user_context 4 passed (integration_test 3 ignored);
      • cargo test -p apache-spark-connect --lib: 815 passed;
    • cargo test -p pyspark-rs --lib: 15 passed;
    • ruff 0.14.0, the stub and dispatch generators, the guards and test_type_stubs_offline.py (13 passed, 3 skipped);
    • the offline suite of coverage.yml: 6516 passed, 77 skipped;
    • the coverage step of coverage.yml, with its commands and its .coveragerc: 93.44% (floor 93.00%). The second commit changes no Python source;
    • test_dropin_e2e.py against a Spark 4.3.0 server: 335 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

HyukjinKwon and others added 2 commits November 19, 2026 06:12
…an Observation in its __dict__ and set its metrics at the action, and let the plan compression lookup catch a signal handler's Exception, as PySpark

PySpark Connect's Observation (connect/observation.py:34-86) is a plain class: `_name`, `_result` and `_error` are
the attributes of its __dict__, which `get` and `_on` read, which a program sets and deletes, and which its client
sets at each action (connect/client/core.py:1766-1799): it reads the `_result` of each Observation, asserts that it
is not None, converts the metrics and updates it in place, and gives the error of collecting them to `_set_error`.
The drop-in's Observation had them as read-only getters of its native state: vars(o) was empty, they could not be
set, each read of `_error` made a new exception, a reference cycle through the result dict, which the native state
held, was never freed, and the metrics were converted when `get` was read, so a metric that does not convert, such
as a timestamp of the year 1, failed `get` where PySpark fails the action and leaves `get` empty.

PySpark's lookup of the plan compression settings of the first plan (`_get_plan_compression_threshold_and_algorithm`,
core.py:2714-2740) catches `Exception`, also the exception that a signal handler raises while it waits, and its
execution interrupts the operation of a KeyboardInterrupt only (core.py:1943-1950). So a SIGINT handler that raises
an Exception subclass while the exit hook's clean_cache waits does not end the clean_cache when it arrives during
the lookup, and the clean_cache waits again; the drop-in ended it at the first signal, and it interrupted the
operation of any handler's exception, so the next signal only ended the wait of the interrupt.

- The Observation keeps `_name`, `_result` and `_error` in its __dict__: `__init__` sets them, `get` raises `_error`
  or gives `_result` as PySpark's, `_on` checks `_result`, names an unnamed one by a random UUID and sets `_result` to
  a new dict, `_set_error` sets `_error`, and the copies, deep copies and pickles are of the __dict__. Each `_on`
  observes by a new state of the core, whose mirror, which holds the Observation weakly, is asked whether it is
  used, of its `_result`, converts the metrics and updates its `_result` in place, and gives the error of the server
  to its `_set_error`: their errors, such as PySpark's AssertionError of a `_result` that is None or the ValueError
  of a metric that does not convert, are those of the action. An Observation that the client is given for an
  execution is set by its attributes too, and DataFrame.observe gives what the `_on` of an Observation gives, of a
  subclass too.
- The core's MetricsMirror says whether the observation is used, and its update and its error are fallible, which
  the routing of the observed metrics of an execution returns; a mirror replaces the one before.
- The core runtime has `catching`, a part of a wait whose caller catches the exception of a signal handler that is
  an Exception, and `catch_signal`, which ends it. The lookup of the plan compression settings, which then disables
  the compression for the client, and the read of the stack-trace settings of an error run in one. The binding's
  watch of the signals catches the exception of a handler that is an Exception there, and ends the wait at once,
  without an interrupt, for the exception of a handler that is not a KeyboardInterrupt.
- docs/compatibility.md describes the attributes of an Observation and the signal handling of the waits;
  known-differences.md notes that an Observation renamed after `observe` keeps its first name in the plan.

Co-authored-by: Isaac <no-reply@databricks.com>
…tion of the str() of its _name each time, as PySpark

PySpark's CollectMetrics (connect/plan.py:1817-1837) sets `str(self._observation._name)` in the name of its message
each time it builds the plan, after its input, and its `observations` name the Observation by that str, by which the
client sets the metrics. Observation._on does not read the name. So an Observation renamed after `observe`
(`o._name = "other"`) is sent, explained and set by its new name, a name of another object is sent as its str (5 as
"5", None as "None", b"n" as "b'n'"), and a name whose `__str__` raises, or a str that UTF-8 cannot encode, fails
each request with that error, the exception of the `__str__` or protobuf's UnicodeEncodeError, until it is renamed
again. The drop-in built the plan of the name that the Observation had at `observe`, which the Rust core kept in the
plan: the server saw the first name, and a name that was not a str was taken as protobuf takes a string field, bytes
as their text and another object as protobuf's TypeError of each request, where PySpark sends its str(). The first
commit of this PR noted the renamed Observation as a known difference.

- The core's MetricsMirror can give the name of the observation of its binding as it is now (`name`, None by
  default), and `Observation::name_of_plan` asks the mirror of a used observation for it. The build of the
  CollectMetrics of an `Observe` node, after its input, is of that name, else of the name of the node, and its error
  is the error of the request (`record_unbuildable`), as of the other values that a plan does not take, so the
  request is not sent. `LogicalPlan::observations` names the Observations by that name too, and so does the rebuild
  of a deep copy, which finds the copy of an Observation by it.
- The binding's mirror reads `_name` from the Python Observation, which it holds weakly, takes its `str()`, and gives
  the error of a str that UTF-8 cannot encode as protobuf's, and an error of the read or of `__str__` as it is.
  `observe` reads no str of the name; the state of the core is of the text of a str name.
- known-differences.md no longer lists the renamed Observation; docs/compatibility.md describes the name of each
  build.

Tests: the offline test_the_plan_of_an_observation_is_built_of_its_name_as_it_is_then_as_pyspark sends the renamed
Observation's new name and sets its metrics by it, sends 5, None and bytes as their str, fails the requests of a
lone surrogate and of a raising `__str__` without sending them, observes by an Observation of a raising name, and
deep-copies the DataFrame of the renamed Observation; the core test
a_plan_of_an_observation_is_built_of_the_name_that_its_mirror_gives_then builds an Observe node of a mirror's name
and error. Probes on both clients (rename, bad names, each request kind of a bad name with the number of `__str__`
calls, deep copy) are identical.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant