Repository navigation
[SPARK-62072][CONNECT][PYTHON] Keep the _name, _result and _error of an Observation in its __dict__, build its plan of its _name each time and set its metrics at the action, and let the plan compression lookup catch a signal handler's Exception, as PySpark - #880
Open
HyukjinKwon wants to merge 2 commits into
Open
HyukjinKwon wants to merge 2 commits into
HyukjinKwon wants to merge 2 commits into
Conversation
…an Observation in its __dict__ and set its metrics at the action, and let the plan compression lookup catch a signal handler's Exception, as PySpark PySpark Connect's Observation (connect/observation.py:34-86) is a plain class: `_name`, `_result` and `_error` are the attributes of its __dict__, which `get` and `_on` read, which a program sets and deletes, and which its client sets at each action (connect/client/core.py:1766-1799): it reads the `_result` of each Observation, asserts that it is not None, converts the metrics and updates it in place, and gives the error of collecting them to `_set_error`. The drop-in's Observation had them as read-only getters of its native state: vars(o) was empty, they could not be set, each read of `_error` made a new exception, a reference cycle through the result dict, which the native state held, was never freed, and the metrics were converted when `get` was read, so a metric that does not convert, such as a timestamp of the year 1, failed `get` where PySpark fails the action and leaves `get` empty. PySpark's lookup of the plan compression settings of the first plan (`_get_plan_compression_threshold_and_algorithm`, core.py:2714-2740) catches `Exception`, also the exception that a signal handler raises while it waits, and its execution interrupts the operation of a KeyboardInterrupt only (core.py:1943-1950). So a SIGINT handler that raises an Exception subclass while the exit hook's clean_cache waits does not end the clean_cache when it arrives during the lookup, and the clean_cache waits again; the drop-in ended it at the first signal, and it interrupted the operation of any handler's exception, so the next signal only ended the wait of the interrupt. - The Observation keeps `_name`, `_result` and `_error` in its __dict__: `__init__` sets them, `get` raises `_error` or gives `_result` as PySpark's, `_on` checks `_result`, names an unnamed one by a random UUID and sets `_result` to a new dict, `_set_error` sets `_error`, and the copies, deep copies and pickles are of the __dict__. Each `_on` observes by a new state of the core, whose mirror, which holds the Observation weakly, is asked whether it is used, of its `_result`, converts the metrics and updates its `_result` in place, and gives the error of the server to its `_set_error`: their errors, such as PySpark's AssertionError of a `_result` that is None or the ValueError of a metric that does not convert, are those of the action. An Observation that the client is given for an execution is set by its attributes too, and DataFrame.observe gives what the `_on` of an Observation gives, of a subclass too. - The core's MetricsMirror says whether the observation is used, and its update and its error are fallible, which the routing of the observed metrics of an execution returns; a mirror replaces the one before. - The core runtime has `catching`, a part of a wait whose caller catches the exception of a signal handler that is an Exception, and `catch_signal`, which ends it. The lookup of the plan compression settings, which then disables the compression for the client, and the read of the stack-trace settings of an error run in one. The binding's watch of the signals catches the exception of a handler that is an Exception there, and ends the wait at once, without an interrupt, for the exception of a handler that is not a KeyboardInterrupt. - docs/compatibility.md describes the attributes of an Observation and the signal handling of the waits; known-differences.md notes that an Observation renamed after `observe` keeps its first name in the plan. Co-authored-by: Isaac <no-reply@databricks.com>
…tion of the str() of its _name each time, as PySpark PySpark's CollectMetrics (connect/plan.py:1817-1837) sets `str(self._observation._name)` in the name of its message each time it builds the plan, after its input, and its `observations` name the Observation by that str, by which the client sets the metrics. Observation._on does not read the name. So an Observation renamed after `observe` (`o._name = "other"`) is sent, explained and set by its new name, a name of another object is sent as its str (5 as "5", None as "None", b"n" as "b'n'"), and a name whose `__str__` raises, or a str that UTF-8 cannot encode, fails each request with that error, the exception of the `__str__` or protobuf's UnicodeEncodeError, until it is renamed again. The drop-in built the plan of the name that the Observation had at `observe`, which the Rust core kept in the plan: the server saw the first name, and a name that was not a str was taken as protobuf takes a string field, bytes as their text and another object as protobuf's TypeError of each request, where PySpark sends its str(). The first commit of this PR noted the renamed Observation as a known difference. - The core's MetricsMirror can give the name of the observation of its binding as it is now (`name`, None by default), and `Observation::name_of_plan` asks the mirror of a used observation for it. The build of the CollectMetrics of an `Observe` node, after its input, is of that name, else of the name of the node, and its error is the error of the request (`record_unbuildable`), as of the other values that a plan does not take, so the request is not sent. `LogicalPlan::observations` names the Observations by that name too, and so does the rebuild of a deep copy, which finds the copy of an Observation by it. - The binding's mirror reads `_name` from the Python Observation, which it holds weakly, takes its `str()`, and gives the error of a str that UTF-8 cannot encode as protobuf's, and an error of the read or of `__str__` as it is. `observe` reads no str of the name; the state of the core is of the text of a str name. - known-differences.md no longer lists the renamed Observation; docs/compatibility.md describes the name of each build. Tests: the offline test_the_plan_of_an_observation_is_built_of_its_name_as_it_is_then_as_pyspark sends the renamed Observation's new name and sets its metrics by it, sends 5, None and bytes as their str, fails the requests of a lone surrogate and of a raising `__str__` without sending them, observes by an Observation of a raising name, and deep-copies the DataFrame of the renamed Observation; the core test a_plan_of_an_observation_is_built_of_the_name_that_its_mirror_gives_then builds an Observe node of a mirror's name and error. Probes on both clients (rename, bad names, each request kind of a bad name with the number of `__str__` calls, deep copy) are identical. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Follow-ups of SPARK-62009 (#853) and SPARK-62005 (#850), each matching pyspark-client 4.3.0, in two commits: the first has items 1 and 2, the second item 3. Details are in
jira-SPARK-62072-observation-and-exit-leftovers.txt.1. The attributes of an Observation. PySpark Connect's Observation is a plain class. Its
_name,_resultand_errorare attributes of its__dict__, which its client sets at each action (connect/client/core.py:1766-1799). The drop-in had them as read-only getters of its native state. Now:__dict__, sovars()shows them, and a program sets and deletes them.__init__sets them.getraises_erroror gives_result, as PySpark's._onchecks_result, so an Observation whose_resultwas set to None is used again. It names an unnamed Observation by a random UUID, sets_resultto a new dict, and observes by a new state of the core._set_errorsets_error.__dict__._result, converts the metrics and updates_resultin place. It gives the error of the server to_set_error. The errors of these steps are the action's, as PySpark's:AssertionErrorof a_resultthat is None;ValueErrorof a timestamp of the year 1, which no longer failsgetinstead;AttributeErrorof a deleted_result._erroris one object on every read. A reference cycle through the result dict is freed.DataFrame.observegives what the_onof the Observation gives, of a subclass too, as PySpark's does.MetricsMirrornow says whether the observation is used. Its update and its error are fallible, and the routing of an execution's observed metrics returns those errors. A new mirror replaces the one before.2. A signal handler's
Exceptionwhile a call waits. PySpark's lookup of the plan compression settings (core.py:2714-2740) and its read of the stack-trace settings catchException, including one that a SIGINT handler raises while they wait. Its execution interrupts the operation for aKeyboardInterruptonly (core.py:1943-1950). Now:catching, a part of a wait whose caller catches such an exception, andcatch_signal, which ends that part;Exceptionduring such a part. For any handler exception other than aKeyboardInterrupt, it ends the wait at once without an interrupt.So the
clean_cacheof the exit hook ends after the same number of signals as PySpark's: two when the first arrives during the lookup. The first action of such a session raises the same signal's exception as PySpark.3. The name of an Observation at each build of its plan. PySpark's
CollectMetrics(connect/plan.py:1817-1837) setsstr(self._observation._name)in its message each time it builds the plan, after its input. Itsobservationsname the Observation by that str, and the client sets the metrics by it.Observation._ondoes not read the name. The drop-in's plan, which the core builds, was of the name atobserve. Now:MetricsMirrorcan give the name of its binding's observation as it is now (name, None by default).Observation::name_of_planasks the mirror of a used observation for it.CollectMetricsof anObservenode is of that name, else of the node's name. A failed name is the error of the request (record_unbuildable), so the request is not sent, as for the other values that a plan does not take.LogicalPlan::observationsnames the Observations by that name too. So does the rebuild of a deep copy (udf.rs,PicklerRebuild::observation), which finds the copy of an Observation by it._nameof the Python Observation and takes itsstr(). A str that UTF-8 cannot encode gives protobuf'sUnicodeEncodeError. An error of the read or of__str__is raised as it is.observereads no str of the name. The core's state is of the text of a str name, which a plan uses only without the mirror.So a renamed Observation is sent, explained and set by its new name. 5, None and
b"n"are sent as"5","None"and"b'n'". A raising__str__or a lone surrogate fails each request until the Observation is renamed again. The number of__str__calls per request is PySpark's too: 2 for an action and 1 for a schema request.Docs.
docs/compatibility.mddescribes the attributes of an Observation, the name of each build of its plan, and the signal handling of the waits.dev/parity/known-differences.md: the__dict__entry says that an Observation keeps its attributes in its__dict__. The first commit listed the renamed Observation as a difference; the second removes that entry.Why are the changes needed?
So that an Observation behaves as PySpark's. Code that reads
vars(o), setso._result = Noneto observe again, renames an Observation, or catches the action's error of a metric gets PySpark's behavior. A program whose SIGINT handler raises anExceptionsees PySpark's waits.Does this PR introduce any user-facing change?
Yes, as pyspark-client 4.3.0:
vars(Observation("o"))is{'_name': 'o', '_result': None, '_error': None}, and the attributes can be set and deleted;_erroris one object;getis then{};str(o._name)as it is then, by which the metrics are set. A raising__str__or a str that UTF-8 cannot encode fails the request, notobserve;Exceptionof a signal handler, and only aKeyboardInterruptinterrupts an operation.How was this patch tested?
test_dropin_offline.py:test_the_attributes_of_an_observation_are_those_of_its_dict_as_pyspark;test_observe_gives_the_dataframe_of_the_on_of_an_observation_as_pyspark;test_an_observation_in_a_cycle_through_its_result_is_collected_as_pyspark;test_an_action_converts_the_metrics_of_its_observations_as_pyspark;test_the_plan_compression_lookup_catches_the_exception_of_a_signal_handler_as_pyspark(collect and toLocalIterator);test_the_stack_trace_settings_of_an_error_catch_the_exception_of_a_handler_as_pyspark;test_exception_of_a_signal_handler_is_raised_by_collect_as_pyspark, whoseExceptioncase now asserts no interrupt. A probe of PySpark showed thatinterrupt_operationis called for aKeyboardInterruptonly;test_the_plan_of_an_observation_is_built_of_its_name_as_it_is_then_as_pyspark(second commit). It covers a rename and the metrics set by the new name, 5, None and bytes as their str, a lone surrogate and a raising__str__(no request sent), anobserveof a raising name, and a deep copy that holds a copy of the Observation by its name. It fails on master, where_nameis not writable, and on the first commit ('before' == 'after'). Without theudf.rschange it fails at the deep copy, which held the original Observation;test_exit_cleanup_offline.py:test_f_the_plan_compression_lookup_of_the_exit_hook_catches_the_exception_of_a_handler;test_a_catching_scope_ends_for_a_caught_signal_and_the_wait_goes_on;a_signal_that_the_plan_compression_lookup_catches_ends_it_and_the_execution_goes_on;a_mirror_is_told_of_each_update_and_error_and_says_whether_it_is_used;a_plan_of_an_observation_is_built_of_the_name_that_its_mirror_gives_then(second commit), which fails without theplan.rschange ("first"where"now"is expected).p15_observation_attrs.py: 21 of 21 lines identical;p_ts_observe.pyin UTC and Asia/Kolkata: identical;p14_exit_parent.py: both exit hooks end after 2 signals;p14_action.py: both raise the second signal's exception after 2 signals;p15_rename.pyandp15_rename_deepcopy.py: identical, including the plan's name, the metrics, and a deep copy's own Observation;p15_bad_names.pyandp15_bad_names2.py: identical for 5, None, bytes, a str subclass, a float, a tuple,""(the server'sName must not be empty), a lone surrogate, a raising__str__, a deleted_name, and anobserveof a raising name;p15_bad_name_requests.py: 36 of 36 lines identical, for 18 request kinds of a lone surrogate and of a raising__str__, with the number of__str__calls;p15_name_reads.py: identical numbers of__str__calls for 7 request kinds;p15_reentrant_name.py: identical for a__str__that runs a request of the session, reads the Observation'sget, or observes and collects another DataFrame while the plan is built.run_official_tests.py --dropin --onlyoftest_parity_observation,test_connect_session,test_connect_dataframe_propertyandtest_parity_job_cancellation): 13, 10 (17 skipped), 0 (12 skipped) and 8 (1 skipped) passed, the same on master and on both commits.cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-depsof the four crates--features apache-spark-connect/wasm-udf;cargo test -p apache-spark-connect-core: lib 258,server_session12,reattach_cleanup19,proxy20,deadlines17,artifact_uploads7,user_context4 passed (integration_test3 ignored);cargo test -p apache-spark-connect --lib: 815 passed;cargo test -p pyspark-rs --lib: 15 passed;test_type_stubs_offline.py(13 passed, 3 skipped);coverage.yml: 6516 passed, 77 skipped;coverage.yml, with its commands and its.coveragerc: 93.44% (floor 93.00%). The second commit changes no Python source;test_dropin_e2e.pyagainst a Spark 4.3.0 server: 335 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.