Document supported databricks_retry_args usage for deferrable Databricks operators#68017
Open
kosiew wants to merge 2 commits into
Open
Document supported databricks_retry_args usage for deferrable Databricks operators#68017kosiew wants to merge 2 commits into
databricks_retry_args usage for deferrable Databricks operators#68017kosiew wants to merge 2 commits into
Conversation
…ferrable operators Add "Retry args in deferrable mode" subsection under DatabricksSubmitRunDeferrableOperator and DatabricksRunNowDeferrableOperator explaining: - Serialization requirement: only plain Python primitives allowed across the trigger boundary - Supported shapes (int/float primitives, nested plain-dict) - Unsupported shapes (Tenacity objects, callables) with note that a ValueError is raised at task submission - Recommended workaround: use non-deferrable mode for custom retry strategies Also update changelog for 7.16.0.
kosiew
force-pushed
the
callable-deserialization-02-64609
branch
from
8b7caa8 to
283df3e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Follow up for #64960
Adds documentation clarifying which
databricks_retry_argsconfigurations are supported when using Databricks operators in deferrable mode.The new documentation:
databricks_retry_argsmust be serialization-safe because it is serialized across the trigger boundary whendeferrable=True.{"reraise": True}.retry_limitandretry_delayoperator parameters.stop_after_attempt,wait_incrementing, etc.) and arbitrary callables.Does this PR introduce any user-facing change?
Yes. Documentation now explicitly describes the serialization requirements and supported shapes for
databricks_retry_argsin deferrable Databricks operators, helping users avoid unsupported retry configurations.How was this patch tested?
No tests were added or modified. This PR contains documentation and changelog updates only.
Was generative AI tooling used to co-author this PR?
ChatGPT