Repository navigation
Python: fix(workflow): keep a sibling pause when a nested step failure was handled - #9571
H23 (Madin-H23) wants to merge 2 commits into
Conversation
…e was handled Closes microsoft#9549 _settle_workflow_tasks decided whether a failed task's exception escaped the workflow by looking only at that task's direct concurrency waiter. A step that runs as a task inside another step fails inside the inner boundary, so that waiter keeps holding the exception even after the awaiting code caught it, and the already-handled failure replaced a sibling branch's pending request. A boundary failure counts as handled only when the waiter failed with that very exception, so the failure really reached the awaiting code, and the task that awaited the waiter completed without raising it, so the awaiting code caught it. A failure that was never delivered still surfaces.
There was a problem hiding this comment.
🟡 Changes recommended
Failure-delivery detection can suppress unhandled errors and still misses handled errors across additional nesting.
3 open issues
What changed in this PR
Addresses #9549 by allowing Python functional workflows to preserve a sibling pause after catching a nested step failure.
Changes:
- Adds a check for failures handled by an awaiting task.
- Adds regression coverage for pausing and resuming after a handled nested failure.
| File | Description |
|---|---|
| python/packages/core/tests/workflow/test_functional_workflow.py | Adds nested-failure pause/resume coverage. |
| python/packages/core/agent_framework/_workflows/_functional.py | Updates failure handling during pause settlement. |
Review effort: Balanced
| awaiter = self._workflow_task_parents.get(task) | ||
| if awaiter is None or not awaiter.done() or awaiter.cancelled(): | ||
| return False | ||
| return awaiter.exception() is None |
| awaiter = self._workflow_task_parents.get(task) | ||
| if awaiter is None or not awaiter.done() or awaiter.cancelled(): | ||
| return False | ||
| return awaiter.exception() is None |
| awaiter = self._workflow_task_parents.get(task) | ||
| if awaiter is None or not awaiter.done() or awaiter.cancelled(): | ||
| return False | ||
| return awaiter.exception() is None |
|
Thanks for the review — I verified all three findings against the current implementation instead of taking them at face value, and two of them hold. How I checked: I read the head of this branch ( Finding 2 — creation ancestry misidentifies gather failure handling — confirmed, and it is a regression from this PR. Finding 1 — cancellation can hide undelivered child failures — confirmed, with a narrow trigger. Finding 3 — nested waiter handlers are not detected — correct mechanism, but pre-existing. Next: I will push the two fixes above together with the regression tests (detached gather, cancellation window) and update the description to match the final rule. Please focus the review on how delivery to the awaiter is established. One process note: the |
|
@microsoft-github-policy-service agree |
…ailure as handled The first version of this fix decided that a boundary failure was handled when the waiter held the very same exception and the task that created the boundary finished without raising. That is not enough. The task that created a gather is not necessarily the task that awaited it, so a gather nobody awaited was taken for a handled one and its failure was swallowed. And a cancellation between the gather finishing and the awaiting task resuming means the awaiting task received CancelledError instead of the failure, so the failure was never delivered at all. A failure now counts as handled only when it was delivered. A future's exception is retrieved exactly when the awaiting code receives it, and retrieving it clears the future's traceback flag, so a waiter that still carries the flag never delivered its exception. The awaiting task must also not have been cancelled, because cancelling a waiter clears that flag too. Two regression tests cover the gaps: a gather nobody awaited, and an awaiting task that is cancelled between the gather finishing and its resumption.
|
The automated review found two real gaps in the first version of this patch, and the second commit fixes both. Details of what changed and why: Finding — "creation ancestry misidentifies gather failure handling" (confirmed, and a regression from my first version). Finding — "cancellation can hide undelivered child failures" (confirmed). The first version inferred delivery from the waiter holding the same exception. That is not enough: cancelling a waiter clears its traceback flag, and a task cancelled between the gather finishing and its resumption receives The rule now is: a boundary failure counts as handled only when it was actually delivered. A future's exception is retrieved exactly when the awaiting code receives it, and retrieving it clears the future's traceback flag — so a waiter that still carries the flag never delivered its exception. The awaiting task must also not have been cancelled, because cancelling a waiter clears that flag as well. Coverage added: One limitation, stated plainly: the cancellation case is detected through The third finding (a deeper nesting that
|
|
Closing to finalize the design first. |


Motivation & Context
A functional workflow that runs a step as a task inside another step re-raises the step failure — even when the workflow's own code caught it — as soon as a sibling branch pauses for
request_info. The workflow then cannot be paused or resumed.left()handles the failure, but the firstrun()raisesBoominstead of returning the pendingreviewrequest, so the workflow can never be answered. The same code works on 1.20.0 and 1.21.0; the pause-settlement rewrite in 1.22.0 introduced the regression.The cause is in
_settle_workflow_tasks→_raise_completed_failure: whether a failed task's exception escaped the workflow is decided by looking only at that task's direct concurrency waiter. For a step that runs as a task inside another step the exception is raised inside the inner boundary, so that waiter keeps holding it even after the awaiting code caught it. The awaiting task then completes normally, which is exactly the evidence that the failure never escaped the workflow's own code.Description & Review Guide
What are the major changes?
_raise_completed_failurenow also asks_failure_was_handled_by_awaiterwhether the failure was already handled. A boundary task's failure counts as handled when all three hold:What is the impact of these changes?
A workflow whose own code handles a nested step failure can pause and resume again instead of failing the run. A failure that was never delivered still surfaces, so an unhandled sibling failure keeps replacing the pause exactly as before.
What do you want reviewers to focus on?
The delivery condition, because the other two are not enough on their own.
Checking only "the awaiting task completed cleanly" swallows failures that were never delivered: the task that created a gather is not necessarily the task that awaited it, so a gather nobody awaited was taken for a handled one, and a cancellation between the gather finishing and the awaiting task resuming means the awaiting task received
CancelledErrorinstead of the failure. Both gaps are covered by new regression tests —test_a_gather_nobody_awaited_still_surfaces_its_failureandtest_a_cancellation_before_the_failure_is_delivered_still_surfaces_it— and the existingtest_swallowed_pause_still_surfaces_unhandled_sibling_failureandtest_nested_unhandled_sibling_failure_replaces_pausecases keep the unhandled sibling failure surfacing.One limitation worth stating: the cancellation case is detected through
Task.cancelling(), which exists from Python 3.11, so that test is skipped on 3.10 and the cancellation window stays undetectable there.Related Issue
Fixes #9549
The direction follows the expectation stated in the issue: a failure the workflow's own code already handled must not be re-raised, while a failure that was never delivered still surfaces. The issue does not carry the
ready-for-implementationlabel yet. We asked on the issue on 2026-10-22, and we will close this pull request if there is no confirmation by 2026-10-27.AI Assistance
AI assistance details: the issue analysis, the implementation in
agent_framework/_workflows/_functional.py, the regression test, and this description were drafted with an AI coding agent (MiniMax M3.1 Flash via the ZCode CLI) under the repository's AI-Assisted Contributions Policy. What I verified myself: the failure reproduces on the currentmain(the run raisesNestedStepErrorinstead of returning the pending request); the new testtest_nested_handled_step_failure_does_not_replace_sibling_pausefails without the change and passes with it, while the existing sibling-failure tests keep passing unchanged; the two delivery-gap tests fail against the first version of this fix and pass against the current one;packages/core/tests/workflowis green locally (1278 passed, 6 skipped, 1 xfailed);ruff checkandruff format --checkwith the pinned 0.17.0 are clean on both changed files;pyrightreports the same 11 pre-existing diagnostics before and after the change; and the MCP-area tests that fail in my local Windows environment fail identically on the unmodified tree (missing optional dependencies, unrelated to this change).Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.