Skip to content

Python: fix(workflow): keep a sibling pause when a nested step failure was handled - #9571

Closed
H23 (Madin-H23) wants to merge 2 commits into
microsoft:mainfrom
Madin-H23:fix/workflow-handled-nested-step-failure
Closed

H23 (Madin-H23) wants to merge 2 commits into
microsoft:mainfrom
Madin-H23:fix/workflow-handled-nested-step-failure

Conversation

@Madin-H23

@Madin-H23 H23 (Madin-H23) commented Oct 21, 2026 •

Copy link
Copy Markdown

Motivation & Context

A functional workflow that runs a step as a task inside another step re-raises the step failure — even when the workflow's own code caught it — as soon as a sibling branch pauses for request_info. The workflow then cannot be paused or resumed.

@step
async def boom() -> int:
    raise Boom("boom")

@step
async def outer() -> int:
    (value,) = await asyncio.gather(boom())   # a step run as a task inside a step
    return value

async def left() -> int:
    try:
        return await outer()
    except Boom:
        return -1                              # handled by the workflow's own code

@workflow
async def flow(x: int) -> list:
    return list(await asyncio.gather(left(), ask("review")))   # ask() pauses

left() handles the failure, but the first run() raises Boom instead of returning the pending review request, so the workflow can never be answered. The same code works on 1.20.0 and 1.21.0; the pause-settlement rewrite in 1.22.0 introduced the regression.

The cause is in _settle_workflow_tasks → _raise_completed_failure: whether a failed task's exception escaped the workflow is decided by looking only at that task's direct concurrency waiter. For a step that runs as a task inside another step the exception is raised inside the inner boundary, so that waiter keeps holding it even after the awaiting code caught it. The awaiting task then completes normally, which is exactly the evidence that the failure never escaped the workflow's own code.

Description & Review Guide

  • What are the major changes?

    _raise_completed_failure now also asks _failure_was_handled_by_awaiter whether the failure was already handled. A boundary task's failure counts as handled when all three hold:

    1. the waiter failed with that very exception, so the failure reached the boundary,
    2. the failure was delivered: a future's exception is retrieved exactly when the awaiting code receives it, and retrieving it clears the future's traceback flag, so a waiter that still carries the flag never delivered its exception, and
    3. the task that awaited the waiter completed without raising it and was never cancelled, so the awaiting code caught it.
  • What is the impact of these changes?

    A workflow whose own code handles a nested step failure can pause and resume again instead of failing the run. A failure that was never delivered still surfaces, so an unhandled sibling failure keeps replacing the pause exactly as before.

  • What do you want reviewers to focus on?

    The delivery condition, because the other two are not enough on their own.

    Checking only "the awaiting task completed cleanly" swallows failures that were never delivered: the task that created a gather is not necessarily the task that awaited it, so a gather nobody awaited was taken for a handled one, and a cancellation between the gather finishing and the awaiting task resuming means the awaiting task received CancelledError instead of the failure. Both gaps are covered by new regression tests — test_a_gather_nobody_awaited_still_surfaces_its_failure and test_a_cancellation_before_the_failure_is_delivered_still_surfaces_it — and the existing test_swallowed_pause_still_surfaces_unhandled_sibling_failure and test_nested_unhandled_sibling_failure_replaces_pause cases keep the unhandled sibling failure surfacing.

    One limitation worth stating: the cancellation case is detected through Task.cancelling(), which exists from Python 3.11, so that test is skipped on 3.10 and the cancellation window stays undetectable there.

Related Issue

Fixes #9549

The direction follows the expectation stated in the issue: a failure the workflow's own code already handled must not be re-raised, while a failure that was never delivered still surfaces. The issue does not carry the ready-for-implementation label yet. We asked on the issue on 2026-10-22, and we will close this pull request if there is no confirmation by 2026-10-27.

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: the issue analysis, the implementation in agent_framework/_workflows/_functional.py, the regression test, and this description were drafted with an AI coding agent (MiniMax M3.1 Flash via the ZCode CLI) under the repository's AI-Assisted Contributions Policy. What I verified myself: the failure reproduces on the current main (the run raises NestedStepError instead of returning the pending request); the new test test_nested_handled_step_failure_does_not_replace_sibling_pause fails without the change and passes with it, while the existing sibling-failure tests keep passing unchanged; the two delivery-gap tests fail against the first version of this fix and pass against the current one; packages/core/tests/workflow is green locally (1278 passed, 6 skipped, 1 xfailed); ruff check and ruff format --check with the pinned 0.17.0 are clean on both changed files; pyright reports the same 11 pre-existing diagnostics before and after the change; and the MCP-area tests that fail in my local Windows environment fail identically on the unmodified tree (missing optional dependencies, unrelated to this change).

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

…e was handled

Closes microsoft#9549

_settle_workflow_tasks decided whether a failed task's exception escaped the
workflow by looking only at that task's direct concurrency waiter. A step
that runs as a task inside another step fails inside the inner boundary, so
that waiter keeps holding the exception even after the awaiting code caught
it, and the already-handled failure replaced a sibling branch's pending
request.

A boundary failure counts as handled only when the waiter failed with that
very exception, so the failure really reached the awaiting code, and the task
that awaited the waiter completed without raising it, so the awaiting code
caught it. A failure that was never delivered still surfaces.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failure-delivery detection can suppress unhandled errors and still misses handled errors across additional nesting.

3 open issues
What changed in this PR

Addresses #9549 by allowing Python functional workflows to preserve a sibling pause after catching a nested step failure.

Changes:

  • Adds a check for failures handled by an awaiting task.
  • Adds regression coverage for pausing and resuming after a handled nested failure.
File Description
python/​packages/​core/​tests/​workflow/​test_functional_workflow.py Adds nested-failure pause/resume coverage.
python/​packages/​core/​agent_framework/​_workflows/​_functional.py Updates failure handling during pause settlement.

Review effort: Balanced

Comment on lines +1034 to +1037
awaiter = self._workflow_task_parents.get(task)
if awaiter is None or not awaiter.done() or awaiter.cancelled():
return False
return awaiter.exception() is None
Comment on lines +1034 to +1037
awaiter = self._workflow_task_parents.get(task)
if awaiter is None or not awaiter.done() or awaiter.cancelled():
return False
return awaiter.exception() is None
Comment on lines +1034 to +1037
awaiter = self._workflow_task_parents.get(task)
if awaiter is None or not awaiter.done() or awaiter.cancelled():
return False
return awaiter.exception() is None
@Madin-H23

Copy link
Copy Markdown
Author

Thanks for the review — I verified all three findings against the current implementation instead of taking them at face value, and two of them hold.

How I checked: I read the head of this branch (_functional.py:1011-1037 plus the call site at 1186-1188) and reproduced each scenario in a standalone simulation that mirrors the task/awaiter bookkeeping and the settlement path, comparing the result with the same simulation running the main version of the helper. I did not run the repository test suite locally.

Finding 2 — creation ancestry misidentifies gather failure handling — confirmed, and it is a regression from this PR.
_workflow_task_parents records the creator of a task (_create_task), but _failure_was_handled_by_awaiter uses it as the awaiter. A helper that starts a gather (one failing child plus one request_info child) and then returns only the request child leaves that gather unawaited; the helper returns cleanly, so the "the awaiting task completed without raising" condition holds for the wrong task and the failing child's exception is suppressed. In my simulation main raises RuntimeError while this branch pauses and the failure is lost entirely — which is the opposite of the invariant this PR states in its description ("a failure that was never delivered still surfaces"). I will replace creation ancestry with the task that actually awaits the boundary waiter, require that the waiter's exception was really retrieved by that task, and add a regression test for the detached-gather case.

Finding 1 — cancellation can hide undelivered child failures — confirmed, with a narrow trigger.
Task.cancel() returns False when the task's _fut_waiter is already done, so the task is resumed with CancelledError instead of the child's exception, while the gather keeps the exception. The first condition ("the waiter holds that very exception") is therefore satisfied without the awaiting code ever seeing it. If that code catches the cancellation and returns, the second condition holds too and an undelivered failure is treated as handled. The trigger needs the cancellation to land between gather completion and the parent's resumption, and the awaiting code to swallow CancelledError. I will make the first condition assert actual delivery to the awaiter rather than inferring it from the waiter's exception, and add a regression test for that window.

Finding 3 — nested waiter handlers are not detected — correct mechanism, but pre-existing.
With the extra nesting, the innermost failing task's recorded parent is the failed intermediate task, so the helper returns False and the error is re-raised. That is the same on main as on this branch, so this PR neither introduces nor worsens it; the literal example also cannot reach the path, because a new step is refused once the workflow has paused. I will treat it as a separate coverage issue rather than block this PR on it.

Next: I will push the two fixes above together with the regression tests (detached gather, cancellation window) and update the description to match the final rule. Please focus the review on how delivery to the awaiter is established.

One process note: the license/cla check is still pending on my side — I am completing that agreement separately, and it is not part of the code review.

@Madin-H23

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

…ailure as handled

The first version of this fix decided that a boundary failure was handled
when the waiter held the very same exception and the task that created the
boundary finished without raising. That is not enough. The task that created
a gather is not necessarily the task that awaited it, so a gather nobody
awaited was taken for a handled one and its failure was swallowed. And a
cancellation between the gather finishing and the awaiting task resuming
means the awaiting task received CancelledError instead of the failure, so
the failure was never delivered at all.

A failure now counts as handled only when it was delivered. A future's
exception is retrieved exactly when the awaiting code receives it, and
retrieving it clears the future's traceback flag, so a waiter that still
carries the flag never delivered its exception. The awaiting task must also
not have been cancelled, because cancelling a waiter clears that flag too.

Two regression tests cover the gaps: a gather nobody awaited, and an
awaiting task that is cancelled between the gather finishing and its
resumption.
@Madin-H23
H23 (Madin-H23) deployed to github-app-auth October 21, 2026 08:01 — with GitHub Actions Active
@Madin-H23
H23 (Madin-H23) deployed to github-app-auth October 21, 2026 08:02 — with GitHub Actions Active
@Madin-H23

Copy link
Copy Markdown
Author

The automated review found two real gaps in the first version of this patch, and the second commit fixes both. Details of what changed and why:

Finding — "creation ancestry misidentifies gather failure handling" (confirmed, and a regression from my first version). _workflow_task_parents records the task that created a task, not the task that awaited it. My helper used it as the awaiter, so a gather that nobody awaited was taken for a handled one: the creating task returned normally, the failing child's exception was swallowed, and the run paused as if nothing had failed. The fix no longer infers delivery from that relationship.

Finding — "cancellation can hide undelivered child failures" (confirmed). The first version inferred delivery from the waiter holding the same exception. That is not enough: cancelling a waiter clears its traceback flag, and a task cancelled between the gather finishing and its resumption receives CancelledError instead of the child's error.

The rule now is: a boundary failure counts as handled only when it was actually delivered. A future's exception is retrieved exactly when the awaiting code receives it, and retrieving it clears the future's traceback flag — so a waiter that still carries the flag never delivered its exception. The awaiting task must also not have been cancelled, because cancelling a waiter clears that flag as well.

Coverage added: test_a_gather_nobody_awaited_still_surfaces_its_failure and test_a_cancellation_before_the_failure_is_delivered_still_surfaces_it. Both fail against the first version of this patch and pass against the current one. The existing test_swallowed_pause_still_surfaces_unhandled_sibling_failure and test_nested_unhandled_sibling_failure_replaces_pause cases keep passing unchanged, so an unhandled sibling failure still replaces the pause exactly as before.

One limitation, stated plainly: the cancellation case is detected through Task.cancelling(), which exists from Python 3.11. On 3.10 that test is skipped and the cancellation window remains undetectable — the framework has the same 3.10 gap elsewhere, but I would rather name it than leave it implied.

The third finding (a deeper nesting that main also re-raises) is a pre-existing gap rather than something this pull request introduces, so I left it out of scope and noted it on the issue.

packages/core/tests/workflow is green locally (1278 passed, 6 skipped, 1 xfailed), ruff check and ruff format --check with the pinned 0.17.0 are clean, and pyright reports the same 11 pre-existing diagnostics as the unmodified file.

@eavanvalkenburg

Copy link
Copy Markdown
Member

Closing to finalize the design first.

This branch was successfully deployed

1 active deployment
github-app-auth — e5ed9b0b Deployed Oct 22, 2026 by Madin-H23 via add_label #25887
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Bug]: Functional workflow re-raises a step failure that the workflow already handled, when a sibling branch pauses for request_info

3 participants