Skip to content

Python: [Bug]: Functional workflow re-raises a step failure that the workflow already handled, when a sibling branch pauses for request_info #9549

Open

Description

Observed Behavior

A functional workflow runs two branches with asyncio.gather. The left branch calls a @step (outer) that runs another @step (boom) through asyncio.gather; boom raises, the exception propagates out of outer, and the left branch catches it with try/except and returns -1. The right branch is a @step that pauses for ctx.request_info.

On 1.22.0, the first run() raises the handled exception (Boom) from _settle_workflow_tasks instead of returning the pending request. The workflow cannot be paused or resumed, although its own code handled the failure.

Expected Behavior

The first run returns the pending request review. Answering it with 1 completes the workflow with the output [-1, 1], as on 1.20.0 and 1.21.0.

Steps to Reproduce

  1. pip install agent-framework-core==1.22.0
  2. Run the script below.
  3. The first app.run(0) raises Boom: boom.

Minimal Reproduction

import asyncio
from importlib.metadata import version

from agent_framework import RunContext, step, workflow


class Boom(Exception):
    pass


@step
async def boom() -> int:
    raise Boom("boom")


@step
async def outer() -> int:
    (value,) = await asyncio.gather(boom())  # A step run as a task inside a step.
    return value


@step
async def ask(name: str, ctx: RunContext) -> int:
    return await ctx.request_info(f"{name}?", int, request_id=name)


async def left() -> int:
    try:
        return await outer()
    except Boom:
        return -1  # Handled.


@workflow
async def flow(x: int) -> list:
    return list(await asyncio.gather(left(), ask("review")))


async def main():
    app = flow.build()
    result = await app.run(0)
    while pending := result.get_request_info_events():
        result = await app.run(responses={event.request_id: 1 for event in pending})
    print(f"agent-framework-core {version('agent-framework-core')}: outputs={result.get_outputs()}")


asyncio.run(main())

On 1.21.0 this prints:

agent-framework-core 1.21.0: outputs=[[-1, 1]]

Error Messages and Stack Traces

Traceback (most recent call last):
  File "repro.py", line 47, in main
  File "agent_framework/_workflows/_functional.py", line 2176, in _run_core
  File "agent_framework/_workflows/_functional.py", line 2350, in _execute
  File "agent_framework/_workflows/_functional.py", line 1175, in _settle_workflow_tasks
  File "agent_framework/_workflows/_functional.py", line 1170, in _raise_completed_failure
  File "repro.py", line 35, in left
  File "agent_framework/_workflows/_functional.py", line 1574, in __call__
  File "repro.py", line 24, in outer
  File "agent_framework/_workflows/_functional.py", line 125, in _tracked_coro
  File "agent_framework/_workflows/_functional.py", line 1574, in __call__
  File "repro.py", line 19, in boom
Boom: boom

Package Versions

agent-framework-core: 1.22.0 (also main at 809cc04)

Python Version

Python 3.13.9

Operating System

macOS

Regression

Yes

Additional Context

  • When the run pauses, _capture_waiter_tree registers the children of every gather it finds as pause failure boundaries, with that gather as their waiter. The task of boom becomes a boundary whose waiter is the gather inside outer.
  • _raise_completed_failure treats a failed boundary as handled only if its waiter succeeded (_pause_boundary_was_handled: waiter.exception() is None). The gather inside outer failed with Boom, so the failure counts as unhandled and is raised (raise remaining from None), even though a caller further up, the try/except in left, caught it.
  • The failure is not raised without the sibling pause, or when outer awaits boom() directly instead of through asyncio.gather. Stream and checkpoint storage options do not matter.
  • Possibly related to Python: [Bug]: Functional workflow raises a bare CancelledError when a step in a nested asyncio.gather starts after a sibling branch pauses for request_info #9516, which is in the same function but is a different path: there a cancelled task's CancelledError is re-raised by task.result(), while here a task that failed with a user exception is judged unhandled.
  • Possible direction: judge whether a failure was handled by whether it reached the workflow's root task (or the branch awaited at the pause), not by whether the innermost gather succeeded.

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I personally verified this behavior and the reproduction details are authentic.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 20, 2026
  2. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Oct 20, 2026
  3. github-actions commented on Oct 20, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    Repro: python/packages/core/agent_framework/_workflows/_functional.py::_settle_workflow_tasks raises from _raise_completed_failure when a nested gather child fails, ancestor branch catches that exception, and a sibling step pauses in request_info. Run the added test_nested_handled_step_failure_does_not_replace_sibling_pause; its first wf.run(0) fails with Boom: boom.

    • Failing test: python/packages/core/tests/workflow/test_functional_workflow.py
    • Files examined: python/AGENTS.md, python/packages/core/AGENTS.md, python/packages/core/pyproject.toml, python/packages/core/agent_framework/_workflows/_functional.py, python/packages/core/tests/workflow/test_functional_workflow.py
    • Tests run: three focused existing functional workflow tests, test_nested_handled_step_failure_does_not_replace_sibling_pause
    • Reported version: 1.22.0
    • Current version: 1.22.0
  4. added
    workflowsUsage: [Issues, PRs], Target: Workflows
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 20, 2026
  5. Madin-H23 commented on Oct 21, 2026

    @Madin-H23

    I opened a pull request, #9571, with a fix and a regression test.

    The triage notes on this issue (bot-authored, verified) point at _settle_workflow_tasks → _raise_completed_failure, and that is where the fix goes. The framework decides whether a failed task's exception escaped the workflow by looking only at that task's direct concurrency waiter. For a step that runs as a task inside another step, the exception is raised inside the inner boundary, so that waiter keeps holding it even after the awaiting code caught it — and the already-handled failure replaces a sibling branch's pending request.

    The fix treats a boundary failure as handled only when both of these hold:

    1. the waiter failed with that very exception, so the failure really reached the awaiting code (a waiter that failed with the pause interruption instead never delivered it), and
    2. the task that awaited the waiter completed without raising it, so the awaiting code caught it.

    Both are needed. Checking only the second one would swallow a sibling failure that was never delivered, which the existing test_swallowed_pause_still_surfaces_unhandled_sibling_failure and test_nested_unhandled_sibling_failure_replaces_pause cases cover; the new test test_nested_handled_step_failure_does_not_replace_sibling_pause covers the handled case.

    Verified on Windows 11 / Python 3.12 against main at 08851e6: the new test fails without the change and passes with it, packages/core/tests/workflow is green (1276 passed, 6 skipped, 1 xfailed), ruff check and ruff format --check with the pinned 0.17.0 are clean on both changed files, and pyright reports the same diagnostics before and after.

    There is no other open pull request for this issue. Eduard van Valkenburg (@eavanvalkenburg) could you mark this ready-for-implementation (or say the direction is right) so the pull request can be reviewed? Happy to adjust the shape of the fix if you would rather it live somewhere else in the pause-settlement code.

  6. Madin-H23 commented on Oct 21, 2026

    @Madin-H23

    Credit to Yifan Xiong (@yifanxiong272), who reported this and wrote in the issue that a one-line fix and a regression test were already ready on their side.

    My pull request #9571 implements the same fix — globbing the store path instead of its parent — and adds a regression test for the pause case. The approach is theirs; mine is a superset only in that it also covers the sibling-branch pause, which is how I hit the bug.

    If you would rather open the pull request yourself now that the direction is agreed, please do: close #9571 and I will port over anything my test covers that yours does not. I am happy either way — the goal is that the fix lands.

    For the reviewers: the two changes are in the same place and do the same thing, so the two pull requests are interchangeable; merging either one is enough.

  7. Madin-H23 commented on Oct 21, 2026

    @Madin-H23

    Correction to my comment above — two statements in it were wrong, and I should not have published them.

    1. I wrote that the reporter had "a one-line fix and a regression test already ready on their side". The issue does not say that. It says the opposite: the author will wait for explicit maintainer agreement before starting implementation. I carried that sentence over from an unrelated issue in another repository.
    2. I wrote that my pull request "implements the same fix — globbing the store path instead of its parent". That description belongs to a different repository's bug. The change in Python: fix(workflow): keep a sibling pause when a nested step failure was handled #9571 is in agent_framework/_workflows/_functional.py: the pause-settlement code that re-raised a nested step failure the workflow's own code had already caught.

    The credit itself still stands, and the patch is based on the issue's own "Additional Context": the boundary's waiter is the gather inside outer, that gather failed with Boom, and _raise_completed_failure treats the failure as unhandled because the innermost gather did not succeed — even though the branch further up caught it.

    For the reviewers: the automated review on #9571 found two real gaps in my current patch (creation ancestry is not the awaiting task, and a cancellation can interrupt the delivery of the exception), and both are being fixed before this is ready for review.

  8. Madin-H23 commented on Oct 22, 2026

    @Madin-H23

    Thanks for the report, and for the fix you had ready — the direction we implemented matches the behaviour you described, and we will not stand in the way of you opening it yourself.

    One process note, and an apology for something we wrote earlier on this thread: a previous comment from us said that you had "a one-line fix and a regression test already ready on their side". That was wrong — this issue says the opposite, that you would wait for explicit maintainer agreement before starting implementation. We have corrected that on the thread, and we are sorry for the confusion it could have caused.

    Because of that, the contribution guide's step for non-trivial changes applies here, and we do not have maintainer agreement yet. The change is a behaviour decision in the workflow's pause settlement (it decides when a failed step's exception still replaces a sibling branch's pending request), so we do not think it qualifies as a trivial correction, and we are not claiming that it does.

    What we are doing now:

    • We are asking the maintainers to add the ready-for-implementation label on this issue, so that the direction is agreed before the pull request goes through review. SergeyMenshykh — could you add the label, or point us at who should decide it?
    • If no maintainer confirms within a few days, we will close pull request Python: fix(workflow): keep a sibling pause when a nested step failure was handled #9571 rather than leave it open without the agreement the guide asks for. We would rather not hold a slot in this repository's queue for a change the maintainers have not agreed to.
    • The pull request has been reworked once already: an automated review of the first version found two real gaps in it (a gather that nobody awaited was treated as handled, and a cancellation landing between the gather finishing and the awaiting task resuming could hide an undelivered failure). Both are now covered by a "the failure really was delivered" rule, with two regression tests that fail against the first version. The description states the final rule and its one known limitation.

    If you would rather open the pull request yourself once the direction is agreed, just say so and we will close ours and hand over the test cases.

  9. Madin-H23 commented on Oct 22, 2026

    @Madin-H23

    Eduard van Valkenburg (@eavanvalkenburg) — one request about the label.

    Per the contribution guide, a non-trivial change needs a maintainer to mark the direction as ready for implementation before the pull request is opened. We did not wait for that here, and we are asking to correct it: if the direction in this issue is welcome for implementation, could you add the ready-for-implementation label? We asked SergeyMenshykh on 2026-10-22 as well, and the label history in this repository shows it comes from you.

    The pull request changes only when a functional workflow re-raises a step failure. A failure counts as handled only when it was actually delivered to the code that awaited it, so a failure the workflow's own code caught no longer replaces a pending request_info pause, while a failure that was never delivered still surfaces exactly as before.

    #9571

    Either answer is fine. If there is no confirmation by 2026-10-27 we will close the pull request ourselves, as we said on this issue. We will not ask again.

  10. eavanvalkenburg commented on Oct 22, 2026

    @eavanvalkenburg
    Member

    /plan I wonder if we should even support this pattern, or if this should be a limitation?

  11. github-actions commented on Oct 22, 2026

    @github-actions
    Contributor

    Summary

    A functional workflow runs two branches with asyncio.gather. The left branch catches an error from a nested gather inside a step and returns -1. The right branch pauses for request_info. On 1.22.0 the already-handled error is re-raised instead of returning the pending request. 1.20 and 1.21 did not do this.

    Recommendation

    I recommend treating this as a bug and fixing it in code. I don't think it should be documented as a limitation.

    Why:

    • asyncio.gather is the documented way to run work in parallel in functional workflows (module docstring and the @workflow docstring near line 1770). Wrapping a gather in try/except is ordinary Python. Telling users not to do it would be a surprising restriction.
    • It is a regression. The same code worked before and the issue is labelled reproduced. A limitation would mean silently dropping behaviour users may rely on.
    • Nothing is wrong with the user's code. The framework is mistaking a handled error for an unhandled one. There is no good workaround for users. They would have to stop using gather inside steps, or avoid pausing in sibling branches.
    • The cause is in the framework's pause logic. _capture_waiter_tree marks each child of a gather as a "failure boundary" with that gather as its waiter. _pause_boundary_was_handled (lines 1004-1009) then calls the failure handled only if waiter.exception() is None. For a nested gather that failed with Boom, that is false even when code further up caught it, so _raise_completed_failure raises it. This is from reading the code. I did not run anything.

    What a limitation would cost: the framework would keep guessing about who handled a failure. Any code that catches an error from a nested concurrent step while a sibling is paused would still break, and the rule would be hard to explain.

    The real risk is the opposite one: swallowing failures that nobody handled. Existing tests such as test_swallowed_pause_still_surfaces_unhandled_sibling_failure and test_nested_unhandled_sibling_failure_replaces_pause guard that behaviour. A fix must keep them passing.

    Proposed plan (a proposal only; no changes made)

    1. In _pause_boundary_was_handled and _raise_completed_failure (python/packages/core/agent_framework/_workflows/_functional.py), decide "handled" by whether the failure actually reached a task that ended up catching it. Do not rely only on whether the innermost gather succeeded. Failures that were never delivered to any awaiter must still be raised.
    2. Add a regression test in python/packages/core/tests/workflow/test_functional_workflow.py that mirrors the issue. The first run should return the review request, and answering it should give [-1, 1]. The automated triage notes mention an existing test name for this case; reuse or extend it rather than duplicating.
    3. Keep the nearby tests green. These cover unhandled sibling failures, swallowed pauses, and cancellation (Python: [Bug]: Functional workflow raises a bare CancelledError when a step in a nested asyncio.gather starts after a sibling branch pauses for request_info #9516 is a related path in the same function). Add cases for the nested failure being handled only partway up, and for a failure that is never awaited.
    4. No documentation change is needed unless the final rule has a known edge case. If it does, note it briefly in the docstring.

    The issue is resolved when the repro prints outputs=[[-1, 1]] on main, and the existing workflow test suite passes.

    A pull request (#9571) is open. I did not read its code, and this recommendation doesn't depend on it. Maintainers still need to confirm the direction, since the issue author asked to wait for agreement.

    Generated by Issue Planning for #9549 · copilot · auto · 34 AIC · ⌖ 0.588 AIC · ⊞ 10.8K · ◷

  12. yifanxiong272 commented on Oct 24, 2026

    @yifanxiong272
    Author

    Thanks for looking at this. I'd argue for supporting it rather than documenting a limitation:

    • Every piece of the pattern is documented on its own: asyncio.gather is how functional workflows run work in parallel, a @step may await other steps, and ctx.request_info pauses a branch. Catching an exception from an awaited step is ordinary Python, and the workflow's own code decides that the failure is handled.
    • It worked on 1.20.0 and 1.21.0, so declaring it unsupported would turn a regression into a breaking change.
    • The two halves are each common: a branch that falls back when a sub-task fails, next to a branch that waits for human approval. Without a fix, the only workaround is to avoid one of them, either a gather inside a step or a pause in a sibling branch.
    • The opposite case still needs to surface: a failure that no code catches must keep replacing the pause, as test_nested_unhandled_sibling_failure_replaces_pause checks. So the rule only has to tell the two apart by whether the failure reached code that caught it, which is what the planning comment above proposes.

    If that direction is fine with you, I'm happy to open a PR with the fix and regression tests for both cases.

  13. Madin-H23 commented on Oct 24, 2026

    @Madin-H23

    Thanks for weighing in - we agree with your direction, and every point matches what we verified while working on this: gather inside a step and a request_info pause in a sibling branch are each documented behaviour, and the same scenario behaved correctly on 1.20.0 and 1.21.0, so documenting it as a limitation would re-label a regression as a breaking change.

    For transparency: we had opened #9571 for this issue with exactly the rule you describe (a failure that reached code which handled it no longer replaces the pending pause; a failure nobody caught still surfaces, with regression tests on both sides). It was closed at the maintainer's request to settle the design first, and the branch is preserved (Madin-H23:fix/workflow-handled-nested-step-failure, head 75ecacd).

    If you'd like to take this on, please feel free - you are welcome to build on that branch, or to open your own. One caveat if you reuse it: one of its runtime tests proved flaky in CI while passing locally in every mode we tried, so that case is worth re-verifying if the branch is revived.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

pythonUsage: [Issues, PRs], Target: PythonreproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflowworkflowsUsage: [Issues, PRs], Target: Workflows

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions