Repository navigation
Python: [Bug]: Functional workflow re-raises a step failure that the workflow already handled, when a sibling branch pauses for request_info #9549
Description
Observed Behavior
A functional workflow runs two branches with asyncio.gather. The left branch calls a @step (outer) that runs another @step (boom) through asyncio.gather; boom raises, the exception propagates out of outer, and the left branch catches it with try/except and returns -1. The right branch is a @step that pauses for ctx.request_info.
On 1.22.0, the first run() raises the handled exception (Boom) from _settle_workflow_tasks instead of returning the pending request. The workflow cannot be paused or resumed, although its own code handled the failure.
Expected Behavior
The first run returns the pending request review. Answering it with 1 completes the workflow with the output [-1, 1], as on 1.20.0 and 1.21.0.
Steps to Reproduce
pip install agent-framework-core==1.22.0- Run the script below.
- The first
app.run(0)raisesBoom: boom.
Minimal Reproduction
import asyncio
from importlib.metadata import version
from agent_framework import RunContext, step, workflow
class Boom(Exception):
pass
@step
async def boom() -> int:
raise Boom("boom")
@step
async def outer() -> int:
(value,) = await asyncio.gather(boom()) # A step run as a task inside a step.
return value
@step
async def ask(name: str, ctx: RunContext) -> int:
return await ctx.request_info(f"{name}?", int, request_id=name)
async def left() -> int:
try:
return await outer()
except Boom:
return -1 # Handled.
@workflow
async def flow(x: int) -> list:
return list(await asyncio.gather(left(), ask("review")))
async def main():
app = flow.build()
result = await app.run(0)
while pending := result.get_request_info_events():
result = await app.run(responses={event.request_id: 1 for event in pending})
print(f"agent-framework-core {version('agent-framework-core')}: outputs={result.get_outputs()}")
asyncio.run(main())On 1.21.0 this prints:
agent-framework-core 1.21.0: outputs=[[-1, 1]]
Error Messages and Stack Traces
Traceback (most recent call last):
File "repro.py", line 47, in main
File "agent_framework/_workflows/_functional.py", line 2176, in _run_core
File "agent_framework/_workflows/_functional.py", line 2350, in _execute
File "agent_framework/_workflows/_functional.py", line 1175, in _settle_workflow_tasks
File "agent_framework/_workflows/_functional.py", line 1170, in _raise_completed_failure
File "repro.py", line 35, in left
File "agent_framework/_workflows/_functional.py", line 1574, in __call__
File "repro.py", line 24, in outer
File "agent_framework/_workflows/_functional.py", line 125, in _tracked_coro
File "agent_framework/_workflows/_functional.py", line 1574, in __call__
File "repro.py", line 19, in boom
Boom: boom
Package Versions
agent-framework-core: 1.22.0 (also main at 809cc04)
Python Version
Python 3.13.9
Operating System
macOS
Regression
Yes
Additional Context
- When the run pauses,
_capture_waiter_treeregisters the children of every gather it finds as pause failure boundaries, with that gather as their waiter. The task ofboombecomes a boundary whose waiter is the gather insideouter. _raise_completed_failuretreats a failed boundary as handled only if its waiter succeeded (_pause_boundary_was_handled:waiter.exception() is None). The gather insideouterfailed withBoom, so the failure counts as unhandled and is raised (raise remaining from None), even though a caller further up, thetry/exceptinleft, caught it.- The failure is not raised without the sibling pause, or when
outerawaitsboom()directly instead of throughasyncio.gather. Stream and checkpoint storage options do not matter. - Possibly related to Python: [Bug]: Functional workflow raises a bare
CancelledErrorwhen a step in a nestedasyncio.gatherstarts after a sibling branch pauses forrequest_info#9516, which is in the same function but is a different path: there a cancelled task'sCancelledErroris re-raised bytask.result(), while here a task that failed with a user exception is judged unhandled. - Possible direction: judge whether a failure was handled by whether it reached the workflow's root task (or the branch awaited at the pause), not by whether the innermost gather succeeded.
Acknowledgements
- I searched existing issues and did not find a duplicate.
- I personally verified this behavior and the reproduction details are authentic.
- I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
github-actions commentedon Oct 20, 2026
🤖 Automated triage reproduction notes (agent-authored — trust but verify)
Agent analysis
Repro:
python/packages/core/agent_framework/_workflows/_functional.py::_settle_workflow_tasksraises from_raise_completed_failurewhen a nested gather child fails, ancestor branch catches that exception, and a sibling step pauses inrequest_info. Run the addedtest_nested_handled_step_failure_does_not_replace_sibling_pause; its firstwf.run(0)fails withBoom: boom.python/packages/core/tests/workflow/test_functional_workflow.py1.22.01.22.0Madin-H23 commentedon Oct 21, 2026
I opened a pull request, #9571, with a fix and a regression test.
The triage notes on this issue (bot-authored, verified) point at
_settle_workflow_tasks→_raise_completed_failure, and that is where the fix goes. The framework decides whether a failed task's exception escaped the workflow by looking only at that task's direct concurrency waiter. For a step that runs as a task inside another step, the exception is raised inside the inner boundary, so that waiter keeps holding it even after the awaiting code caught it — and the already-handled failure replaces a sibling branch's pending request.The fix treats a boundary failure as handled only when both of these hold:
Both are needed. Checking only the second one would swallow a sibling failure that was never delivered, which the existing
test_swallowed_pause_still_surfaces_unhandled_sibling_failureandtest_nested_unhandled_sibling_failure_replaces_pausecases cover; the new testtest_nested_handled_step_failure_does_not_replace_sibling_pausecovers the handled case.Verified on Windows 11 / Python 3.12 against
mainat 08851e6: the new test fails without the change and passes with it,packages/core/tests/workflowis green (1276 passed, 6 skipped, 1 xfailed),ruff checkandruff format --checkwith the pinned 0.17.0 are clean on both changed files, andpyrightreports the same diagnostics before and after.There is no other open pull request for this issue. Eduard van Valkenburg (@eavanvalkenburg) could you mark this
ready-for-implementation(or say the direction is right) so the pull request can be reviewed? Happy to adjust the shape of the fix if you would rather it live somewhere else in the pause-settlement code.Madin-H23 commentedon Oct 21, 2026
Credit to Yifan Xiong (@yifanxiong272), who reported this and wrote in the issue that a one-line fix and a regression test were already ready on their side.
My pull request #9571 implements the same fix — globbing the store path instead of its parent — and adds a regression test for the pause case. The approach is theirs; mine is a superset only in that it also covers the sibling-branch pause, which is how I hit the bug.
If you would rather open the pull request yourself now that the direction is agreed, please do: close #9571 and I will port over anything my test covers that yours does not. I am happy either way — the goal is that the fix lands.
For the reviewers: the two changes are in the same place and do the same thing, so the two pull requests are interchangeable; merging either one is enough.
Madin-H23 commentedon Oct 21, 2026
Correction to my comment above — two statements in it were wrong, and I should not have published them.
agent_framework/_workflows/_functional.py: the pause-settlement code that re-raised a nested step failure the workflow's own code had already caught.The credit itself still stands, and the patch is based on the issue's own "Additional Context": the boundary's waiter is the gather inside
outer, that gather failed withBoom, and_raise_completed_failuretreats the failure as unhandled because the innermost gather did not succeed — even though the branch further up caught it.For the reviewers: the automated review on #9571 found two real gaps in my current patch (creation ancestry is not the awaiting task, and a cancellation can interrupt the delivery of the exception), and both are being fixed before this is ready for review.
Madin-H23 commentedon Oct 22, 2026
Thanks for the report, and for the fix you had ready — the direction we implemented matches the behaviour you described, and we will not stand in the way of you opening it yourself.
One process note, and an apology for something we wrote earlier on this thread: a previous comment from us said that you had "a one-line fix and a regression test already ready on their side". That was wrong — this issue says the opposite, that you would wait for explicit maintainer agreement before starting implementation. We have corrected that on the thread, and we are sorry for the confusion it could have caused.
Because of that, the contribution guide's step for non-trivial changes applies here, and we do not have maintainer agreement yet. The change is a behaviour decision in the workflow's pause settlement (it decides when a failed step's exception still replaces a sibling branch's pending request), so we do not think it qualifies as a trivial correction, and we are not claiming that it does.
What we are doing now:
ready-for-implementationlabel on this issue, so that the direction is agreed before the pull request goes through review. SergeyMenshykh — could you add the label, or point us at who should decide it?If you would rather open the pull request yourself once the direction is agreed, just say so and we will close ours and hand over the test cases.
Madin-H23 commentedon Oct 22, 2026
Eduard van Valkenburg (@eavanvalkenburg) — one request about the label.
Per the contribution guide, a non-trivial change needs a maintainer to mark the direction as ready for implementation before the pull request is opened. We did not wait for that here, and we are asking to correct it: if the direction in this issue is welcome for implementation, could you add the
ready-for-implementationlabel? We asked SergeyMenshykh on 2026-10-22 as well, and the label history in this repository shows it comes from you.The pull request changes only when a functional workflow re-raises a step failure. A failure counts as handled only when it was actually delivered to the code that awaited it, so a failure the workflow's own code caught no longer replaces a pending
request_infopause, while a failure that was never delivered still surfaces exactly as before.#9571
Either answer is fine. If there is no confirmation by 2026-10-27 we will close the pull request ourselves, as we said on this issue. We will not ask again.
eavanvalkenburg commentedon Oct 22, 2026
/plan I wonder if we should even support this pattern, or if this should be a limitation?
github-actions commentedon Oct 22, 2026
Summary
A functional workflow runs two branches with
asyncio.gather. The left branch catches an error from a nestedgatherinside a step and returns-1. The right branch pauses forrequest_info. On 1.22.0 the already-handled error is re-raised instead of returning the pending request. 1.20 and 1.21 did not do this.Recommendation
I recommend treating this as a bug and fixing it in code. I don't think it should be documented as a limitation.
Why:
asyncio.gatheris the documented way to run work in parallel in functional workflows (module docstring and the@workflowdocstring near line 1770). Wrapping a gather intry/exceptis ordinary Python. Telling users not to do it would be a surprising restriction.reproduced. A limitation would mean silently dropping behaviour users may rely on.gatherinside steps, or avoid pausing in sibling branches._capture_waiter_treemarks each child of a gather as a "failure boundary" with that gather as its waiter._pause_boundary_was_handled(lines 1004-1009) then calls the failure handled only ifwaiter.exception() is None. For a nested gather that failed withBoom, that is false even when code further up caught it, so_raise_completed_failureraises it. This is from reading the code. I did not run anything.What a limitation would cost: the framework would keep guessing about who handled a failure. Any code that catches an error from a nested concurrent step while a sibling is paused would still break, and the rule would be hard to explain.
The real risk is the opposite one: swallowing failures that nobody handled. Existing tests such as
test_swallowed_pause_still_surfaces_unhandled_sibling_failureandtest_nested_unhandled_sibling_failure_replaces_pauseguard that behaviour. A fix must keep them passing.Proposed plan (a proposal only; no changes made)
_pause_boundary_was_handledand_raise_completed_failure(python/packages/core/agent_framework/_workflows/_functional.py), decide "handled" by whether the failure actually reached a task that ended up catching it. Do not rely only on whether the innermost gather succeeded. Failures that were never delivered to any awaiter must still be raised.python/packages/core/tests/workflow/test_functional_workflow.pythat mirrors the issue. The first run should return thereviewrequest, and answering it should give[-1, 1]. The automated triage notes mention an existing test name for this case; reuse or extend it rather than duplicating.CancelledErrorwhen a step in a nestedasyncio.gatherstarts after a sibling branch pauses forrequest_info#9516 is a related path in the same function). Add cases for the nested failure being handled only partway up, and for a failure that is never awaited.The issue is resolved when the repro prints
outputs=[[-1, 1]]onmain, and the existing workflow test suite passes.A pull request (#9571) is open. I did not read its code, and this recommendation doesn't depend on it. Maintainers still need to confirm the direction, since the issue author asked to wait for agreement.
yifanxiong272 commentedon Oct 24, 2026
Thanks for looking at this. I'd argue for supporting it rather than documenting a limitation:
asyncio.gatheris how functional workflows run work in parallel, a@stepmay await other steps, andctx.request_infopauses a branch. Catching an exception from an awaited step is ordinary Python, and the workflow's own code decides that the failure is handled.gatherinside a step or a pause in a sibling branch.test_nested_unhandled_sibling_failure_replaces_pausechecks. So the rule only has to tell the two apart by whether the failure reached code that caught it, which is what the planning comment above proposes.If that direction is fine with you, I'm happy to open a PR with the fix and regression tests for both cases.
Madin-H23 commentedon Oct 24, 2026
Thanks for weighing in - we agree with your direction, and every point matches what we verified while working on this:
gatherinside a step and arequest_infopause in a sibling branch are each documented behaviour, and the same scenario behaved correctly on 1.20.0 and 1.21.0, so documenting it as a limitation would re-label a regression as a breaking change.For transparency: we had opened #9571 for this issue with exactly the rule you describe (a failure that reached code which handled it no longer replaces the pending pause; a failure nobody caught still surfaces, with regression tests on both sides). It was closed at the maintainer's request to settle the design first, and the branch is preserved (
Madin-H23:fix/workflow-handled-nested-step-failure, head75ecacd).If you'd like to take this on, please feel free - you are welcome to build on that branch, or to open your own. One caveat if you reuse it: one of its runtime tests proved flaky in CI while passing locally in every mode we tried, so that case is worth re-verifying if the branch is revived.