Skip to content

.NET: Release the workflow when a run ends after an executor fails to initialize - #9585

Open
Ruiqi Wang (RachelWanggg) wants to merge 2 commits into
microsoft:mainfrom
RachelWanggg:fix/dotnet-release-workflow-after-executor-init-failure
Open

Ruiqi Wang (RachelWanggg) wants to merge 2 commits into
microsoft:mainfrom
RachelWanggg:fix/dotnet-release-workflow-after-executor-init-failure

Conversation

@RachelWanggg

Copy link
Copy Markdown
Contributor

Motivation & Context

A Workflow that hits one executor initialization failure can never run again. InProcessRunnerContext.EndRunAsync awaits every cached executor creation task before it releases ownership, so when an executor's InitializeAsync (or factory) threw, or was cancelled because the run was disposed while it was still initializing, ending the run rethrows that error and skips ReleaseOwnershipAsync. Run.DisposeAsync / StreamingRun.DisposeAsync throws the old error, and every later run on the same instance fails with "Cannot use a Workflow that is already owned by another runner or parent workflow." RunStreamingAsync and RunAsync have a second path to the same state: they take ownership, then enqueue the first input, which can create the start executor. If that throws, the run handle is dropped without being disposed, so the run never ends.

Workflows are usually built once and reused (AddWorkflow registers a singleton by default), so a transient failure while an executor connects to something in InitializeAsync, or a request abandoned while an executor is still initializing, leaves the workflow unusable until the process restarts.

Description & Review Guide

  • What are the major changes?
    • EndRunAsync skips executors whose creation task faulted or was cancelled, since there is no instance to dispose and the failure was already reported while the run executed, and releases ownership in a finally.
    • RunStreamingAsync and BeginRunHandlingChatProtocolAsync (used by RunAsync) dispose the run handle before rethrowing when enqueueing the first input fails, because the caller never receives that run and cannot end it.
  • What is the impact of these changes? After an executor fails to initialize, the next run on the same workflow starts normally. Disposing such a run no longer rethrows the initialization error, which was already surfaced as an ExecutorFailedEvent / WorkflowErrorEvent, or thrown by RunStreamingAsync / RunAsync. If an executor's own DisposeAsync throws, that exception still propagates from disposing the run, as before, but the workflow is now released. Runs without creation failures end exactly as before. No public API change.
  • What do you want reviewers to focus on? Whether EndRunAsync should skip a faulted creation task, as here, or still surface the error. The error has already reached the caller by the time the run is disposed, and rethrowing it from DisposeAsync is what kept the workflow owned.

Tests in InProcessExecutionTests.cs, all failing on main (the theories run on Lockstep and OffThread):

  • RunAsync_ReleasesWorkflowAfterExecutorInitializationFailsAsync: the second executor's InitializeAsync throws on the first run. The run reports an ExecutorFailedEvent, disposing it does not throw, and the next run yields its output. On main, disposing throws the TimeoutException.
  • StreamingRun_ReleasesWorkflowWhenDisposedDuringExecutorInitializationAsync: the run is disposed while the second executor's InitializeAsync waits on its cancellation token, then the workflow runs again. On main, disposing throws TaskCanceledException.
  • RunAsync_ReleasesWorkflowAfterExecutorDisposalFailsAsync: the executor's DisposeAsync throws once. Disposing the run throws that exception, and the next run still works. On main it fails with "already owned".
  • RunStreamingAsync_ReleasesWorkflowAfterStartExecutorInitializationFailsAsync: the start executor fails to initialize while RunStreamingAsync enqueues the first input. The call throws, and the next call works. On main it fails with "already owned".
  • RunAsync_ReleasesWorkflowAfterStartExecutorInitializationFailsAsync: the same through RunAsync, with an input type the start executor does not declare, so the start executor is created while enqueueing. On main it fails with "already owned".

Local verification on macOS arm64, .NET SDK 10.0.402, dotnet test --project tests/<project> -f net10.0, branch on top of 0200599. The main column is a clean checkout of 0200599 with only the new InProcessExecutionTests.cs.

Unit test project (Microsoft.Agents.AI.<name>.UnitTests) main This PR
Workflows 865 passed, 9 failed (the new tests) 874 passed
Workflows.Declarative 1159 passed 1159 passed
Workflows.Declarative.Mcp not run 102 passed
Workflows.Generators not run 36 passed
Hosting not run 181 passed
Hosting.A2A not run 167 passed
Hosting.AGUI.AspNetCore not run 29 passed
Hosting.OpenAI not run 585 passed
Hosting.AzureStorage not run 2 passed, 12 skipped (Azurite not available)
Foundry.Hosting not run 520 passed, 1 skipped (existing Skip)
DevUI not run 21 passed, 2 skipped (existing Skip)
CosmosNoSql not run 25 passed, 54 skipped (Cosmos DB Emulator not available)
  • Mutation checks, InProcessExecutionTests only: awaiting the creation task without the catch fails the initialization and cancellation tests. Releasing ownership only after a clean loop (no finally) fails RunAsync_ReleasesWorkflowAfterExecutorDisposalFailsAsync. Skipping only faulted, not cancelled, creation tasks fails StreamingRun_ReleasesWorkflowWhenDisposedDuringExecutorInitializationAsync. Removing the dispose from RunStreamingAsync or from BeginRunHandlingChatProtocolAsync fails the matching start executor test. Not caught: break instead of continue, which would stop disposing the executors after the failed one. Which executors come after it depends on ConcurrentDictionary order, so the tests do not pin it.
  • The repro from .NET: [Bug]: A Workflow stays owned after an executor fails to initialize, so every later run fails #9584 prints the expected output on this branch for both cases: requests 2 and 3 return results, and disposing request 1 does not throw.
  • dotnet format <csproj> --verify-no-changes passes for Microsoft.Agents.AI.Workflows and Microsoft.Agents.AI.Workflows.UnitTests.
  • dotnet build src/Microsoft.Agents.AI.Workflows --warnaserror passes for net10.0, net9.0, net8.0, netstandard2.0 and net472 with 0 warnings, and so does the unit test project for net10.0 and net472.
  • Not run: net472 tests (Windows only), integration tests, and unit test projects that do not depend on Microsoft.Agents.AI.Workflows.

Related Issue

Fixes #9584

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix). A workflow keeps the label and title prefix in sync automatically.

… initialize

EndRunAsync awaited every executor creation task before releasing
ownership, so a factory or InitializeAsync that threw or was cancelled
made disposing the run rethrow that error and left the Workflow owned:
every later run on it failed with "already owned by another runner".
Skip executors that were never created, and release ownership in a
finally so a failing executor dispose cannot keep it either.

RunStreamingAsync and RunAsync also dropped the run handle when
enqueueing the first input threw (the start executor is created there),
so the run was never ended. Dispose the handle before rethrowing.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@agent-framework-automation agent-framework-automation Bot added .NET Usage: [Issues, PRs], Target: .Net workflows Usage: [Issues, PRs], Target: Workflows labels Oct 21, 2026
@baywet
Vincent Biret (baywet) requested a balanced review from Copilot October 21, 2026 17:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused cleanup changes address the ownership leaks, with regression coverage for the affected failure paths and no unresolved findings.

No open issues 🎉

Review effort: Balanced

@baywet Vincent Biret (baywet) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution!

@baywet
Vincent Biret (baywet) added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 21, 2026
@baywet
Vincent Biret (baywet) added this pull request to the merge queue Oct 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 23, 2026

This branch was successfully deployed

1 active deployment
github-app-auth — 3e348794 Deployed Oct 23, 2026 by baywet via add_label #26038
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

.NET Usage: [Issues, PRs], Target: .Net workflows Usage: [Issues, PRs], Target: Workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.NET: [Bug]: A Workflow stays owned after an executor fails to initialize, so every later run fails

3 participants