Repository navigation
Cancel sub-workflow child runs that no parent waits for - #2141
Merged
Merged
Conversation
ppXD
force-pushed
the
fix/cancel-unwaited-child-runs
branch
2 times, most recently
from
November 7, 2026 19:42
e8e8576 to
02a020a
Compare
A parent that ends while a child is still live (a sibling fails after a barrier-free wake, a stop's child cancel throws, a Continue stages a fresh child) left that child running: the engine's failure cleanup cancels only Pending/Enqueued children, and the child's own timer or self-advance wake kept re-walking it, spending LLM turns for work nothing reads. - The stuck-run reconciler now runs CancelUnwaitedChildRunsAsync first: non-terminal ChildWorkflow runs past a committed staging grace period with no live parent holding a Pending Subworkflow wait naming them are cancelled through the full CancelRunAsync, oldest first, in batches, each in its own try/catch, with a summary counter. Deeper levels clear on later ticks. - WorkflowRunDispatcher refuses a run with a finished ancestor, walking only through sub-workflow children so rerun/replay lineage never counts. - One shared direct-parent exclusion now guards every reconciler wake query, replacing the per-query copies and the "left alone" count. - Continue's CloseEndedAttemptAsync cancels the ended attempt's live children through the full cancel after commit, not only Pending ones. - The sweep runs after the cheap sweeps and in its own catch, so a failed find cannot stop them. A failed or no-op attempt stamps the child's LastModifiedDate and the batch is least-recently-attempted first, so stuck children cannot hold every slot. - Continue refuses a sub-workflow child with a finished ancestor, which the dispatcher would refuse and the sweep cancel anyway. - A Running orphan is left to the sweep's full teardown instead of being marked abandoned bare.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
StuckRunReconcilerService.CancelUnwaitedChildRunsAsyncruns first inReconcileAsync: non-terminalChildWorkflowruns older thanUnwaitedChildStagingGrace(2 min, committed) with no live parent holding a PendingSubworkflowwait whoseTokenis the child id are cancelled viaIWorkflowService.CancelRunAsync,BatchSizeper tick, each in its own try/catch, counted inCancelledUnwaitedChildRun. It runs after the cheap sweeps and in its own catch (a failed find is logged and cannot stop the others). The batch is ordered byLastModifiedDatethenCreatedDate; a failed or no-op cancel stampsLastModifiedDate, so stuck children rotate to the back instead of holding every slot.WorkflowRunDispatcher.DispatchAsyncrefuses a run with a finished ancestor (walks only through sub-workflow children, cycle-safe; rerun/replay lineage never counts), so a still-armed timer cannot re-walk an orphan.RunsWithNoFinishedParentexclusion now backs every reconciler wake query; the "left alone" counting is dropped.ContinueRunAsyncrefuses (returnsfalse, logs the reason) a run that is a sub-workflow child with a finished ancestor, reusingRunAncestry.CloseEndedAttemptAsync(Continue) now returns the ended attempt's live children, cancelled through the full cancel after commit.Changed existing methods:
StuckRunReconcilerService: constructor (+IWorkflowService),ReconcileAsync,RedispatchStuckPendingAsync,FindStrandedSuspendedIdsAsync,RecoverSupervisorAdvancesAsync,RecoverStrandedTimerWaitsAsync,RecoverStrandedSupervisorInfraParkWaitsAsync,FindRecoverableSupervisorRunsAsync,FindPreClaimSupervisorCrashCandidatesAsync,MarkAbandonedRunningAsync(a Running orphan is cancelled with full teardown by the sweep instead of being marked Failure bare); class is nowpartial.WorkflowRunDispatcher.DispatchAsync.WorkflowService:ContinueRunAsync,LiveChildRunIdsAsyncnow uses the sharedWorkflowRunState.NonTerminalset;CloseEndedAttemptAsync(return type),ReviveTerminalRunAsync,CancelStagedChildRunsAsyncreplaced byCancelEndedAttemptChildrenAsync,StagedChildRunIdsAsyncreplaced byLiveChildRunIdsAsync; doc comments that admitted the gap.ReconcileStuckRunsCommandHandler.Handle,StuckRunReconcileSummary,ReconcileStuckRunsResponse(+counter);IWorkflowRunDispatcherandIWorkflowServicedocs.#1990 edits several of the same files; this PR keeps the new logic in new files (
StuckRunReconcilerService.UnwaitedChildren.cs,RunAncestry.cs) to ease that rebase.Test plan
StuckRunReconcilerFlowTests37,UnwaitedChildRunFlowTests38,DispatcherFinishedAncestorFlowTests9,SubworkflowCancelFlowTests8, allContinueintegration tests 106 (2 skipped), all greenUnwaitedChildRunE2ETests3 green