resume a restart-killed agent in its own worktree instead of redoing the work - #3204
Merged
Conversation
…the work An agent killed by a server restart never reaches its completion hook, so nothing recorded what it left behind. Its task was requeued to a fresh agent that built a new worktree off the default branch and redid work still sitting on disk — including everything the dead run had not committed yet, which is most of it when the kill lands mid-edit. resolveResumePointer now answers one question for every path that retires a dead run, and handleOrphanedTask folds the answer into the same write that flips the task to pending (that flip emits tasks:changed and can spawn the retry immediately, so a pointer written afterwards could land too late). The retry adopts the dead run's actual worktree via git worktree move — carrying its commits, uncommitted edits, and untracked files — falling back to attaching the preserved branch, then to a clean start. The move (rather than adopting in place) keeps <worktrees>/<agentId> == the agent that owns the tree, which is the invariant cleanupOrphanedWorktrees reaps on; left at the dead agent's path the live retry's worktree would be reaped out from under it. Landed work is still never resumed onto: a branch reads as merged either because its commits shipped (ahead > 0, refuse — the agent-d2ae0352 shape) or because it never committed at all (ahead == 0, safe to adopt a dirty tree on top of). A spent pointer is cleared so a later attempt can't attach to a since-merged branch.
The resume path needs raw `git status --porcelain` to reuse classifyWorktreeDirt, but getStatus is returned verbatim by POST /api/git/status, where carrying the raw text alongside the parsed files[] just doubles the response on every dirty tree. Split it into getStatusPorcelain instead.
…etection A run whose worktree came from conflict auto-detection rather than an explicit useWorktree/openPR still records a resume pointer when it dies, but the retry never reached the adoption path: it is gated on explicitWorktree, and conflict detection returns 'proceed' once the dead agent is gone. The retry ran in the shared workspace and abandoned the work the pointer existed to save. Take the worktree path whenever the task carries an existingBranch — the branch has to be checked out somewhere to be reachable at all. A task that never asked for isolation degrades to the shared workspace if the worktree can't be created, rather than blocking, since that is what it would have gotten before the pointer existed; an explicitly-isolated task still fails closed.
…urn on adoption Two gaps the second review pass surfaced: The completion path never wrote the clear. resolveTaskResumePointer returned null and recordTaskResumePointer bailed before the write, so a task that WAS resuming kept pointing at a branch that had since merged or vanished, and the next retry attached to it. Collapsed resolve+patch into one resolveTaskResumePatch so both writers get the same three outcomes — set, clear, or leave alone — and the gate can't diverge between them. The distinction between "looked and found nothing" (clear) and "never looked" (leave alone) is load-bearing: a resuming task whose retry couldn't get a worktree still has its predecessor's tree on disk for the next attempt. Adoption preserved lockfile-only dirt that removeWorktree discards. Since the resume prompt tells the retry to finish and commit whatever it finds uncommitted, a stale npm-install package-lock bump would ship in the PR. adoptWorktree now applies the same classifyWorktreeDirt treatment, and only when that churn is all there is.
…ldown An orphan-cooldown block is a timed pause, not a terminal state — unblockExpiredOrphanCooldowns flips it back to pending once cooldownUntil passes. The terminal-status cleanup was stripping the resume pointer on the way in, so the revived task started clean and abandoned the worktree its dead agent left behind: the exact recovery this mechanism exists for, lost on any orphan that dies twice inside 30 minutes.
Every suite exercising the resume path mocks ./git.js wholesale, so a renamed or missing export is invisible there and surfaces only at runtime. An ollama review pass flagged git.getBranch as missing — it isn't, but the suites genuinely could not have told us either way, so assert the real module against the real caller.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An agent killed by a server restart never reaches its completion hook, so nothing recorded what it left behind. Its task was requeued to a fresh agent that built a new worktree off the default branch and redid work that was still sitting on disk — including everything the dead run had not committed yet, which is most of it when the kill lands mid-edit.
The retry now picks up where the interrupted run stopped:
resolveTaskResumePatchis the one answer to "can this be resumed?", used by every path that retires a dead run — the boot/health-check orphan sweep,resetOrphanedTasks, the post-restart completion recovery, and the normal failure path. It returns a set patch (resume this), a clear patch (the pointer is spent — the branch merged or vanished), or an empty patch (nothing to say). Previously only the normal failure path recorded anything at all, which is why the restart case — the one that can't reach that path — was the one that redid its work.handleOrphanedTaskfolds the patch into the same write that flips the task topending. That flip emitstasks:changed, which can spawn the retry immediately, so a pointer written afterwards could land too late to be read.adoptWorktreemoves the dead run's worktree to the retry's own directory viagit worktree move, carrying its commits, uncommitted edits, and untracked files intact. The move (rather than adopting in place) preserves<worktrees>/<agentId>== "the agent that owns this tree", which is the invariantcleanupOrphanedWorktreesreaps on — left at the dead agent's path, the live retry's worktree would be reaped out from under it. When the tree is gone but the branch survived, the retry attaches to the branch instead; when neither is usable it starts clean.git status, the commit log, and whether a PR was already opened or merged before writing any code.Work that already landed is still never resumed onto. A branch reads as merged for two opposite reasons, and the change separates them: commits that shipped (
ahead > 0— refuse, the agent-d2ae0352 shape) versus a branch that never committed at all (ahead == 0— nothing to duplicate, so a dirty tree on top of it is safe to adopt). That second case is precisely the restart shape, and treating it as "merged" is what would have thrown the work away.Test plan
cd server && npm test— 53 suites fail on this machine both before and after this branch, allrequires PostgreSQL(no test DB provisioned here); no agent/worktree/task suite is among them.worktreeManager.test.js), the landed-work guard, lockfile-only dirt, repurposed directory, the early bail before the branch comparison, patch set/clear/leave-alone (cleanupAgentWorktree.test.js), the orphan sweep's ordering and its single combined write (agentManagement.test.js), prep's adopt → attach → clean fallback chain (agentWorkspacePrep.test.js), and pointer survival through an orphan cooldown (cosTaskStore.test.js).git worktree moveverified by hand against a scratch repo: modified tracked files, untracked files, and the branch all survive the move.codex[gpt-5.6-terra](3 rounds, every finding fixed) andollama[Qwythos-9B](clean). Codex found three real gaps this branch introduced or left open: a resume was unreachable for runs isolated by conflict auto-detection rather than an explicit flag; the completion path never wrote the clear patch, so a task kept pointing at a since-merged branch; and the terminal-status cleanup stripped the pointer on anorphan-cooldownblock, which auto-revives topendingand so silently abandoned the work. Ollama's one finding (git.getBranchmissing) was a false positive — but the suites mockgit.jswholesale and structurally could not have told us, so this adds a guard asserting the real module against the real caller.