Skip to content

resume a restart-killed agent in its own worktree instead of redoing the work - #3204

Merged
atomantic merged 6 commits into
mainfrom
cos/task-ms5n5lhb/agent-4845fa10
Jul 29, 2026
Merged

resume a restart-killed agent in its own worktree instead of redoing the work#3204
atomantic merged 6 commits into
mainfrom
cos/task-ms5n5lhb/agent-4845fa10

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

An agent killed by a server restart never reaches its completion hook, so nothing recorded what it left behind. Its task was requeued to a fresh agent that built a new worktree off the default branch and redid work that was still sitting on disk — including everything the dead run had not committed yet, which is most of it when the kill lands mid-edit.

The retry now picks up where the interrupted run stopped:

  • resolveTaskResumePatch is the one answer to "can this be resumed?", used by every path that retires a dead run — the boot/health-check orphan sweep, resetOrphanedTasks, the post-restart completion recovery, and the normal failure path. It returns a set patch (resume this), a clear patch (the pointer is spent — the branch merged or vanished), or an empty patch (nothing to say). Previously only the normal failure path recorded anything at all, which is why the restart case — the one that can't reach that path — was the one that redid its work.
  • handleOrphanedTask folds the patch into the same write that flips the task to pending. That flip emits tasks:changed, which can spawn the retry immediately, so a pointer written afterwards could land too late to be read.
  • adoptWorktree moves the dead run's worktree to the retry's own directory via git worktree move, carrying its commits, uncommitted edits, and untracked files intact. The move (rather than adopting in place) preserves <worktrees>/<agentId> == "the agent that owns this tree", which is the invariant cleanupOrphanedWorktrees reaps on — left at the dead agent's path, the live retry's worktree would be reaped out from under it. When the tree is gone but the branch survived, the retry attaches to the branch instead; when neither is usable it starts clean.
  • The prompt tells the retry what it's holding — that it is continuing someone else's run, and to read git status, the commit log, and whether a PR was already opened or merged before writing any code.

Work that already landed is still never resumed onto. A branch reads as merged for two opposite reasons, and the change separates them: commits that shipped (ahead > 0 — refuse, the agent-d2ae0352 shape) versus a branch that never committed at all (ahead == 0 — nothing to duplicate, so a dirty tree on top of it is safe to adopt). That second case is precisely the restart shape, and treating it as "merged" is what would have thrown the work away.

Test plan

  • cd server && npm test — 53 suites fail on this machine both before and after this branch, all requires PostgreSQL (no test DB provisioned here); no agent/worktree/task suite is among them.
  • New coverage: worktree adoption and its refusals (worktreeManager.test.js), the landed-work guard, lockfile-only dirt, repurposed directory, the early bail before the branch comparison, patch set/clear/leave-alone (cleanupAgentWorktree.test.js), the orphan sweep's ordering and its single combined write (agentManagement.test.js), prep's adopt → attach → clean fallback chain (agentWorkspacePrep.test.js), and pointer survival through an orphan cooldown (cosTaskStore.test.js).
  • git worktree move verified by hand against a scratch repo: modified tracked files, untracked files, and the branch all survive the move.
  • Reviewed by codex[gpt-5.6-terra] (3 rounds, every finding fixed) and ollama[Qwythos-9B] (clean). Codex found three real gaps this branch introduced or left open: a resume was unreachable for runs isolated by conflict auto-detection rather than an explicit flag; the completion path never wrote the clear patch, so a task kept pointing at a since-merged branch; and the terminal-status cleanup stripped the pointer on an orphan-cooldown block, which auto-revives to pending and so silently abandoned the work. Ollama's one finding (git.getBranch missing) was a false positive — but the suites mock git.js wholesale and structurally could not have told us, so this adds a guard asserting the real module against the real caller.

…the work

An agent killed by a server restart never reaches its completion hook, so
nothing recorded what it left behind. Its task was requeued to a fresh agent
that built a new worktree off the default branch and redid work still sitting
on disk — including everything the dead run had not committed yet, which is
most of it when the kill lands mid-edit.

resolveResumePointer now answers one question for every path that retires a
dead run, and handleOrphanedTask folds the answer into the same write that
flips the task to pending (that flip emits tasks:changed and can spawn the
retry immediately, so a pointer written afterwards could land too late). The
retry adopts the dead run's actual worktree via git worktree move — carrying
its commits, uncommitted edits, and untracked files — falling back to
attaching the preserved branch, then to a clean start.

The move (rather than adopting in place) keeps <worktrees>/<agentId> == the
agent that owns the tree, which is the invariant cleanupOrphanedWorktrees
reaps on; left at the dead agent's path the live retry's worktree would be
reaped out from under it.

Landed work is still never resumed onto: a branch reads as merged either
because its commits shipped (ahead > 0, refuse — the agent-d2ae0352 shape) or
because it never committed at all (ahead == 0, safe to adopt a dirty tree on
top of). A spent pointer is cleared so a later attempt can't attach to a
since-merged branch.
The resume path needs raw `git status --porcelain` to reuse
classifyWorktreeDirt, but getStatus is returned verbatim by
POST /api/git/status, where carrying the raw text alongside the parsed
files[] just doubles the response on every dirty tree. Split it into
getStatusPorcelain instead.
…etection

A run whose worktree came from conflict auto-detection rather than an
explicit useWorktree/openPR still records a resume pointer when it dies,
but the retry never reached the adoption path: it is gated on
explicitWorktree, and conflict detection returns 'proceed' once the dead
agent is gone. The retry ran in the shared workspace and abandoned the
work the pointer existed to save.

Take the worktree path whenever the task carries an existingBranch — the
branch has to be checked out somewhere to be reachable at all. A task
that never asked for isolation degrades to the shared workspace if the
worktree can't be created, rather than blocking, since that is what it
would have gotten before the pointer existed; an explicitly-isolated task
still fails closed.
…urn on adoption

Two gaps the second review pass surfaced:

The completion path never wrote the clear. resolveTaskResumePointer
returned null and recordTaskResumePointer bailed before the write, so a
task that WAS resuming kept pointing at a branch that had since merged or
vanished, and the next retry attached to it. Collapsed resolve+patch into
one resolveTaskResumePatch so both writers get the same three outcomes —
set, clear, or leave alone — and the gate can't diverge between them. The
distinction between "looked and found nothing" (clear) and "never looked"
(leave alone) is load-bearing: a resuming task whose retry couldn't get a
worktree still has its predecessor's tree on disk for the next attempt.

Adoption preserved lockfile-only dirt that removeWorktree discards. Since
the resume prompt tells the retry to finish and commit whatever it finds
uncommitted, a stale npm-install package-lock bump would ship in the PR.
adoptWorktree now applies the same classifyWorktreeDirt treatment, and
only when that churn is all there is.
…ldown

An orphan-cooldown block is a timed pause, not a terminal state —
unblockExpiredOrphanCooldowns flips it back to pending once cooldownUntil
passes. The terminal-status cleanup was stripping the resume pointer on
the way in, so the revived task started clean and abandoned the worktree
its dead agent left behind: the exact recovery this mechanism exists for,
lost on any orphan that dies twice inside 30 minutes.
Every suite exercising the resume path mocks ./git.js wholesale, so a
renamed or missing export is invisible there and surfaces only at runtime.
An ollama review pass flagged git.getBranch as missing — it isn't, but the
suites genuinely could not have told us either way, so assert the real
module against the real caller.
@atomantic
atomantic merged commit fefdb84 into main Jul 29, 2026
1 check passed
@atomantic
atomantic deleted the cos/task-ms5n5lhb/agent-4845fa10 branch July 29, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant