Skip to content

Keep a run live, and its questions asked, when its machine resumes after a server restart - #887

Merged
lavaman131 merged 1 commit into
mainfrom
run-state-after-restart
Nov 13, 2026
Merged

lavaman131 merged 1 commit into
mainfrom
run-state-after-restart

Conversation

@lavaman131

Copy link
Copy Markdown
Collaborator

What the maintainer saw

A hosted deployment was redeployed while a machine's Planner had a workflow run under way. The machine's
connection dropped with close code 1012 and resumed one second later, as designed. Then:

  1. Chat's run strip flipped to "stopped" and the run's stages disappeared. On the machine, the run carried
    on: it finished its verify stages, started a second round and published a new plan revision about six
    minutes later.
  2. The run then asked a question of its own. The question never appeared: not in Decisions, not inline in
    the document, and not in Chat. The run waited for an answer nobody could give.

Cause

There are two faults. Both are in how the new process treats the sessions a restart rebuilt after their
client has taken them back.

The cards were restored as stopped. When a document opens, Chat restores its stored run cards. They
stay live only if PlannerAttachments.restoredWork says a rebuilt hold still keeps runs. That check looked
only at the parked hold, from before its client came back. A client that resumes moves the sessions onto
its new link (AgentLink.#hand), and the parked hold is cleared. So once the client was back, which took
one second, the check found nothing. Chat.restore then stopped every live card (restoreRuns →
mergeRuns(…, undefined)). The browser reopened the document right after the client was back, at
16:39:43, which is exactly when the strip flipped.

The resumed sessions could not reach the document. AgentLink.rebuilt made each session with a bare
planner, { channelId, isolated: false }, with no onRuns and no humanInput. As a result:

  • The client's runs reports for a resumed session updated nothing in Chat. The protocol doc even said
    so: "update nothing Chat shows until a Planner turn starts a session of its own."
  • An input-request on a resumed session was refused with failed: "Decisions are not available to this
    session." I reproduced this on 74772ee with the new test:
    {"type":"input-result","request":"ask-2","status":"failed","message":"Decisions are not available to this session."}.
    No card was created, so there was nothing to show anywhere. The client was told its question had
    failed.

Fix

A session a restart rebuilt now reaches its document through the room's host. The document already gave
this host to the link when it opened, so the link could wait again on questions left open at the cut
(rejoinInputs). The changes:

  • RejoinHost (harness/remote/link.ts) can now also:

    • say whether its document is still open (live)
    • ask the members as Decisions (humanInput)
    • show run reports (runs)

    main.ts builds all three from the open plan.

  • Each rebuilt session's planner gets the following:

    • onRuns and linkChanged hand its reports to the host. When its link drops again, the cards are
      marked disconnected, never stopped.
    • humanInput opens the document for the question if nobody has it open, through a new
      PlannerAttachments option, openDocument. It keeps the document open until the question ends,
      and asks through the room's host.
    • rootOf is set, so a paused root still holds a workflow question's answer.
  • rejoinInputs gives the host to the rebuilt sessions and publishes what each one last reported.
    So a document that opens after the client is back shows current stages right away.

  • restoredWork also answers live when the attached link holds a rebuilt session with work. That
    means a live or paused run, an open question, or no report yet. It answers down when that link has
    dropped again. Cards are only ever stopped when the client says so (a runs report, or the link
    ending for good).

  • Chat.rebuiltRuns shows a rebuilt session's reports through the same publishRuns path a
    retained session uses: stage storage, finish notices and card windows. It uses a stand-in owner per
    session id. publishRuns now takes only what it needs of the room. Following up on a run that ended
    is passed in as a callback, so a rebuilt session never starts a Planner turn about its runs. That
    matches remote sessions today (followsUpRuns: false).

  • createHumanInput takes only the parts of the room it reads (a type-only change).

  • Docs. docs/agent-connection.md ("After a server restart") and docs/hosted-agent.md now say that
    a resumed session's runs and questions reach the document.

No UI changed, so there are no screenshots.

Tests

Three new tests in apps/server/src/harness/remote/handover-window.test.ts, under "a workflow run its
client carries on through a restart". Each one runs a real cut and takeover between two processes over one
database, with a raw agent-link client standing in for the machine. Before the cut, the first process's
Chat shows the run. The client resumes after the restart and reports its run still running, with a new
stage.

  1. When the document opens after the client is back, the run shows as running (not stopped, not
    disconnected), with the stage it reported.
  2. The run's question after the restart (input-request on the resumed session, naming its workflow run)
    becomes a Decision. A member's answer reaches the client as answered.
  3. With nobody having the document open, the question opens it and keeps it open until the answer. The
    answer reaches the client, and then the document is let go.

All three fail on 74772ee. Test 1 gets "stopped". Tests 2 and 3 get no card, because the request is
refused with the message above. All three pass on this branch. The file's 34 other tests also pass.

The reopen helper in that file now wires the document the way main.ts does (plannerKeepsRuns, and the
host's live, humanInput and runs). Its plannerKeepsRuns names the link's document id, because the
fixture's document and the link's have different ids.

Checks

  • bun test apps/server/src/harness/remote/handover-window.test.ts: 37 pass.
  • bun test apps/server/src/harness/remote apps/server/src/harness/atomic/human-input.test.ts apps/server/src/testing/resume-after-restart.test.ts:
    342 pass.
  • bun test of harness/session, harness/atomic/full, chat/invoke, chat/service, chat/steering,
    chat/job-heading-memory, chat/remote-planner and plan/persistence: 297 pass.
  • bun run types: clean.
  • bun run ci: clean. Lint warnings are back to the baseline count.

Judgement calls

  • The document is opened only while a rebuilt session's question waits. It is not held for the whole
    life of the run. Before the restart, a retained session held the document while its runs were live.
    Nothing in the new process retains a rebuilt session, and holding the room until the run ends would
    need a new end-of-work signal.

    Run reports that arrive while the document is closed are not stored. The cards stay live, and the
    latest report is shown the moment the document opens. A run that ends while nobody has the document
    open gets its finish notice when someone next opens it.

  • A rebuilt session's cards and a retained session's cards can both exist in one Chat. They are
    owned per session, as before. A later turn's session that reports the same run takes the card over.

  • Out of scope. Stop and pause on a run carried across a restart are still not wired to Chat's
    buttons. Those buttons act on the retained session, and a rebuilt session has none. A follow-up can give
    rebuilt sessions control the way start() does.

…esumes after a restart

After a deploy restarted the server, a run a machine was still working on showed as stopped in
Chat, its stages gone, and the question it asked a few minutes later never became a Decision, so
the run waited for an answer nobody could give. The machine had resumed its sessions a second
after the restart, as designed. Two things then broke.

The cards stopped because a document that opens after its client is back restored them as
stopped: whether a restart's rebuilt hold still keeps runs was asked only of the hold before its
client took it back. A client that resumes moves the sessions to its new link, where nothing in
the new process retains them, so the answer was "nothing", and Chat stopped every live card. The
sessions a restart rebuilt also had no way to reach the document: their run reports went nowhere,
and a question on one was refused with "Decisions are not available to this session."

A session a restart rebuilt now reaches its document through the room's host, which the document
gives it when it opens, as it already did for the questions left open at the cut. Its run reports
are Chat's run cards, its link dropping again marks them disconnected, and a document that opens
while a link holds such a session with work shows its cards live, with what the client last
reported. A question asked on it becomes Decisions as one asked before the restart did; the
document is opened for it if nobody has it open and kept open until the question ends, and the
answer goes to the client's request.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline (subagent)
Assistant-verification: bun test apps/server/src/harness/remote/handover-window.test.ts (37 pass; the 3 new restart-mid-run tests fail on 74772ee: card "stopped", question refused "Decisions are not available to this session.")
Assistant-verification: bun test apps/server/src/harness/remote apps/server/src/harness/atomic/human-input.test.ts apps/server/src/testing/resume-after-restart.test.ts (342 pass)
Assistant-verification: bun test apps/server/src/harness/session.test.ts apps/server/src/harness/atomic/full.test.ts apps/server/src/chat/invoke.test.ts apps/server/src/chat/service.test.ts apps/server/src/chat/steering.test.ts apps/server/src/chat/job-heading-memory.test.ts apps/server/src/chat/remote-planner.test.ts apps/server/src/plan/persistence.test.ts (297 pass)
Assistant-verification: bun run types; bun run ci
User-preference: Things should be polished like the Slack UX
Co-authored-by: Alex Lavaee <lavaman131@github.com>
@lavaman131
lavaman131 merged commit 16c55ff into main Nov 13, 2026
24 checks passed
@lavaman131
lavaman131 deleted the run-state-after-restart branch November 13, 2026 01:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant