Repository navigation
Keep a run live, and its questions asked, when its machine resumes after a server restart - #887
Merged
Merged
Conversation
…esumes after a restart After a deploy restarted the server, a run a machine was still working on showed as stopped in Chat, its stages gone, and the question it asked a few minutes later never became a Decision, so the run waited for an answer nobody could give. The machine had resumed its sessions a second after the restart, as designed. Two things then broke. The cards stopped because a document that opens after its client is back restored them as stopped: whether a restart's rebuilt hold still keeps runs was asked only of the hold before its client took it back. A client that resumes moves the sessions to its new link, where nothing in the new process retains them, so the answer was "nothing", and Chat stopped every live card. The sessions a restart rebuilt also had no way to reach the document: their run reports went nowhere, and a question on one was refused with "Decisions are not available to this session." A session a restart rebuilt now reaches its document through the room's host, which the document gives it when it opens, as it already did for the questions left open at the cut. Its run reports are Chat's run cards, its link dropping again marks them disconnected, and a document that opens while a link holds such a session with work shows its cards live, with what the client last reported. A question asked on it becomes Decisions as one asked before the restart did; the document is opened for it if nobody has it open and kept open until the question ends, and the answer goes to the client's request. Assistant-model: Claude Opus 5.5 Assistant-workflow: inline (subagent) Assistant-verification: bun test apps/server/src/harness/remote/handover-window.test.ts (37 pass; the 3 new restart-mid-run tests fail on 74772ee: card "stopped", question refused "Decisions are not available to this session.") Assistant-verification: bun test apps/server/src/harness/remote apps/server/src/harness/atomic/human-input.test.ts apps/server/src/testing/resume-after-restart.test.ts (342 pass) Assistant-verification: bun test apps/server/src/harness/session.test.ts apps/server/src/harness/atomic/full.test.ts apps/server/src/chat/invoke.test.ts apps/server/src/chat/service.test.ts apps/server/src/chat/steering.test.ts apps/server/src/chat/job-heading-memory.test.ts apps/server/src/chat/remote-planner.test.ts apps/server/src/plan/persistence.test.ts (297 pass) Assistant-verification: bun run types; bun run ci User-preference: Things should be polished like the Slack UX Co-authored-by: Alex Lavaee <lavaman131@github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the maintainer saw
A hosted deployment was redeployed while a machine's Planner had a workflow run under way. The machine's
connection dropped with close code 1012 and resumed one second later, as designed. Then:
on: it finished its verify stages, started a second round and published a new plan revision about six
minutes later.
the document, and not in Chat. The run waited for an answer nobody could give.
Cause
There are two faults. Both are in how the new process treats the sessions a restart rebuilt after their
client has taken them back.
The cards were restored as stopped. When a document opens, Chat restores its stored run cards. They
stay live only if
PlannerAttachments.restoredWorksays a rebuilt hold still keeps runs. That check lookedonly at the parked hold, from before its client came back. A client that resumes moves the sessions onto
its new link (
AgentLink.#hand), and the parked hold is cleared. So once the client was back, which tookone second, the check found nothing.
Chat.restorethen stopped every live card (restoreRuns→mergeRuns(…, undefined)). The browser reopened the document right after the client was back, at16:39:43, which is exactly when the strip flipped.
The resumed sessions could not reach the document.
AgentLink.rebuiltmade each session with a bareplanner,
{ channelId, isolated: false }, with noonRunsand nohumanInput. As a result:runsreports for a resumed session updated nothing in Chat. The protocol doc even saidso: "update nothing Chat shows until a Planner turn starts a session of its own."
input-requeston a resumed session was refused withfailed: "Decisions are not available to thissession." I reproduced this on 74772ee with the new test:
{"type":"input-result","request":"ask-2","status":"failed","message":"Decisions are not available to this session."}.No card was created, so there was nothing to show anywhere. The client was told its question had
failed.
Fix
A session a restart rebuilt now reaches its document through the room's host. The document already gave
this host to the link when it opened, so the link could wait again on questions left open at the cut
(
rejoinInputs). The changes:RejoinHost(harness/remote/link.ts) can now also:live)humanInput)runs)main.tsbuilds all three from the open plan.Each rebuilt session's planner gets the following:
onRunsandlinkChangedhand its reports to the host. When its link drops again, the cards aremarked disconnected, never stopped.
humanInputopens the document for the question if nobody has it open, through a newPlannerAttachmentsoption,openDocument. It keeps the document open until the question ends,and asks through the room's host.
rootOfis set, so a paused root still holds a workflow question's answer.rejoinInputsgives the host to the rebuilt sessions and publishes what each one last reported.So a document that opens after the client is back shows current stages right away.
restoredWorkalso answerslivewhen the attached link holds a rebuilt session with work. Thatmeans a live or paused run, an open question, or no report yet. It answers
downwhen that link hasdropped again. Cards are only ever stopped when the client says so (a
runsreport, or the linkending for good).
Chat.rebuiltRunsshows a rebuilt session's reports through the samepublishRunspath aretained session uses: stage storage, finish notices and card windows. It uses a stand-in owner per
session id.
publishRunsnow takes only what it needs of the room. Following up on a run that endedis passed in as a callback, so a rebuilt session never starts a Planner turn about its runs. That
matches remote sessions today (
followsUpRuns: false).createHumanInputtakes only the parts of the room it reads (a type-only change).Docs.
docs/agent-connection.md("After a server restart") anddocs/hosted-agent.mdnow say thata resumed session's runs and questions reach the document.
No UI changed, so there are no screenshots.
Tests
Three new tests in
apps/server/src/harness/remote/handover-window.test.ts, under "a workflow run itsclient carries on through a restart". Each one runs a real cut and takeover between two processes over one
database, with a raw agent-link client standing in for the machine. Before the cut, the first process's
Chat shows the run. The client resumes after the restart and reports its run still running, with a new
stage.
running(not stopped, notdisconnected), with the stage it reported.
input-requeston the resumed session, naming its workflow run)becomes a Decision. A member's answer reaches the client as
answered.answer reaches the client, and then the document is let go.
All three fail on 74772ee. Test 1 gets
"stopped". Tests 2 and 3 get no card, because the request isrefused with the message above. All three pass on this branch. The file's 34 other tests also pass.
The
reopenhelper in that file now wires the document the waymain.tsdoes (plannerKeepsRuns, and thehost's
live,humanInputandruns). ItsplannerKeepsRunsnames the link's document id, because thefixture's document and the link's have different ids.
Checks
bun test apps/server/src/harness/remote/handover-window.test.ts: 37 pass.bun test apps/server/src/harness/remote apps/server/src/harness/atomic/human-input.test.ts apps/server/src/testing/resume-after-restart.test.ts:342 pass.
bun testofharness/session,harness/atomic/full,chat/invoke,chat/service,chat/steering,chat/job-heading-memory,chat/remote-plannerandplan/persistence: 297 pass.bun run types: clean.bun run ci: clean. Lint warnings are back to the baseline count.Judgement calls
The document is opened only while a rebuilt session's question waits. It is not held for the whole
life of the run. Before the restart, a retained session held the document while its runs were live.
Nothing in the new process retains a rebuilt session, and holding the room until the run ends would
need a new end-of-work signal.
Run reports that arrive while the document is closed are not stored. The cards stay live, and the
latest report is shown the moment the document opens. A run that ends while nobody has the document
open gets its finish notice when someone next opens it.
A rebuilt session's cards and a retained session's cards can both exist in one Chat. They are
owned per session, as before. A later turn's session that reports the same run takes the card over.
Out of scope. Stop and pause on a run carried across a restart are still not wired to Chat's
buttons. Those buttons act on the retained session, and a rebuilt session has none. A follow-up can give
rebuilt sessions
controlthe waystart()does.