Skip to content

Commit ffe0494

Browse files
committed
feat(subagent): enforce one recovery per failure via spawn_agent
The single-successor hint for a transient worker failure was advisory: nothing tied a successor to the failed run, counted it, or stopped a second one. spawn_agent now takes recovers, the failed agent id. The claim is recorded synchronously before the handler awaits, so duplicate or concurrent calls get the same replacement back and spawn nothing; an ineligible request is refused with its typed reason and never falls back to a plain spawn. A replacement is attempt 2, so its own failure is recovery_exhausted, and it is an ordinary spawn that inherits no grants, approvals, or pending asks. Reverting this restores the advisory guidance.
1 parent 114d12a commit ffe0494

7 files changed

Lines changed: 612 additions & 12 deletions

File tree

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ Two directors, selected by role:
153153
Auto mode is toggled by CLI flags (`--auto` / `--no-auto`); there is currently no in-session key to toggle it (default on; constrained envelope — workspace writes and unconstrained shell auto-allow; installs, recursive rm, force/uncontained worktree changes, sensitive-path and opaque-wrapper shell still ask; contained non-force `git worktree add`/`remove`/`prune` and `list` auto-allow; shell file-mutation denied). It is not a separate edit/plan mode.
154154

155155
- **SubAgentDirector** (delegated work, `src/subagent/index.ts`) — Drives a dispatched worker until a turn arrives with no tool calls, then replies with the final assistant text and ends the run. A tool-less turn **after tools** completes only with the four-heading envelope (Summary, Findings, Blockers, Paths). Assistant text that prints explicit `<tool_call>` markup is treated as attempted tool use, not narration: one **verbatim-tool-call** nudge asks the worker to re-issue a real `tool_call` and does not count toward the tool-less spiral. A missing envelope otherwise nudges once (**incomplete-report**) and a second tool-less turn still without the envelope salvages as **incomplete-report-stop** (once; later tool-less turns wait). Explore/read-only workers that used tools then replied with findings remain normal completes; `requireEvidence` (off by default, set per director) additionally requires at least one read before a tool-less spawn-only reply can complete. `requirePlanSubstance` (planner or `intent=plan`, not `modelRole === "plan"`) additionally requires Findings to contain files/paths, acceptance criteria, non-goals, risks, and ordered steps with a non-placeholder line each — four headings with stub Findings are incomplete-report, not an attachable plan. After real tool work, wrap-up Findings that are not placeholder or outline-only complete instead of being salvaged as a stub plan. Reads done through `run_shell` count as evidence too — `src/subagent/shell-evidence.ts` classifies shell reads (`cat`, `grep`, `sed` without `-i`, …) over the same subject expansion the auto-shell policy uses — but there is no corresponding shell-write evidence or file-write requirement: a run that never touches a file still completes normally once it replies with the envelope. There is no turn budget. Operator/parent cancel after any progress returns a **cancelled** salvage report (partial findings + tool activity) instead of a bare cancel string; cancel before progress still surfaces as cancelled-by-operator. There is no repetition/no-progress/never-acted/never-edited hard stop and no fingerprint-based re-dispatch block — a genuinely stuck worker runs until it completes, stalls, hits an opt-in wall-clock deadline, or is cancelled.
156-
`spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound with a Summary + Blockers digest (Findings when Summary is empty) and a blob pointer named for `read_file`. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Every terminal non-clean outcome (failed, cancelled, interrupted, or a forced-stop salvage) also carries a `failure` record, written in the same store mutation that makes the outcome visible and never rewritten: a typed `failure_class` (`provider_retryable`, `provider_fatal`, `loop_guard` for the reactor's doom-loop guard, the forced-stop reasons, or `error`), `recovery` availability with a typed reason, the attempt number, the ask_director `handoff` state at termination (`delivered` only when the worker's own ask_director call returned the answer), and whether `cleanup` released everything. `provider_failure` and `continuable` derive from that record, so wait_agents, list_agents, and mailbox mail cannot disagree. Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer, except a recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; there is no fingerprint-based re-dispatch hard-block. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.
156+
`spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound with a Summary + Blockers digest (Findings when Summary is empty) and a blob pointer named for `read_file`. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Every terminal non-clean outcome (failed, cancelled, interrupted, or a forced-stop salvage) also carries a `failure` record, written in the same store mutation that makes the outcome visible and never rewritten: a typed `failure_class` (`provider_retryable`, `provider_fatal`, `loop_guard` for the reactor's doom-loop guard, the forced-stop reasons, or `error`), `recovery` availability with a typed reason, the attempt number, the ask_director `handoff` state at termination (`delivered` only when the worker's own ask_director call returned the answer), and whether `cleanup` released everything. `provider_failure` and `continuable` derive from that record, so wait_agents, list_agents, and mailbox mail cannot disagree. Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer, except a recoverable/continuable child failure (`continuable: true`), which the parent recovers by calling `spawn_agent` with `recovers` set to the failed agent_id; there is no fingerprint-based re-dispatch hard-block. Recovery is enforced, not advisory: `recovery.available` requires a `provider_retryable` failure on the original spawn (not a resume_agent turn), no prior recovery, an original attempt (a failed replacement is `recovery_exhausted`), and a replay-safe run that touched only read, search, and web read tools (anything else is `side_effects_completed`). The claim lands synchronously before the handler awaits, so a duplicate or concurrent `recovers` call gets the existing replacement back and spawns nothing; an ineligible one is refused with its reason and never falls back to a plain spawn. The replacement is an ordinary spawn: it inherits no grants, approvals, or pending asks from the failed worker. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.
157157

158158
#### Model-family policy (`src/agent/model-family-policy.ts`)
159159

‎src/agent/prompts.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,7 @@ const GUIDELINE_SUB_BLOCKS: Record<
213213
(ctx.waitAgentsMounted
214214
? " or re-wait"
215215
: ", or idle for its mailbox mail") +
216-
"; do not `spawn_agent` a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancel (`stop_reason` cancelled): wait for the operator; do not auto-retry. Identical brief: refuse. Recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; identical brief is still refused otherwise. Merge Summary/Findings into a coherent answer for the operator; do not paste raw fleet-agent dumps.",
216+
"; do not `spawn_agent` a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancel (`stop_reason` cancelled): wait for the operator; do not auto-retry. Identical brief: refuse. Recoverable/continuable child failure (`continuable: true`): recover it at most once with `spawn_agent` and `recovers` set to its agent_id; the runtime refuses any other recovery with a reason. Identical brief is still refused otherwise. Merge Summary/Findings into a coherent answer for the operator; do not paste raw fleet-agent dumps.",
217217
"- Use manage_tasks for your own coordination checklist; spawning workers is `spawn_agent`, not manage_tasks.",
218218
"- If context is compacted automatically, do not stop tasks early due to token fear; persist progress via manage_tasks and worker reports.",
219219
];

‎src/subagent/agent-fleet.ts‎

Lines changed: 91 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,9 @@ import { formatSubAgentSpawnAuthFailureMessage } from "./inference-auth-failure.
124124
import {
125125
classifySubAgentFailure,
126126
isProviderFailureClass,
127+
withRecoveryReplacement,
127128
type SubAgentFailureRecord,
129+
type SubAgentRecoveryReason,
128130
} from "./terminal-failure.js";
129131
import { errorMessage } from "../agent/error-message.js";
130132
import { isSubAgentCancelError } from "./dispose.js";
@@ -284,6 +286,25 @@ class FleetMailbox {
284286
this.sessions.wake();
285287
}
286288

289+
/**
290+
* Record that `replacementId` recovers `failedId`. Runs synchronously in
291+
* the spawn handler before anything awaits, so a second claim in the same
292+
* tick already sees the replacement. An evicted session's record lives
293+
* only on the overlay, so it is linked there.
294+
*/
295+
claimRecovery(failedId: string, replacementId: string): void {
296+
const overlay = this.records.get(failedId);
297+
if (overlay === undefined) return;
298+
if (this.sessions.get(failedId) !== undefined) {
299+
this.sessions.linkRecoveryReplacement(failedId, replacementId);
300+
this.mirrorFailure(failedId, overlay);
301+
return;
302+
}
303+
if (overlay.failure !== undefined) {
304+
overlay.failure = withRecoveryReplacement(overlay.failure, replacementId);
305+
}
306+
}
307+
287308
markQueued(id: string): void {
288309
const existing = this.records.get(id);
289310
if (existing === undefined || existing.collected === true) return;
@@ -558,6 +579,7 @@ const SpawnAgentArgs = type({
558579
"do_not?": "string[]",
559580
"report_focus?": "string",
560581
"requires_tools?": "string[]",
582+
"recovers?": "string",
561583
});
562584

563585
export const spawnAgentToolDefinition: ToolDefinition = {
@@ -610,6 +632,11 @@ export const spawnAgentToolDefinition: ToolDefinition = {
610632
description:
611633
"Hard tool requirements (canonical names). Preflight fails closed when missing.",
612634
},
635+
recovers: {
636+
type: "string",
637+
description:
638+
"agent_id of a failed+continuable worker this spawn replaces. Once per failure; refused with a reason otherwise.",
639+
},
613640
},
614641
required: ["description", "prompt"],
615642
},
@@ -627,7 +654,7 @@ export const MAX_WAIT_TIMEOUT_MS = 300_000;
627654
export const waitAgentsToolDefinition: ToolDefinition = {
628655
name: "wait_agents",
629656
description:
630-
"Block until targets finish, fail, or ask (awaiting_director), or timeout_ms. mode any (default) or all. Timeout is not an error; workers keep running. Answer awaiting_director with send_input. failed+continuable:true may be respawned once.",
657+
"Block until targets finish, fail, or ask (awaiting_director), or timeout_ms. mode any (default) or all. Timeout is not an error; workers keep running. Answer awaiting_director with send_input. failed+continuable:true: recover once via spawn_agent recovers=agent_id.",
631658
inputSchema: {
632659
type: "object",
633660
properties: {
@@ -915,6 +942,49 @@ function handoffRequiresSuccessCriteria(
915942
);
916943
}
917944

945+
/**
946+
* Why a `recovers` request spawns nothing, or undefined when it may proceed.
947+
* A duplicate is not an error: it hands back the replacement already
948+
* started so every caller converges on one agent_id.
949+
*/
950+
function recoveryRefusal(
951+
callId: string,
952+
fleetRecords: FleetMailboxHandle,
953+
failedId: string,
954+
): ToolResult | undefined {
955+
const record = fleetRecords.peek(failedId);
956+
if (record === undefined) {
957+
return fleetResult(
958+
callId,
959+
`Error: spawn_agent recovers="${failedId}" is not one of your workers; nothing was spawned.`,
960+
);
961+
}
962+
const replacementId = record.failure?.recovery.replacement_id;
963+
if (replacementId !== undefined) {
964+
return fleetResult(
965+
callId,
966+
fleetJson({
967+
agent_id: replacementId,
968+
recovers: failedId,
969+
status: "recovery_already_started",
970+
reason: "already_recovered" satisfies SubAgentRecoveryReason,
971+
}),
972+
);
973+
}
974+
if (
975+
record.status === "failed" &&
976+
record.failure?.recovery.available === true
977+
) {
978+
return undefined;
979+
}
980+
const reason: SubAgentRecoveryReason =
981+
record.failure?.recovery.reason ?? "not_retryable";
982+
return fleetResult(
983+
callId,
984+
`Error: recovery_unavailable reason=${reason}: spawn_agent recovers="${failedId}" spawned nothing. Diagnose from the failure and continue with a changed brief, or wait for the operator.`,
985+
);
986+
}
987+
918988
export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
919989
const telemetry = deps.telemetry ?? NOOP_TELEMETRY;
920990
// Concurrent-lane overlap detection, replacing the static per-package
@@ -995,6 +1065,7 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
9951065
do_not: rawDoNot,
9961066
report_focus: rawReportFocus,
9971067
requires_tools: rawRequiresTools,
1068+
recovers: rawRecovers,
9981069
} = parsed;
9991070
const description = rawDesc.trim();
10001071
const prompt = rawPrompt.trim();
@@ -1004,6 +1075,11 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
10041075
"Error: spawn_agent requires a non-empty description and prompt.",
10051076
);
10061077
}
1078+
const recovers = rawRecovers?.trim();
1079+
if (recovers !== undefined) {
1080+
const refusal = recoveryRefusal(call.id, deps.fleetRecords, recovers);
1081+
if (refusal !== undefined) return refusal;
1082+
}
10071083
const context = rawCtx?.trim();
10081084
const goals =
10091085
rawGoals?.map((g) => g.trim()).filter((g) => g.length > 0) ?? [];
@@ -1254,6 +1330,11 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
12541330
: {}),
12551331
});
12561332

1333+
// CL-10202: the claim lands before this handler's first await (and
1334+
// before the session exists), so a concurrent duplicate sees it.
1335+
if (recovers !== undefined) {
1336+
deps.fleetRecords.claimRecovery(recovers, call.id);
1337+
}
12571338
const session = deps.sessions.start({
12581339
id: call.id,
12591340
description,
@@ -1268,6 +1349,7 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
12681349
...(deps.parentSessionId !== undefined
12691350
? { parentSessionId: deps.parentSessionId }
12701351
: {}),
1352+
...(recovers !== undefined ? { recovers } : {}),
12711353
});
12721354
deps.fleetRecords.register(session.id);
12731355
const agentName = classifyAgentName(resolved.directorId);
@@ -1760,7 +1842,14 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
17601842
start,
17611843
});
17621844
if (status === "queued") deps.fleetRecords.markQueued(session.id);
1763-
return fleetResult(call.id, fleetJson({ agent_id: session.id, status }));
1845+
return fleetResult(
1846+
call.id,
1847+
fleetJson({
1848+
agent_id: session.id,
1849+
status,
1850+
...(recovers !== undefined ? { recovers, attempt: 2 } : {}),
1851+
}),
1852+
);
17641853
},
17651854
});
17661855
}

‎src/subagent/fleet-dry-drive.ts‎

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -144,13 +144,12 @@ export interface MailboxWorkerDigest {
144144
}
145145

146146
/**
147-
* Single-successor guidance for a failed+continuable entry. The marker is
148-
* advisory only — no runtime auto-retry backs it.
147+
* Guidance for a failed+continuable entry. spawn_agent enforces the "at most
148+
* once": a second recovers call gets the first replacement back.
149149
*/
150-
export const RECOVERABLE_FAILURE_CONTINUE_GUIDANCE =
151-
"This worker failed with a transient provider error (retryable/timeout/overload) " +
152-
"and is terminal — do not re-wait it. You may spawn at most one successor with " +
153-
"the same brief; do not retry in a loop.";
150+
function recoverableFailureContinueGuidance(agentId: string): string {
151+
return `Recoverable: call spawn_agent with recovers="${agentId}" at most once.`;
152+
}
154153

155154
export function shouldDriveOpenTasks(input: {
156155
previousRunning?: number | undefined;
@@ -441,7 +440,7 @@ export function projectMailboxRecord(
441440
...(taken.status === "failed" && taken.recoverableFailure === true
442441
? {
443442
continuable: true as const,
444-
continue_with: RECOVERABLE_FAILURE_CONTINUE_GUIDANCE,
443+
continue_with: recoverableFailureContinueGuidance(id),
445444
}
446445
: {}),
447446
...(taken.stopReason !== undefined

0 commit comments

Comments
 (0)