Skip to content

Commit 77fe63b

Browse files
committed
feat(subagent): record typed terminal failures for the parent
A failed worker reached the parent as two booleans and error prose, so the parent could not tell a transient provider fault from a stuck or cancelled lane, and an answered ask_director left no trace once the run died. The store now writes one frozen failure record in the same mutation that makes a non-clean terminal visible, and provider_failure and continuable derive from its class so every projection agrees.
1 parent fb9b903 commit 77fe63b

14 files changed

Lines changed: 1472 additions & 73 deletions

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ Two directors, selected by role:
153153
Auto mode is toggled by CLI flags (`--auto` / `--no-auto`); there is currently no in-session key to toggle it (default on; constrained envelope — workspace writes and unconstrained shell auto-allow; installs, recursive rm, force/uncontained worktree changes, sensitive-path and opaque-wrapper shell still ask; contained non-force `git worktree add`/`remove`/`prune` and `list` auto-allow; shell file-mutation denied). It is not a separate edit/plan mode.
154154

155155
- **SubAgentDirector** (delegated work, `src/subagent/index.ts`) — Drives a dispatched worker until a turn arrives with no tool calls, then replies with the final assistant text and ends the run. A tool-less turn **after tools** completes only with the four-heading envelope (Summary, Findings, Blockers, Paths). Assistant text that prints explicit `<tool_call>` markup is treated as attempted tool use, not narration: one **verbatim-tool-call** nudge asks the worker to re-issue a real `tool_call` and does not count toward the tool-less spiral. A missing envelope otherwise nudges once (**incomplete-report**) and a second tool-less turn still without the envelope salvages as **incomplete-report-stop** (once; later tool-less turns wait). Explore/read-only workers that used tools then replied with findings remain normal completes; `requireEvidence` (off by default, set per director) additionally requires at least one read before a tool-less spawn-only reply can complete. `requirePlanSubstance` (planner or `intent=plan`, not `modelRole === "plan"`) additionally requires Findings to contain files/paths, acceptance criteria, non-goals, risks, and ordered steps with a non-placeholder line each — four headings with stub Findings are incomplete-report, not an attachable plan. After real tool work, wrap-up Findings that are not placeholder or outline-only complete instead of being salvaged as a stub plan. Reads done through `run_shell` count as evidence too — `src/subagent/shell-evidence.ts` classifies shell reads (`cat`, `grep`, `sed` without `-i`, …) over the same subject expansion the auto-shell policy uses — but there is no corresponding shell-write evidence or file-write requirement: a run that never touches a file still completes normally once it replies with the envelope. There is no turn budget. Operator/parent cancel after any progress returns a **cancelled** salvage report (partial findings + tool activity) instead of a bare cancel string; cancel before progress still surfaces as cancelled-by-operator. There is no repetition/no-progress/never-acted/never-edited hard stop and no fingerprint-based re-dispatch block — a genuinely stuck worker runs until it completes, stalls, hits an opt-in wall-clock deadline, or is cancelled.
156-
`spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound with a Summary + Blockers digest (Findings when Summary is empty) and a blob pointer named for `read_file`. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer, except a recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; there is no fingerprint-based re-dispatch hard-block. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.
156+
`spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound with a Summary + Blockers digest (Findings when Summary is empty) and a blob pointer named for `read_file`. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Every terminal non-clean outcome (failed, cancelled, interrupted, or a forced-stop salvage) also carries a `failure` record, written in the same store mutation that makes the outcome visible and never rewritten: a typed `failure_class` (`provider_retryable`, `provider_fatal`, `loop_guard` for the reactor's doom-loop guard, the forced-stop reasons, or `error`), `recovery` availability with a typed reason, the attempt number, the ask_director `handoff` state at termination (`delivered` only when the worker's own ask_director call returned the answer), and whether `cleanup` released everything. `provider_failure` and `continuable` derive from that record, so wait_agents, list_agents, and mailbox mail cannot disagree. Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer, except a recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; there is no fingerprint-based re-dispatch hard-block. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.
157157

158158
#### Model-family policy (`src/agent/model-family-policy.ts`)
159159

‎src/subagent/agent-fleet.test.ts‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1227,9 +1227,15 @@ describe("interrupt_agent unblocks wait_agents", () => {
12271227
agent_id: string;
12281228
status: string;
12291229
stop_reason?: string;
1230+
failure?: unknown;
12301231
}[];
12311232
expect(results).toEqual([
1232-
{ agent_id: id, status: "interrupted", stop_reason: "interrupted" },
1233+
{
1234+
agent_id: id,
1235+
status: "interrupted",
1236+
stop_reason: "interrupted",
1237+
failure: expect.objectContaining({ failure_class: "interrupted" }),
1238+
},
12331239
]);
12341240
expect(deps.sessions.get(id)?.lifecycleStatus).toBe("interrupted");
12351241
expect(deps.sessions.get(id)?.status).toBe("running");
@@ -1589,9 +1595,15 @@ describe("interrupt_agent unblocks wait_agents", () => {
15891595
agent_id: string;
15901596
status: string;
15911597
stop_reason?: string;
1598+
failure?: unknown;
15921599
}[];
15931600
expect(results).toEqual([
1594-
{ agent_id: id, status: "interrupted", stop_reason: "interrupted" },
1601+
{
1602+
agent_id: id,
1603+
status: "interrupted",
1604+
stop_reason: "interrupted",
1605+
failure: expect.objectContaining({ failure_class: "interrupted" }),
1606+
},
15951607
]);
15961608
expect(deps.fleetRecords.peek(id)?.status).toBe("interrupted");
15971609
expect(deps.fleetRecords.peek(id)?.collected).toBe(true);
@@ -2520,9 +2532,15 @@ describe("admission queue", () => {
25202532
agent_id: string;
25212533
status: string;
25222534
stop_reason?: string;
2535+
failure?: unknown;
25232536
}[];
25242537
expect(results).toEqual([
2525-
{ agent_id: queuedId, status: "interrupted", stop_reason: "cancelled" },
2538+
{
2539+
agent_id: queuedId,
2540+
status: "interrupted",
2541+
stop_reason: "cancelled",
2542+
failure: expect.objectContaining({ failure_class: "cancelled" }),
2543+
},
25262544
]);
25272545
expect(started()).toBe(1);
25282546

‎src/subagent/agent-fleet.ts‎

Lines changed: 38 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -111,9 +111,10 @@ import {
111111

112112
import { formatSubAgentSpawnAuthFailureMessage } from "./inference-auth-failure.js";
113113
import {
114-
isRecoverableProviderFailureCategory,
115-
isResolvedProviderFailureError,
116-
} from "../inference-error-message.js";
114+
classifySubAgentFailure,
115+
isProviderFailureClass,
116+
type SubAgentFailureRecord,
117+
} from "./terminal-failure.js";
117118
import { errorMessage } from "../agent/error-message.js";
118119
import { isSubAgentCancelError } from "./dispose.js";
119120
import {
@@ -132,9 +133,11 @@ interface FleetRecord {
132133
report?: string;
133134
error?: string;
134135
stopReason?: string;
136+
/** Derived from `failure.failure_class`. */
135137
providerFailure?: true;
136-
/** Transient provider failure — the parent may spawn one successor. */
138+
/** Transient provider failure, derived from `failure.recovery.available`. */
137139
recoverableFailure?: true;
140+
failure?: SubAgentFailureRecord;
138141
/** Set once a wait_agents caller has been handed this result. */
139142
collected?: boolean;
140143
/** Set once a waiter or occupancy take handed report/error. */
@@ -169,9 +172,8 @@ interface FleetOverlay {
169172
bodyHanded?: boolean;
170173
tombstoned?: boolean;
171174
hint?: string;
172-
providerFailure?: true;
173-
/** Transient provider failure — same meaning as the FleetRecord flag. */
174-
recoverableFailure?: true;
175+
/** Mirror of the session's failure record so it survives eviction. */
176+
failure?: SubAgentFailureRecord;
175177
}
176178

177179
const RECOVERY_HINT =
@@ -267,20 +269,6 @@ class FleetMailbox {
267269
this.sessions.wake();
268270
}
269271

270-
markProviderFailure(id: string): void {
271-
const existing = this.records.get(id);
272-
if (existing === undefined) return;
273-
existing.providerFailure = true;
274-
}
275-
276-
/** Stamp a transient provider failure beside sessions.fail; the snapshot
277-
* projects it even once the payload is tombstoned. */
278-
markRecoverable(id: string): void {
279-
const existing = this.records.get(id);
280-
if (existing === undefined) return;
281-
existing.recoverableFailure = true;
282-
}
283-
284272
markQueued(id: string): void {
285273
const existing = this.records.get(id);
286274
if (existing === undefined || existing.collected === true) return;
@@ -381,6 +369,7 @@ class FleetMailbox {
381369

382370
private rememberLiveWaitStatuses(): void {
383371
for (const [id, overlay] of this.records) {
372+
this.mirrorFailure(id, overlay);
384373
const wait = this.sessionWaitStatus(id);
385374
if (wait === undefined) continue;
386375
// Keep lastWait while admission-queued; the live projection would
@@ -390,6 +379,13 @@ class FleetMailbox {
390379
}
391380
}
392381

382+
private mirrorFailure(id: string, overlay: FleetOverlay): void {
383+
const session = this.sessions.get(id);
384+
if (session === undefined) return;
385+
if (session.failure !== undefined) overlay.failure = session.failure;
386+
else delete overlay.failure;
387+
}
388+
393389
private waitStatusFromEvicted(id: string): WaitJSONStatus | undefined {
394390
return waitStatusFromVerbLifecycle(this.sessions.evictedLifecycle(id));
395391
}
@@ -430,6 +426,7 @@ class FleetMailbox {
430426
return { status: "running" };
431427
}
432428
const session = this.sessions.get(id);
429+
this.mirrorFailure(id, overlay);
433430
if (
434431
session === undefined &&
435432
overlay.tombstoned !== true &&
@@ -457,6 +454,7 @@ class FleetMailbox {
457454
: payload?.stopReason;
458455
const ask =
459456
status === "awaiting_director" ? this.sessions.peekAsk(id) : undefined;
457+
const failure = isLiveWaitStatus(status) ? undefined : overlay.failure;
460458
return {
461459
status,
462460
...(overlay.collected === true ? { collected: true } : {}),
@@ -468,10 +466,13 @@ class FleetMailbox {
468466
? { error: payload.error }
469467
: {}),
470468
...(stopReason !== undefined ? { stopReason } : {}),
471-
...(overlay.providerFailure === true ? { providerFailure: true } : {}),
472-
...(overlay.recoverableFailure === true
469+
...(failure !== undefined && isProviderFailureClass(failure.failure_class)
470+
? { providerFailure: true }
471+
: {}),
472+
...(failure?.recovery.available === true
473473
? { recoverableFailure: true }
474474
: {}),
475+
...(failure !== undefined ? { failure } : {}),
475476
...(ask !== undefined
476477
? {
477478
question: ask.question,
@@ -1409,7 +1410,7 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
14091410
log.error("spawn_agent worktree setup failed: {error}", {
14101411
error: message,
14111412
});
1412-
deps.sessions.fail(session.id, message);
1413+
await deps.sessions.fail(session.id, message);
14131414
finalizeEnd(true);
14141415
admission.release(session.id);
14151416
return;
@@ -1660,7 +1661,7 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
16601661
: {}),
16611662
});
16621663
})
1663-
.catch((err) => {
1664+
.catch(async (err) => {
16641665
const alreadyCancelled =
16651666
deps.sessions.get(session.id)?.status === "cancelled";
16661667
if (
@@ -1673,25 +1674,23 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
16731674
deps.sessions.settleRun(session.id);
16741675
return;
16751676
}
1676-
const isProviderFailure = isResolvedProviderFailureError(err);
16771677
const diagnosticMessage = errorMessage(err);
16781678
const authMessage = formatSubAgentSpawnAuthFailureMessage(
16791679
description,
16801680
err,
16811681
);
16821682
const failReason = authMessage ?? diagnosticMessage;
1683-
if (isProviderFailure || providerFailureObserved) {
1684-
deps.fleetRecords.markProviderFailure(session.id);
1685-
}
1686-
// A transient failure stays wait-terminal failed but carries a
1687-
// continuable marker so the parent can spawn one successor.
1688-
if (
1689-
isResolvedProviderFailureError(err) &&
1690-
isRecoverableProviderFailureCategory(err.category)
1691-
) {
1692-
deps.fleetRecords.markRecoverable(session.id);
1693-
}
1694-
deps.sessions.fail(session.id, failReason);
1683+
// CL-10202: the typed class is the single source for the
1684+
// provider_failure and continuable projections. A transient
1685+
// failure stays wait-terminal failed; nothing here retries.
1686+
await deps.sessions.fail(session.id, failReason, {
1687+
failure_class: classifySubAgentFailure(err, {
1688+
providerFailureObserved,
1689+
}),
1690+
...(settlement?.teardown_failed === true
1691+
? { teardownFailed: true }
1692+
: {}),
1693+
});
16951694
})
16961695
.finally(() => {
16971696
activeLanes.delete(call.id);
@@ -2031,6 +2030,7 @@ export function createListAgentsTool(deps: WaitAgentsDeps): AgentTool {
20312030
}
20322031
: {}),
20332032
...(stopReason !== undefined ? { stop_reason: stopReason } : {}),
2033+
...(record?.failure !== undefined ? { failure: record.failure } : {}),
20342034
...(record?.status === "awaiting_director" &&
20352035
record.question !== undefined
20362036
? { question: record.question }

‎src/subagent/fleet-dry-drive.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -263,6 +263,7 @@ describe("collectUncollectedTerminals", () => {
263263
status: "failed",
264264
description: "failed lane",
265265
error: "boom",
266+
failure: expect.objectContaining({ failure_class: "error" }),
266267
});
267268
expect(mailbox.peek("done")?.collected).toBe(true);
268269
expect(mailbox.peek("fail")?.collected).toBe(true);

‎src/subagent/fleet-dry-drive.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import {
1111
} from "../plugins/result-truncation-plugin.js";
1212
import { isLiveWaitStatus, type WaitJSONStatus } from "./lifecycle.js";
1313
import { parseSubAgentReport } from "./report.js";
14+
import type { SubAgentFailureRecord } from "./terminal-failure.js";
1415

1516
/** Enough of a lane report for a parent continuation; traces stay on disk. */
1617
export const FLEET_DRY_REPORT_CHARS = 8_192;
@@ -44,6 +45,7 @@ export interface FleetDryMailboxRecord {
4445
/** Transient provider failure — the parent may spawn one successor. */
4546
readonly recoverableFailure?: true;
4647
readonly stopReason?: string;
48+
readonly failure?: SubAgentFailureRecord;
4749
}
4850

4951
export interface FleetDryMailbox {
@@ -118,6 +120,7 @@ export interface CollectedWorkerReport {
118120
continuable?: true;
119121
continue_with?: string;
120122
stop_reason?: string;
123+
failure?: SubAgentFailureRecord;
121124
}
122125

123126
/** Mailbox parent payload: envelope digest plus a blob pointer, not the full report. */
@@ -137,6 +140,7 @@ export interface MailboxWorkerDigest {
137140
continuable?: true;
138141
continue_with?: string;
139142
stop_reason?: string;
143+
failure?: SubAgentFailureRecord;
140144
}
141145

142146
/**
@@ -344,6 +348,7 @@ export async function digestCollectedReport(
344348
...(report.stop_reason !== undefined
345349
? { stop_reason: report.stop_reason }
346350
: {}),
351+
...(report.failure !== undefined ? { failure: report.failure } : {}),
347352
};
348353
}
349354

@@ -442,6 +447,7 @@ export function projectMailboxRecord(
442447
...(taken.stopReason !== undefined
443448
? { stop_reason: taken.stopReason }
444449
: {}),
450+
...(taken.failure !== undefined ? { failure: taken.failure } : {}),
445451
};
446452
}
447453

0 commit comments

Comments
 (0)