Skip to content

Commit cd94e2c

Browse files
committed
feat(subagent): record typed terminal failures for the parent
A failed worker reached the parent as two booleans and error prose, so the parent could not tell a transient provider fault from a stuck or cancelled lane, and an answered ask_director left no trace once the run died. The store now writes one frozen failure record in the same mutation that makes a non-clean terminal visible, and provider_failure and continuable derive from its class so every projection agrees.
1 parent c5113a2 commit cd94e2c

14 files changed

Lines changed: 1159 additions & 49 deletions

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ Two directors, selected by role:
153153
Auto mode is toggled by CLI flags (`--auto` / `--no-auto`); there is currently no in-session key to toggle it (default on; constrained envelope — workspace writes and unconstrained shell auto-allow; installs, recursive rm, force/uncontained worktree changes, sensitive-path and opaque-wrapper shell still ask; contained non-force `git worktree add`/`remove`/`prune` and `list` auto-allow; shell file-mutation denied). It is not a separate edit/plan mode.
154154

155155
- **SubAgentDirector** (delegated work, `src/subagent/index.ts`) — Drives a dispatched worker until a turn arrives with no tool calls, then replies with the final assistant text and ends the run. A tool-less turn **after tools** completes only with the four-heading envelope (Summary, Findings, Blockers, Paths). Assistant text that prints explicit `<tool_call>` markup is treated as attempted tool use, not narration: one **verbatim-tool-call** nudge asks the worker to re-issue a real `tool_call` and does not count toward the tool-less spiral. A missing envelope otherwise nudges once (**incomplete-report**) and a second tool-less turn still without the envelope salvages as **incomplete-report-stop** (once; later tool-less turns wait). Explore/read-only workers that used tools then replied with findings remain normal completes; `requireEvidence` (off by default, set per director) additionally requires at least one read before a tool-less spawn-only reply can complete. `requirePlanSubstance` (planner or `intent=plan`, not `modelRole === "plan"`) additionally requires Findings to contain files/paths, acceptance criteria, non-goals, risks, and ordered steps with a non-placeholder line each — four headings with stub Findings are incomplete-report, not an attachable plan. After real tool work, wrap-up Findings that are not placeholder or outline-only complete instead of being salvaged as a stub plan. Reads done through `run_shell` count as evidence too — `src/subagent/shell-evidence.ts` classifies shell reads (`cat`, `grep`, `sed` without `-i`, …) over the same subject expansion the auto-shell policy uses — but there is no corresponding shell-write evidence or file-write requirement: a run that never touches a file still completes normally once it replies with the envelope. There is no turn budget. Operator/parent cancel after any progress returns a **cancelled** salvage report (partial findings + tool activity) instead of a bare cancel string; cancel before progress still surfaces as cancelled-by-operator. There is no repetition/no-progress/never-acted/never-edited hard stop and no fingerprint-based re-dispatch block — a genuinely stuck worker runs until it completes, stalls, hits an opt-in wall-clock deadline, or is cancelled.
156-
`spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound with a Summary + Blockers digest (Findings when Summary is empty) and a blob pointer named for `read_file`. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer, except a recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; there is no fingerprint-based re-dispatch hard-block. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.
156+
`spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound with a Summary + Blockers digest (Findings when Summary is empty) and a blob pointer named for `read_file`. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Every terminal non-clean outcome (failed, cancelled, interrupted, or a forced-stop salvage) also carries a `failure` record, written in the same store mutation that makes the outcome visible and never rewritten: a typed `failure_class` (`provider_retryable`, `provider_fatal`, `loop_guard` for the reactor's doom-loop guard, the forced-stop reasons, or `error`), `recovery` availability with a typed reason, the attempt number, the ask_director `handoff` state at termination (`delivered` only when the worker's own ask_director call returned the answer), and whether `cleanup` released everything. `provider_failure` and `continuable` derive from that record, so wait_agents, list_agents, and mailbox mail cannot disagree. Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer, except a recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; there is no fingerprint-based re-dispatch hard-block. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.
157157

158158
#### Model-family policy (`src/agent/model-family-policy.ts`)
159159

‎src/subagent/agent-fleet.test.ts‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1216,9 +1216,15 @@ describe("interrupt_agent unblocks wait_agents", () => {
12161216
agent_id: string;
12171217
status: string;
12181218
stop_reason?: string;
1219+
failure?: unknown;
12191220
}[];
12201221
expect(results).toEqual([
1221-
{ agent_id: id, status: "interrupted", stop_reason: "interrupted" },
1222+
{
1223+
agent_id: id,
1224+
status: "interrupted",
1225+
stop_reason: "interrupted",
1226+
failure: expect.objectContaining({ failure_class: "interrupted" }),
1227+
},
12221228
]);
12231229
expect(deps.sessions.get(id)?.lifecycleStatus).toBe("interrupted");
12241230
expect(deps.sessions.get(id)?.status).toBe("running");
@@ -1579,9 +1585,15 @@ describe("interrupt_agent unblocks wait_agents", () => {
15791585
agent_id: string;
15801586
status: string;
15811587
stop_reason?: string;
1588+
failure?: unknown;
15821589
}[];
15831590
expect(results).toEqual([
1584-
{ agent_id: id, status: "interrupted", stop_reason: "interrupted" },
1591+
{
1592+
agent_id: id,
1593+
status: "interrupted",
1594+
stop_reason: "interrupted",
1595+
failure: expect.objectContaining({ failure_class: "interrupted" }),
1596+
},
15851597
]);
15861598
expect(deps.fleetRecords.peek(id)?.status).toBe("interrupted");
15871599
expect(deps.fleetRecords.peek(id)?.collected).toBe(true);
@@ -2454,9 +2466,15 @@ describe("admission queue", () => {
24542466
agent_id: string;
24552467
status: string;
24562468
stop_reason?: string;
2469+
failure?: unknown;
24572470
}[];
24582471
expect(results).toEqual([
2459-
{ agent_id: queuedId, status: "interrupted", stop_reason: "cancelled" },
2472+
{
2473+
agent_id: queuedId,
2474+
status: "interrupted",
2475+
stop_reason: "cancelled",
2476+
failure: expect.objectContaining({ failure_class: "cancelled" }),
2477+
},
24602478
]);
24612479
expect(started()).toBe(1);
24622480

‎src/subagent/agent-fleet.ts‎

Lines changed: 36 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -122,9 +122,10 @@ import {
122122

123123
import { formatSubAgentSpawnAuthFailureMessage } from "./inference-auth-failure.js";
124124
import {
125-
isRecoverableProviderFailureCategory,
126-
isResolvedProviderFailureError,
127-
} from "../inference-error-message.js";
125+
classifySubAgentFailure,
126+
isProviderFailureClass,
127+
type SubAgentFailureRecord,
128+
} from "./terminal-failure.js";
128129
import { errorMessage } from "../agent/error-message.js";
129130
import { isSubAgentCancelError } from "./dispose.js";
130131
import {
@@ -143,9 +144,11 @@ interface FleetRecord {
143144
report?: string;
144145
error?: string;
145146
stopReason?: string;
147+
/** Derived from `failure.failure_class`. */
146148
providerFailure?: true;
147-
/** CL-8978: transient provider failure — the parent may spawn one successor. */
149+
/** CL-8978: derived from `failure.recovery.available`. */
148150
recoverableFailure?: true;
151+
failure?: SubAgentFailureRecord;
149152
/** Set once a wait_agents caller has been handed this result. */
150153
collected?: boolean;
151154
/** Set once a waiter or occupancy take handed report/error. */
@@ -178,9 +181,8 @@ interface FleetOverlay {
178181
bodyHanded?: boolean;
179182
tombstoned?: boolean;
180183
hint?: string;
181-
providerFailure?: true;
182-
/** CL-8978: transient provider failure — the parent may spawn one successor. */
183-
recoverableFailure?: true;
184+
/** Mirror of the session's failure record so it survives eviction. */
185+
failure?: SubAgentFailureRecord;
184186
}
185187

186188
const RECOVERY_HINT =
@@ -282,23 +284,6 @@ class FleetMailbox {
282284
this.sessions.wake();
283285
}
284286

285-
markProviderFailure(id: string): void {
286-
const existing = this.records.get(id);
287-
if (existing === undefined) return;
288-
existing.providerFailure = true;
289-
}
290-
291-
/**
292-
* CL-8978: stamp a transient (retryable/timeout/overload) provider failure
293-
* alongside sessions.fail. Survives session eviction like providerFailure —
294-
* snapshot projects it even once the payload is tombstoned.
295-
*/
296-
markRecoverable(id: string): void {
297-
const existing = this.records.get(id);
298-
if (existing === undefined) return;
299-
existing.recoverableFailure = true;
300-
}
301-
302287
markQueued(id: string): void {
303288
const existing = this.records.get(id);
304289
if (existing === undefined || existing.collected === true) return;
@@ -408,6 +393,7 @@ class FleetMailbox {
408393

409394
private rememberLiveWaitStatuses(): void {
410395
for (const [id, overlay] of this.records) {
396+
this.mirrorFailure(id, overlay);
411397
const wait = this.sessionWaitStatus(id);
412398
if (wait === undefined) continue;
413399
// Do not overwrite lastWait with the session's pending_init/running
@@ -417,6 +403,13 @@ class FleetMailbox {
417403
}
418404
}
419405

406+
private mirrorFailure(id: string, overlay: FleetOverlay): void {
407+
const session = this.sessions.get(id);
408+
if (session === undefined) return;
409+
if (session.failure !== undefined) overlay.failure = session.failure;
410+
else delete overlay.failure;
411+
}
412+
420413
private waitStatusFromEvicted(id: string): WaitJSONStatus | undefined {
421414
return waitStatusFromVerbLifecycle(this.sessions.evictedLifecycle(id));
422415
}
@@ -457,6 +450,7 @@ class FleetMailbox {
457450
return { status: "running" };
458451
}
459452
const session = this.sessions.get(id);
453+
this.mirrorFailure(id, overlay);
460454
if (
461455
session === undefined &&
462456
overlay.tombstoned !== true &&
@@ -484,6 +478,7 @@ class FleetMailbox {
484478
: payload?.stopReason;
485479
const ask =
486480
status === "awaiting_director" ? this.sessions.peekAsk(id) : undefined;
481+
const failure = isLiveWaitStatus(status) ? undefined : overlay.failure;
487482
return {
488483
status,
489484
...(overlay.collected === true ? { collected: true } : {}),
@@ -495,10 +490,13 @@ class FleetMailbox {
495490
? { error: payload.error }
496491
: {}),
497492
...(stopReason !== undefined ? { stopReason } : {}),
498-
...(overlay.providerFailure === true ? { providerFailure: true } : {}),
499-
...(overlay.recoverableFailure === true
493+
...(failure !== undefined && isProviderFailureClass(failure.failure_class)
494+
? { providerFailure: true }
495+
: {}),
496+
...(failure?.recovery.available === true
500497
? { recoverableFailure: true }
501498
: {}),
499+
...(failure !== undefined ? { failure } : {}),
502500
...(ask !== undefined
503501
? { question: ask.question, questionId: ask.questionId }
504502
: {}),
@@ -1727,28 +1725,23 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
17271725
deps.sessions.settleRun(session.id);
17281726
return;
17291727
}
1730-
const isProviderFailure = isResolvedProviderFailureError(err);
17311728
const diagnosticMessage = errorMessage(err);
17321729
const authMessage = formatSubAgentSpawnAuthFailureMessage(
17331730
description,
17341731
err,
17351732
);
17361733
const failReason = authMessage ?? diagnosticMessage;
1737-
if (isProviderFailure || providerFailureObserved) {
1738-
deps.fleetRecords.markProviderFailure(session.id);
1739-
}
1740-
// CL-8978: a classified transient failure stays wait-terminal
1741-
// failed, but carries a continuable marker so the parent can
1742-
// spawn one successor instead of stalling on the failure.
1743-
// Fatal categories (credential/quota/context-overflow) and
1744-
// unclassified throws never mark — no auto-retry is added here.
1745-
if (
1746-
isResolvedProviderFailureError(err) &&
1747-
isRecoverableProviderFailureCategory(err.category)
1748-
) {
1749-
deps.fleetRecords.markRecoverable(session.id);
1750-
}
1751-
deps.sessions.fail(session.id, failReason);
1734+
// CL-10202: the typed class is the single source for the
1735+
// provider_failure and continuable projections. A transient
1736+
// failure stays wait-terminal failed; nothing here retries.
1737+
deps.sessions.fail(session.id, failReason, {
1738+
failure_class: classifySubAgentFailure(err, {
1739+
providerFailureObserved,
1740+
}),
1741+
...(settlement?.teardown_failed === true
1742+
? { teardownFailed: true }
1743+
: {}),
1744+
});
17521745
})
17531746
.finally(() => {
17541747
activeLanes.delete(call.id);
@@ -2091,6 +2084,7 @@ export function createListAgentsTool(deps: WaitAgentsDeps): AgentTool {
20912084
}
20922085
: {}),
20932086
...(stopReason !== undefined ? { stop_reason: stopReason } : {}),
2087+
...(record?.failure !== undefined ? { failure: record.failure } : {}),
20942088
...(record?.status === "awaiting_director" &&
20952089
record.question !== undefined
20962090
? { question: record.question }

‎src/subagent/fleet-dry-drive.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -248,6 +248,7 @@ describe("collectUncollectedTerminals", () => {
248248
status: "failed",
249249
description: "failed lane",
250250
error: "boom",
251+
failure: expect.objectContaining({ failure_class: "error" }),
251252
});
252253
expect(mailbox.peek("done")?.collected).toBe(true);
253254
expect(mailbox.peek("fail")?.collected).toBe(true);

‎src/subagent/fleet-dry-drive.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import {
1111
} from "../plugins/result-truncation-plugin.js";
1212
import { isLiveWaitStatus, type WaitJSONStatus } from "./lifecycle.js";
1313
import { parseSubAgentReport } from "./report.js";
14+
import type { SubAgentFailureRecord } from "./terminal-failure.js";
1415

1516
/** Enough of a lane report for a parent continuation; traces stay on disk. */
1617
export const FLEET_DRY_REPORT_CHARS = 8_192;
@@ -44,6 +45,7 @@ export interface FleetDryMailboxRecord {
4445
/** CL-8978: transient provider failure — the parent may spawn one successor. */
4546
readonly recoverableFailure?: true;
4647
readonly stopReason?: string;
48+
readonly failure?: SubAgentFailureRecord;
4749
}
4850

4951
export interface FleetDryMailbox {
@@ -118,6 +120,7 @@ export interface CollectedWorkerReport {
118120
continuable?: true;
119121
continue_with?: string;
120122
stop_reason?: string;
123+
failure?: SubAgentFailureRecord;
121124
}
122125

123126
/** Mailbox parent payload: envelope digest plus a blob pointer, not the full report. */
@@ -137,6 +140,7 @@ export interface MailboxWorkerDigest {
137140
continuable?: true;
138141
continue_with?: string;
139142
stop_reason?: string;
143+
failure?: SubAgentFailureRecord;
140144
}
141145

142146
/**
@@ -344,6 +348,7 @@ export async function digestCollectedReport(
344348
...(report.stop_reason !== undefined
345349
? { stop_reason: report.stop_reason }
346350
: {}),
351+
...(report.failure !== undefined ? { failure: report.failure } : {}),
347352
};
348353
}
349354

@@ -442,6 +447,7 @@ export function projectMailboxRecord(
442447
...(taken.stopReason !== undefined
443448
? { stop_reason: taken.stopReason }
444449
: {}),
450+
...(taken.failure !== undefined ? { failure: taken.failure } : {}),
445451
};
446452
}
447453

0 commit comments

Comments
 (0)