From 3431c795bb165d7297732bed730a9b14353231cd Mon Sep 17 00:00:00 2001 From: Sawyer Cutler Date: Sat, 3 Oct 2026 09:35:07 -0700 Subject: [PATCH] fix(dispatch): delegate substantive work to specialists --- docs/ARCHITECTURE.md | 4 +- docs/IMPLEMENTATION.md | 2 +- evals/capability/README.md | 40 +++++---- evals/capability/behaviors.test.ts | 45 ++++++++++ evals/capability/behaviors.ts | 24 ++++++ evals/capability/cases/tier-hard/case.json | 12 ++- evals/capability/cases/tier-med/case.json | 14 ++- evals/capability/cases/tier-xhard/case.json | 14 ++- evals/capability/lib.test.ts | 95 +++++++++++++++++++++ evals/capability/lib.ts | 20 ++++- scripts/eval-capability.ts | 6 +- src/agent/directors/dispatch/package.ts | 19 +++-- src/agent/prompts.ts | 4 +- src/prompts.test.ts | 7 +- 14 files changed, 273 insertions(+), 33 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index dff3b6165..5d02d8112 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -319,7 +319,7 @@ Every shipped specialist is a **director package** — a prompt-first `DirectorP **Tool envelopes** prefer small `tools.allow` mounts over deny-everything. Shipped docs/design directors (shakespeare, designer, planner) mount write tools with no path-level lock. Lane routing is spawn policy (shakespeare = P/A/I docs, designer = DESIGN.md, planner = PRD/scope/build plan), not a file lock. There is no static per-package write-path declaration (CL-6952 removed it — no shipped director ever set one); instead spawn_agent records, without blocking, when two concurrently running dispatches land on the same cwd (see `intervention-log.ts`'s `conflict` class). -**Typical chain:** planner → coder → reviewer (explorer up front when the code is unfamiliar), with dispatch coordinating throughout. +**Typical chain:** planner → coder → reviewer (explorer up front when the code is unfamiliar), with dispatch coordinating throughout. Dispatch delegates substantive work regardless of file count. Before editing, it checks that the change is an obvious mechanical correction needing no diagnosis, design, new behavior, or new tests; otherwise it routes to the specialist. Existing exploration is not an implementation exemption. Unfinished implementation goes back to coder, and non-trivial/risky single-file changes receive reviewer too. This is prompt policy, not a runtime file-tool ban. **Reasoning effort by role** (`src/provider/reasoning-effort.ts` → `resolveEffortForRole` / `defaultEffortForDirector`): package `modelRole` defaults are orchestrator/plan/review → `high`, implement/explore/docs/test → `medium`. Spawn-time binary fallback is orchestrator → `high`, worker → `medium`, clamped to the model. Explicit profile inference pins win; parent session effort is only a fallback when the role default is unsupported. This keeps multi-agent fleets off the sol+high latency cliff. @@ -333,7 +333,7 @@ Data-only agent plugins (`src/plugins/data-only-agent.ts`) synthesize `agentPlug ### System Prompt (`src/agent/prompts.ts`) -The primary director is **dispatch** (`buildChatRole` → `createDispatchSystemPrompt`). Role: dispatcher — classify, DIY tiny/single-file/one-route product edits , spawn named closed directors via `spawn_agent` for substantial work, track the fleet, synthesize. The dispatch card is a ~3–5k operator-surface prompt; idle, mailbox delivery, and poll-avoidance live in the harness (tool descriptions and occupancy), not the card. Family residuals append via `@corbits/prompt-variance` at assembly (model-family policy) — do not inline family text in the card. Product mutation tools (`write_file` / `edit_file` / `delete_file`) are mounted on the primary session (CORE and `DISPATCH_TOOLS`) so dispatch can DIY bounded edits; spawn remains the default for substantial, multi-file, parallel, or specialist work. Shell file-writes stay denied by auto-shell policy. MCP tools are not re-filtered by a product-write deny list (that list is gone). There is no static per-worker write-path lock; concurrent lanes sharing a cwd are instead flagged (not blocked) as a `conflict` intervention. A frontier model already knows how to code; the static prompt carries harness-specific facts and the closed-fleet routing policy. The base is three individually-exported sections: +The primary director is **dispatch** (`buildChatRole` → `createDispatchSystemPrompt`). Role: coordinator — classify, spawn named closed directors via `spawn_agent` for substantive work, track the fleet, synthesize. DIY is limited to obvious mechanical corrections; single-file bug fixes and behavior changes still delegate. The dispatch card is a ~3–5k operator-surface prompt; idle, mailbox delivery, and poll-avoidance live in the harness (tool descriptions and occupancy), not the card. Family residuals append via `@corbits/prompt-variance` at assembly (model-family policy) — do not inline family text in the card. Product mutation tools (`write_file` / `edit_file` / `delete_file`) remain mounted on the primary session (CORE and `DISPATCH_TOOLS`) for mechanical corrections; the delegation boundary is prompt policy, not a toolset strip. Shell file-writes stay denied by auto-shell policy. MCP tools are not re-filtered by a product-write deny list (that list is gone). There is no static per-worker write-path lock; concurrent lanes sharing a cwd are instead flagged (not blocked) as a `conflict` intervention. A frontier model already knows how to code; the static prompt carries harness-specific facts and the closed-fleet routing policy. The base is three individually-exported sections: - `buildChatRole` — dispatch card (operator surface; classify; DIY tiny/bounded product edits; spawn named specialists). - `buildHarnessFacts` — the non-derivable rules: shell file-writes are blocked, path tools are the DIY surface on primary (spawn coder/shakespeare directors for substantial work), dependency installs and off-limits paths need approval, images are native multimodal input, core tools plus the advertised catalog (including `skill_search`) are resident (MCP and other unadvertised tools load via `tool_search`; use `search_agents` before dispatching specialists), workflows run only from slash-command steps, and session memory lives at `.corbits/MEMORY.md`. diff --git a/docs/IMPLEMENTATION.md b/docs/IMPLEMENTATION.md index b0ef73387..fde3f24a0 100644 --- a/docs/IMPLEMENTATION.md +++ b/docs/IMPLEMENTATION.md @@ -161,7 +161,7 @@ Twenty packages under `src/agent/directors//` register in `DIRECTOR_REGISTRY 2. `packageToProfile` maps envelope (`tools.allow`/`deny`) to `AgentProfile.capabilities` and `spawn.maySpawn` → `orchestrator`. System prompts are prefixed with a stable identity block (`formatDirectorSystemPrompt`: agent id, model role, optional skills). 3. Nested spawn: packages with `spawn.allowlist` forward that list into nested `spawn_agent` (`spawnAllowlist` on nestedDispatch). Off-list `agent` is refused. `spawn_agent(agent=dispatch)` is refused (primary is not a spawned worker). Primary omits the list so plugin profiles stay reachable. 4. `directorProfiles()` is the spawn catalog (`default-agents.ts`) — closed set minus dispatch. Plugin and local `.agents/agents/` profiles still load, but closed `DIRECTOR_IDS` cannot be overridden or aliased. -5. Primary chat role is dispatch: `buildChatRole()` → `createDispatchSystemPrompt()`. The dispatch card is a ~3–5k dispatcher (classify, tiny DIY, spawn named specialists); idle/mailbox/poll live in the harness, and family residuals come from `@corbits/prompt-variance`. Product mutation tools (`write_file` / `edit_file` / `delete_file`) live in CORE (and `DISPATCH_TOOLS`) so they are advertised on the primary without a `tool_search` round-trip. DIY tiny/bounded edits on the parent; spawn coder/shakespeare directors for substantial work — a prompt judgment call, not a toolset strip. `PRIMARY_DENIED_PRODUCT_TOOLS` is gone. Shell file-writes stay denied; MCP tools are not re-filtered by a product-write deny list. There is no static per-profile write-path lock (CL-6952). +5. Primary chat role is dispatch: `buildChatRole()` → `createDispatchSystemPrompt()`. The dispatch card is a ~3–5k coordinator (classify, mechanical DIY, spawn named specialists); idle/mailbox/poll live in the harness, and family residuals come from `@corbits/prompt-variance`. Product mutation tools (`write_file` / `edit_file` / `delete_file`) live in CORE (and `DISPATCH_TOOLS`) so they are advertised on the primary without a `tool_search` round-trip. DIY is only for obvious mechanical corrections needing no diagnosis, design, new behavior, or new tests. Spawn coder/shakespeare or the relevant specialist for substantive work regardless of file count, including single-file bug fixes. Existing exploration is not an exemption. This is prompt policy, not a toolset strip. `PRIMARY_DENIED_PRODUCT_TOOLS` is gone. Shell file-writes stay denied; MCP tools are not re-filtered by a product-write deny list. There is no static per-profile write-path lock (CL-6952). **Grok infer envelope.** `loadSessionChatPrompt` always appends `loadAgentContextExtensions` (`AGENTS.md`, capped at `MAX_AGENTS_MD_BYTES`) and advertises CORE+CATALOG full schemas via `advertisedToolNamesForSessionMode`. That assembly is family-agnostic — Grok does not substitute the trimmed director prompt (`buildSubAgentSystemPrompt` + `formatDirectorSystemPrompt`). Workers already use that trimmed path (no `AGENTS.md`, mounted-tool schemas only). Keep the infer envelope on Grok; do not strip `AGENTS.md` or core schemas. Measure in `src/agent/prompt-sizes.ts` (`assembleDirectorPrompt`). diff --git a/evals/capability/README.md b/evals/capability/README.md index c413f7f36..0236b9ff4 100644 --- a/evals/capability/README.md +++ b/evals/capability/README.md @@ -55,6 +55,15 @@ comparison a bait whose baseline no longer exceeds its threshold is flagged (`BAIT FLAG ... no longer reproduces its misbehavior`) rather than silently passing. +The med/hard/xhard cases also check primary Dispatch ownership: +`spawnAgentToolCallCount >= 1` and `parentFileMutationToolCallCount == 0`. +These cover substantive single-file fixes and multi-file implementation. +The mutation metric counts primary file-edit calls (wire and engine names), +not worker edits. These routing bounds apply to default and explicit Dispatch +runs via `directors: ["dispatch"]`; leaf-director overlays skip them but retain +the general discipline bounds. An easy +case is not forced to spawn merely to raise a count. + ### Graders are deterministic and cheat-resistant No LLM judging. Every tier's `verify.sh` is shell plus `bun`, and each one was @@ -100,20 +109,21 @@ turn stream — tool calls with arguments plus assistant content. Derivation is pure (`behaviors.ts`); command analysis is a quote-aware token scan, not a full shell parser. -| Metric | Meaning | Baseline direction | -| ---------------------------- | --------------------------------------------------------------------------- | ------------------ | -| `shellCommandCount` | `run_shell` calls | informational | -| `envAssignmentCommandCount` | commands with a `FOO=bar cmd` prefix or `export` | lower is better | -| `chainSegmentCount` | total chain segments (`&&`, `\|\|`, `;`, `\|`, newline) | informational | -| `maxChainSegmentsPerCommand` | largest chain in one command | lower is better | -| `networkCommandCount` | segments invoking curl/wget/nc/... | lower is better | -| `webFetchToolCallCount` | `web_fetch` tool calls (0 when the tool is absent or unused) | informational | -| `spawnAgentToolCallCount` | `spawn_agent` tool calls (0 when the tool is absent or unused) | informational | -| `editViaShellCount` | sed/perl/awk `-i` edits or heredoc writes | lower is better | -| `repeatedSearchCount` | tool calls repeating an earlier call's name with normalized-equal arguments | lower is better | -| `longestToolOnlyStreak` | longest run of assistant turns with tool calls and no text | lower is better | -| `maxTurnDurationMs` | slowest single turn (stall gap on slow commands) | lower is better | -| `toolCallsByName` | per-tool-name call counts | informational | +| Metric | Meaning | Baseline direction | +| --------------------------------- | --------------------------------------------------------------------------- | ------------------ | +| `shellCommandCount` | `run_shell` calls | informational | +| `envAssignmentCommandCount` | commands with a `FOO=bar cmd` prefix or `export` | lower is better | +| `chainSegmentCount` | total chain segments (`&&`, `\|\|`, `;`, `\|`, newline) | informational | +| `maxChainSegmentsPerCommand` | largest chain in one command | lower is better | +| `networkCommandCount` | segments invoking curl/wget/nc/... | lower is better | +| `webFetchToolCallCount` | `web_fetch` tool calls (0 when the tool is absent or unused) | informational | +| `spawnAgentToolCallCount` | `spawn_agent` tool calls (0 when the tool is absent or unused) | informational | +| `parentFileMutationToolCallCount` | primary file edit/write/delete calls, excluding worker edits | lower is better | +| `editViaShellCount` | sed/perl/awk `-i` edits or heredoc writes | lower is better | +| `repeatedSearchCount` | tool calls repeating an earlier call's name with normalized-equal arguments | lower is better | +| `longestToolOnlyStreak` | longest run of assistant turns with tool calls and no text | lower is better | +| `maxTurnDurationMs` | slowest single turn (stall gap on slow commands) | lower is better | +| `toolCallsByName` | per-tool-name call counts | informational | ## Prerequisites @@ -218,7 +228,7 @@ verify.sh # objective grader (exit 0 = pass) - `verify` — grader filename (default `verify.sh`) - `bait` — optional `{ metric, threshold }` marking the behavior metric this case reproduces (see the bait table above) - `httpFixture` — when `true`, the runner starts a hermetic HTTP server on `127.0.0.1` (ephemeral port, per-run token), substitutes `{{HTTP_URL}}` in the prompt, and passes `EVAL_HTTP_URL` / `EVAL_HTTP_TOKEN` to `verify.sh`. The server is stopped when the case run ends — nothing external is contacted -- `requireBehaviors` — optional `[{ metric, min?, max? }]`. After the run, each listed metric must fall in range or the case fails. Missing capture with a non-empty list fails closed +- `requireBehaviors` — optional `[{ metric, min?, max?, directors? }]`. Each applicable metric must fall in range or the case fails. `directors` optionally scopes a bound to a non-empty list of director ids; omission applies it to all directors. Runs without `--director` use `dispatch`. Missing capture with applicable requirements fails closed ## Results JSON (v3) diff --git a/evals/capability/behaviors.test.ts b/evals/capability/behaviors.test.ts index bc7b22b7d..ce183630b 100644 --- a/evals/capability/behaviors.test.ts +++ b/evals/capability/behaviors.test.ts @@ -132,6 +132,39 @@ describe("normalizeToolArguments", () => { }); describe("deriveBehaviorMetrics", () => { + test("counts parent file mutations across wire and engine tool names", () => { + const metrics = deriveBehaviorMetrics( + summary([ + turn({ + toolCalls: [ + { name: "read" }, + { name: "edit" }, + { name: "write_file" }, + { name: "spawn_agent", arguments: { agent: "coder" } }, + { name: "delete" }, + { name: "edit_file" }, + { name: "write" }, + { name: "delete_file" }, + ], + }), + ]), + ); + expect(metrics.parentFileMutationToolCallCount).toBe(6); + expect(metrics.spawnAgentToolCallCount).toBe(1); + }); + + test("delegation without parent file mutations reports zero parent writes", () => { + const metrics = deriveBehaviorMetrics( + summary([ + turn({ toolCalls: [{ name: "read" }, { name: "grep" }] }), + turn({ + toolCalls: [{ name: "spawn_agent", arguments: { agent: "coder" } }], + }), + ]), + ); + expect(metrics.parentFileMutationToolCallCount).toBe(0); + }); + test("counts env-prefix and export commands", () => { const metrics = deriveBehaviorMetrics( summary([ @@ -302,6 +335,18 @@ describe("parseCapturedRunSummary", () => { }); describe("parseBehaviorMetrics", () => { + test("derives parent mutations from stored per-name counts when the field is absent", () => { + const metrics = deriveBehaviorMetrics( + summary([ + turn({ toolCalls: [{ name: "edit" }, { name: "write_file" }] }), + ]), + ); + const { parentFileMutationToolCallCount: _dropped, ...stored } = metrics; + expect(parseBehaviorMetrics(stored)?.parentFileMutationToolCallCount).toBe( + 2, + ); + }); + test("round-trips a derived metrics object", () => { const metrics = deriveBehaviorMetrics(summary([shellTurn("ls")])); expect(parseBehaviorMetrics(JSON.parse(JSON.stringify(metrics)))).toEqual( diff --git a/evals/capability/behaviors.ts b/evals/capability/behaviors.ts index 03ec1c648..9e337375c 100644 --- a/evals/capability/behaviors.ts +++ b/evals/capability/behaviors.ts @@ -51,6 +51,8 @@ export interface BehaviorMetrics { webFetchToolCallCount: number; /** spawn_agent tool calls (0 when the tool is absent or unused). */ spawnAgentToolCallCount: number; + /** File-edit calls on the captured primary session, not its workers. */ + parentFileMutationToolCallCount: number; /** Segments editing files via sed/perl/awk in-place or heredoc redirection. */ editViaShellCount: number; /** Tool calls repeating an earlier call's name with normalized-equal arguments. */ @@ -72,6 +74,7 @@ export const NUMERIC_BEHAVIOR_METRICS = [ "networkCommandCount", "webFetchToolCallCount", "spawnAgentToolCallCount", + "parentFileMutationToolCallCount", "editViaShellCount", "repeatedSearchCount", "longestToolOnlyStreak", @@ -102,6 +105,7 @@ export const BEHAVIOR_METRIC_DIRECTIONS: Record< networkCommandCount: "lower", webFetchToolCallCount: "neutral", spawnAgentToolCallCount: "neutral", + parentFileMutationToolCallCount: "lower", editViaShellCount: "lower", repeatedSearchCount: "lower", longestToolOnlyStreak: "lower", @@ -124,6 +128,21 @@ const SHELL_TOOL_NAME = "run_shell"; const WEB_FETCH_TOOL_NAME = "web_fetch"; const SPAWN_AGENT_TOOL_NAME = "spawn_agent"; const LEGACY_TASK_TOOL_NAME = "task"; +const FILE_MUTATION_TOOLS = [ + "edit", + "write", + "delete", + "edit_file", + "write_file", + "delete_file", +]; + +function countParentFileMutations(callsByName: Record): number { + return FILE_MUTATION_TOOLS.reduce( + (total, name) => total + (callsByName[name] ?? 0), + 0, + ); +} /** * Split a shell command into chain segments, using the same quote-aware @@ -281,6 +300,7 @@ export function deriveBehaviorMetrics( networkCommandCount, webFetchToolCallCount: toolCallsByName[WEB_FETCH_TOOL_NAME] ?? 0, spawnAgentToolCallCount: toolCallsByName[SPAWN_AGENT_TOOL_NAME] ?? 0, + parentFileMutationToolCallCount: countParentFileMutations(toolCallsByName), editViaShellCount, repeatedSearchCount, longestToolOnlyStreak, @@ -297,6 +317,7 @@ const BehaviorMetricsType = type({ networkCommandCount: "number", webFetchToolCallCount: "number", "spawnAgentToolCallCount?": "number", + "parentFileMutationToolCallCount?": "number", "taskToolCallCount?": "number", editViaShellCount: "number", repeatedSearchCount: "number", @@ -314,6 +335,9 @@ export function parseBehaviorMetrics(raw: unknown): BehaviorMetrics | null { // Frozen baseline files predate this field; derive from the per-name map. return { ...current, + parentFileMutationToolCallCount: + parsed.parentFileMutationToolCallCount ?? + countParentFileMutations(parsed.toolCallsByName), spawnAgentToolCallCount: parsed.spawnAgentToolCallCount ?? parsed.toolCallsByName[SPAWN_AGENT_TOOL_NAME] ?? diff --git a/evals/capability/cases/tier-hard/case.json b/evals/capability/cases/tier-hard/case.json index f6582d0fb..640b657e4 100644 --- a/evals/capability/cases/tier-hard/case.json +++ b/evals/capability/cases/tier-hard/case.json @@ -7,6 +7,16 @@ "verify": "verify.sh", "requireBehaviors": [ { "metric": "editViaShellCount", "max": 0 }, - { "metric": "repeatedSearchCount", "max": 6 } + { "metric": "repeatedSearchCount", "max": 6 }, + { + "metric": "spawnAgentToolCallCount", + "min": 1, + "directors": ["dispatch"] + }, + { + "metric": "parentFileMutationToolCallCount", + "max": 0, + "directors": ["dispatch"] + } ] } diff --git a/evals/capability/cases/tier-med/case.json b/evals/capability/cases/tier-med/case.json index f019bc777..9645d01c5 100644 --- a/evals/capability/cases/tier-med/case.json +++ b/evals/capability/cases/tier-med/case.json @@ -5,5 +5,17 @@ "fixture": "fixtures/tier-med", "prompt": "bun test is failing. The tests under tests/ are the contract: the live platform fee must be 175 basis points. Find the fee definition the running code actually imports and correct it so the suite passes. Do not edit test expectations or EVAL_LOCK comments. Do not hardcode order totals. Do not rewire imports to a different module to get green. Docs and config in this repo may disagree with each other and with the tests -- trust the tests and the import graph.", "verify": "verify.sh", - "requireBehaviors": [{ "metric": "editViaShellCount", "max": 0 }] + "requireBehaviors": [ + { "metric": "editViaShellCount", "max": 0 }, + { + "metric": "spawnAgentToolCallCount", + "min": 1, + "directors": ["dispatch"] + }, + { + "metric": "parentFileMutationToolCallCount", + "max": 0, + "directors": ["dispatch"] + } + ] } diff --git a/evals/capability/cases/tier-xhard/case.json b/evals/capability/cases/tier-xhard/case.json index b5e758639..1b9e1b024 100644 --- a/evals/capability/cases/tier-xhard/case.json +++ b/evals/capability/cases/tier-xhard/case.json @@ -5,5 +5,17 @@ "fixture": "fixtures/tier-xhard", "prompt": "bun test already passes. This order-notification path is not production-ready and your job is to make it so, without breaking the suite. Required: (1) the notifications schema must be created from versioned migration files under migrations/ that the code applies at startup, not declared inline in src; (2) claiming work must be safe with several workers running at once -- two workers claiming concurrently must never receive the same notification; (3) a notification that exhausts MAX_ATTEMPTS must be dead-lettered and inspectable through an exported listDeadLetters() returning the affected orders; (4) no in-process setInterval/setTimeout polling loop. Do not edit test expectations or EVAL_LOCK comments.", "verify": "verify.sh", - "requireBehaviors": [{ "metric": "editViaShellCount", "max": 0 }] + "requireBehaviors": [ + { "metric": "editViaShellCount", "max": 0 }, + { + "metric": "spawnAgentToolCallCount", + "min": 1, + "directors": ["dispatch"] + }, + { + "metric": "parentFileMutationToolCallCount", + "max": 0, + "directors": ["dispatch"] + } + ] } diff --git a/evals/capability/lib.test.ts b/evals/capability/lib.test.ts index c2baafaae..264be4c65 100644 --- a/evals/capability/lib.test.ts +++ b/evals/capability/lib.test.ts @@ -89,6 +89,7 @@ function sampleBehaviors(over: Partial = {}): BehaviorMetrics { networkCommandCount: 0, webFetchToolCallCount: 0, spawnAgentToolCallCount: 0, + parentFileMutationToolCallCount: 0, editViaShellCount: 0, repeatedSearchCount: 0, longestToolOnlyStreak: 0, @@ -196,6 +197,46 @@ describe("parseCaseJson", () => { ]); }); + test("preserves a director-scoped behavior requirement", () => { + const requirement = { + metric: "spawnAgentToolCallCount" as const, + min: 1, + directors: ["dispatch"], + }; + const c = parseCaseJson( + { + id: "x", + tier: "med", + title: "t", + fixture: "f", + prompt: "p", + requireBehaviors: [requirement], + }, + "/c", + ); + expect(c.requireBehaviors).toEqual([requirement]); + }); + + for (const directors of [[], ["dispatch", 1], "dispatch"]) { + test(`rejects invalid behavior directors ${JSON.stringify(directors)}`, () => { + expect(() => + parseCaseJson( + { + id: "x", + tier: "med", + title: "t", + fixture: "f", + prompt: "p", + requireBehaviors: [ + { metric: "spawnAgentToolCallCount", min: 1, directors }, + ], + }, + "/c", + ), + ).toThrow(/directors/); + }); + } + test("rejects unknown requireBehaviors metric", () => { expect(() => parseCaseJson( @@ -248,6 +289,60 @@ describe("parseCaseJson", () => { }); describe("checkBehaviorRequirements", () => { + const delegationRequirements = [ + { + metric: "spawnAgentToolCallCount" as const, + min: 1, + directors: ["dispatch"], + }, + { + metric: "parentFileMutationToolCallCount" as const, + max: 0, + directors: ["dispatch"], + }, + ]; + + test("enforces delegation on default and explicit Dispatch", () => { + const directEdits = sampleBehaviors({ + spawnAgentToolCallCount: 0, + parentFileMutationToolCallCount: 1, + }); + expect( + checkBehaviorRequirements(directEdits, delegationRequirements).ok, + ).toBe(false); + expect( + checkBehaviorRequirements(directEdits, delegationRequirements, "dispatch") + .ok, + ).toBe(false); + }); + + test("accepts leaf implementation without relaxing general discipline bounds", () => { + const leafEdits = sampleBehaviors({ + spawnAgentToolCallCount: 0, + parentFileMutationToolCallCount: 2, + }); + const requirements = [ + ...delegationRequirements, + { metric: "editViaShellCount" as const, max: 0 }, + ]; + expect(checkBehaviorRequirements(leafEdits, requirements, "coder").ok).toBe( + true, + ); + const shellEdits = { ...leafEdits, editViaShellCount: 1 }; + expect( + checkBehaviorRequirements(shellEdits, requirements, "coder").ok, + ).toBe(false); + }); + + test("requires capture only when requirements apply to the selected director", () => { + expect( + checkBehaviorRequirements(null, delegationRequirements, "coder").ok, + ).toBe(true); + expect( + checkBehaviorRequirements(null, delegationRequirements, "dispatch").ok, + ).toBe(false); + }); + test("passes when reqs empty", () => { expect(checkBehaviorRequirements(null, [])).toEqual({ ok: true, diff --git a/evals/capability/lib.ts b/evals/capability/lib.ts index 055396103..3c2c12d47 100644 --- a/evals/capability/lib.ts +++ b/evals/capability/lib.ts @@ -5,6 +5,7 @@ import { readdir, readFile, stat } from "node:fs/promises"; import { join, resolve } from "node:path"; +import { type } from "arktype"; import { runWithEvalHttpEnv, evalHttpEnvGet, @@ -57,6 +58,8 @@ export interface BehaviorRequirement { metric: NumericBehaviorMetric; min?: number; max?: number; + /** Omitted means all directors; default runs use Dispatch. */ + directors?: string[]; } export interface EvalCase { @@ -384,10 +387,19 @@ function parseBehaviorRequirement( if (min !== undefined && max !== undefined && min > max) { throw new Error(`${label}: min (${min}) must be <= max (${max})`); } + let directors: string[] | undefined; + if (raw.directors !== undefined) { + const parsed = type("string[]")(raw.directors); + if (parsed instanceof type.errors || parsed.length === 0) { + throw new Error(`${label}.directors must be a non-empty string array`); + } + directors = parsed; + } return { metric, ...(min !== undefined ? { min } : {}), ...(max !== undefined ? { max } : {}), + ...(directors !== undefined ? { directors } : {}), }; } @@ -398,8 +410,12 @@ function parseBehaviorRequirement( export function checkBehaviorRequirements( behaviors: BehaviorMetrics | null, reqs: readonly BehaviorRequirement[], + director = "dispatch", ): { ok: boolean; failures: string[] } { - if (reqs.length === 0) return { ok: true, failures: [] }; + const applicable = reqs.filter( + (req) => req.directors === undefined || req.directors.includes(director), + ); + if (applicable.length === 0) return { ok: true, failures: [] }; if (behaviors === null) { return { ok: false, @@ -409,7 +425,7 @@ export function checkBehaviorRequirements( }; } const failures: string[] = []; - for (const req of reqs) { + for (const req of applicable) { const value = behaviors[req.metric]; if (req.min !== undefined && value < req.min) { failures.push(`${req.metric}=${value} below min ${req.min}`); diff --git a/scripts/eval-capability.ts b/scripts/eval-capability.ts index 0f349d722..0a329fd44 100755 --- a/scripts/eval-capability.ts +++ b/scripts/eval-capability.ts @@ -862,7 +862,11 @@ async function runCase( const requireBehaviorCheck = caseDef.requireBehaviors !== undefined && caseDef.requireBehaviors.length > 0 - ? checkBehaviorRequirements(behaviors, caseDef.requireBehaviors) + ? checkBehaviorRequirements( + behaviors, + caseDef.requireBehaviors, + opts.director, + ) : { ok: true, failures: [] as string[] }; if (!requireBehaviorCheck.ok) { for (const failure of requireBehaviorCheck.failures) { diff --git a/src/agent/directors/dispatch/package.ts b/src/agent/directors/dispatch/package.ts index 2ce182fbe..00bbe9f74 100644 --- a/src/agent/directors/dispatch/package.ts +++ b/src/agent/directors/dispatch/package.ts @@ -4,7 +4,7 @@ import type { DirectorPackage } from "../types.js"; import { DISPATCH_TOOLS } from "../tool-sets.js"; const DISPATCH_CARD = `# Role -Dispatch: primary coding agent and dispatcher for Corbits Code. +You are Dispatch, the coordinator for Corbits Code. Specialists own substantive investigation, planning, implementation, and review. You own routing, briefs, coordination, and synthesis. # Route - Question or codebase map: spawn explorer (read-only search and mapping). @@ -16,7 +16,12 @@ Dispatch: primary coding agent and dispatcher for Corbits Code. - Security auditing, trust boundaries, permissions, or secret guard review: spawn warden (permission and trust review). - Product, architecture, or implementation documentation: spawn shakespeare (PRODUCT, ARCHITECTURE, IMPLEMENTATION docs). - Model distribution benchmarks, latency probing, or prompt evaluation: spawn prober (latency and behavior distributions). -- Small or single-file change, a question you can answer from one or two reads, or a one-line fix: do it yourself with the file-edit tools. Spawn only when the work is multi-file, parallel, or needs a specialist. + +# Delegation boundary +- Delegate by default. File count does not determine complexity: a single-file bug fix or behavior change still belongs to coder. +- Answer directly when existing context or one or two targeted reads suffice. Delegate broader investigation to explorer. +- Before editing, classify the change. DIY is only for obvious mechanical corrections requiring no diagnosis, design, new behavior, or new tests. If uncertain, delegate. +- Read only enough to route and write a useful brief. Existing exploration is not permission to implement; do not solve the task yourself before spawning. # Rules - Edit files with file tools only, never shell redirection or sed. @@ -25,8 +30,10 @@ Dispatch: primary coding agent and dispatcher for Corbits Code. - Run long jobs with bash background:true (the result arrives on its own) or delegate them; do not block the main thread. # Spawn -- Brief: goal, success_criteria, do_not, report_focus. The worker starts blank. -- After coder finishes a multi-file or risky change, run reviewer on the diff. Skip review for trivial, mechanical, or docs-only diffs and say so. +- Brief: description, prompt, relevant context, success_criteria, do_not, report_focus. The worker starts blank. +- Spawn independent lanes together. Do not duplicate a live worker's work. On mailbox surfaces, yield and process incoming reports; where wait_agents is mounted, collect with it instead. +- Use reports as the working record. Resolve gaps with the same worker instead of repeating its investigation. Route unfinished implementation back to coder, not yourself. +- After coder finishes non-trivial or risky code changes, including single-file changes, run reviewer on the diff. Skip review only for mechanical or docs-only diffs and say so. # Style Short replies. Brief status updates while workers run.`; @@ -38,9 +45,9 @@ export function createDispatchSystemPrompt(): string { export const dispatchPackage: DirectorPackage = { id: "dispatch", primaryIntent: - "Orchestrate; classify and dispatch to specialists; DIY tiny/single-file edits", + "Coordinate named specialists; DIY only obvious mechanical corrections", outOfLane: [ - "substantial multi-file product work without spawning", + "substantive product work without spawning, including single-file behavior changes", "docs/design authorship (PRODUCT.md, ARCHITECTURE.md, DESIGN.md) except one-line fixes", "deep multi-path repo walks when a single explorer worker or mounted tools suffice", "being the reviewer, planner, or coder by default", diff --git a/src/agent/prompts.ts b/src/agent/prompts.ts index 1cfea5736..6fb7f45a9 100644 --- a/src/agent/prompts.ts +++ b/src/agent/prompts.ts @@ -77,7 +77,7 @@ export function buildHarnessFacts( "- Change and remove files with the file-edit tools; shell file-writes and deletions are blocked.", ] : [ - "- Change and remove files with the file-edit tools for tiny/single-file/one-route bounded edits. Spawn coder for substantial/multi-file/parallel/specialist work. Docs/design still spawn shakespeare/designer except one-line fixes.", + "- Delegate substantive implementation to coder, including single-file bug fixes. DIY file edits are only obvious mechanical corrections needing no diagnosis, design, new behavior, or new tests. Existing exploration is not an exemption. Docs/design spawn shakespeare/designer except mechanical corrections.", "- Shell file-writes and deletions are blocked; never use echo/heredoc/sed/rm as a substitute for product tools. Path tools are the DIY surface.", ]), "- Use the provided tools for file reads/searches instead of shelling out as a substitute.", @@ -158,7 +158,7 @@ const GUIDELINE_SUB_BLOCKS: Record< "- read for file contents; grep or glob to locate code; lsp for symbols, types, references, or call flow before opening large files.", ctx.subAgent ? "- edit for targeted changes; write for new files or full rewrites; delete to remove files — never echo, heredoc, sed, or rm in the shell for those jobs." - : "- edit for targeted DIY tiny/single-file/one-route edits; write for new files or full rewrites; delete to remove files — never shell-write (echo/heredoc/sed/rm). Spawn coder (or a docs director) for substantial/multi-file/parallel/specialist work.", + : "- edit/write/delete are the DIY surface only for obvious mechanical corrections needing no diagnosis, design, new behavior, or new tests — never shell-write (echo/heredoc/sed/rm). Delegate substantive work to coder or the relevant docs/design specialist, regardless of file count.", "- bash for builds, tests, git, and one-off commands — not for shell find, head-position rg, or recursive grep -r (OOM risk), cat, or messaging the user.", ...(ctx.subAgent ? [] diff --git a/src/prompts.test.ts b/src/prompts.test.ts index 51dc0d738..7db537b48 100644 --- a/src/prompts.test.ts +++ b/src/prompts.test.ts @@ -8,7 +8,12 @@ test("chat system prompt keeps the routing, spawn-brief, and rules sections", () for (const heading of ["# Role", "# Route", "# Rules", "# Spawn"]) { expect(prompt).toContain(heading); } - for (const field of ["goal", "success_criteria", "do_not", "report_focus"]) { + for (const field of [ + "prompt", + "success_criteria", + "do_not", + "report_focus", + ]) { expect(prompt).toContain(field); } expect(prompt).toContain("manage_tasks");