Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ Every shipped specialist is a **director package** — a prompt-first `DirectorP

**Tool envelopes** prefer small `tools.allow` mounts over deny-everything. Shipped docs/design directors (shakespeare, designer, planner) mount write tools with no path-level lock. Lane routing is spawn policy (shakespeare = P/A/I docs, designer = DESIGN.md, planner = PRD/scope/build plan), not a file lock. There is no static per-package write-path declaration (CL-6952 removed it — no shipped director ever set one); instead spawn_agent records, without blocking, when two concurrently running dispatches land on the same cwd (see `intervention-log.ts`'s `conflict` class).

**Typical chain:** planner → coder → reviewer (explorer up front when the code is unfamiliar), with dispatch coordinating throughout.
**Typical chain:** planner → coder → reviewer (explorer up front when the code is unfamiliar), with dispatch coordinating throughout. Dispatch delegates substantive work regardless of file count. Before editing, it checks that the change is an obvious mechanical correction needing no diagnosis, design, new behavior, or new tests; otherwise it routes to the specialist. Existing exploration is not an implementation exemption. Unfinished implementation goes back to coder, and non-trivial/risky single-file changes receive reviewer too. This is prompt policy, not a runtime file-tool ban.

**Reasoning effort by role** (`src/provider/reasoning-effort.ts` → `resolveEffortForRole` / `defaultEffortForDirector`): package `modelRole` defaults are orchestrator/plan/review → `high`, implement/explore/docs/test → `medium`. Spawn-time binary fallback is orchestrator → `high`, worker → `medium`, clamped to the model. Explicit profile inference pins win; parent session effort is only a fallback when the role default is unsupported. This keeps multi-agent fleets off the sol+high latency cliff.

Expand All @@ -333,7 +333,7 @@ Data-only agent plugins (`src/plugins/data-only-agent.ts`) synthesize `agentPlug

### System Prompt (`src/agent/prompts.ts`)

The primary director is **dispatch** (`buildChatRole` → `createDispatchSystemPrompt`). Role: dispatcher — classify, DIY tiny/single-file/one-route product edits , spawn named closed directors via `spawn_agent` for substantial work, track the fleet, synthesize. The dispatch card is a ~3–5k operator-surface prompt; idle, mailbox delivery, and poll-avoidance live in the harness (tool descriptions and occupancy), not the card. Family residuals append via `@corbits/prompt-variance` at assembly (model-family policy) — do not inline family text in the card. Product mutation tools (`write_file` / `edit_file` / `delete_file`) are mounted on the primary session (CORE and `DISPATCH_TOOLS`) so dispatch can DIY bounded edits; spawn remains the default for substantial, multi-file, parallel, or specialist work. Shell file-writes stay denied by auto-shell policy. MCP tools are not re-filtered by a product-write deny list (that list is gone). There is no static per-worker write-path lock; concurrent lanes sharing a cwd are instead flagged (not blocked) as a `conflict` intervention. A frontier model already knows how to code; the static prompt carries harness-specific facts and the closed-fleet routing policy. The base is three individually-exported sections:
The primary director is **dispatch** (`buildChatRole` → `createDispatchSystemPrompt`). Role: coordinator — classify, spawn named closed directors via `spawn_agent` for substantive work, track the fleet, synthesize. DIY is limited to obvious mechanical corrections; single-file bug fixes and behavior changes still delegate. The dispatch card is a ~3–5k operator-surface prompt; idle, mailbox delivery, and poll-avoidance live in the harness (tool descriptions and occupancy), not the card. Family residuals append via `@corbits/prompt-variance` at assembly (model-family policy) — do not inline family text in the card. Product mutation tools (`write_file` / `edit_file` / `delete_file`) remain mounted on the primary session (CORE and `DISPATCH_TOOLS`) for mechanical corrections; the delegation boundary is prompt policy, not a toolset strip. Shell file-writes stay denied by auto-shell policy. MCP tools are not re-filtered by a product-write deny list (that list is gone). There is no static per-worker write-path lock; concurrent lanes sharing a cwd are instead flagged (not blocked) as a `conflict` intervention. A frontier model already knows how to code; the static prompt carries harness-specific facts and the closed-fleet routing policy. The base is three individually-exported sections:

- `buildChatRole` — dispatch card (operator surface; classify; DIY tiny/bounded product edits; spawn named specialists).
- `buildHarnessFacts` — the non-derivable rules: shell file-writes are blocked, path tools are the DIY surface on primary (spawn coder/shakespeare directors for substantial work), dependency installs and off-limits paths need approval, images are native multimodal input, core tools plus the advertised catalog (including `skill_search`) are resident (MCP and other unadvertised tools load via `tool_search`; use `search_agents` before dispatching specialists), workflows run only from slash-command steps, and session memory lives at `.corbits/MEMORY.md`.
Expand Down
2 changes: 1 addition & 1 deletion docs/IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ Twenty packages under `src/agent/directors/<id>/` register in `DIRECTOR_REGISTRY
2. `packageToProfile` maps envelope (`tools.allow`/`deny`) to `AgentProfile.capabilities` and `spawn.maySpawn` → `orchestrator`. System prompts are prefixed with a stable identity block (`formatDirectorSystemPrompt`: agent id, model role, optional skills).
3. Nested spawn: packages with `spawn.allowlist` forward that list into nested `spawn_agent` (`spawnAllowlist` on nestedDispatch). Off-list `agent` is refused. `spawn_agent(agent=dispatch)` is refused (primary is not a spawned worker). Primary omits the list so plugin profiles stay reachable.
4. `directorProfiles()` is the spawn catalog (`default-agents.ts`) — closed set minus dispatch. Plugin and local `.agents/agents/` profiles still load, but closed `DIRECTOR_IDS` cannot be overridden or aliased.
5. Primary chat role is dispatch: `buildChatRole()` → `createDispatchSystemPrompt()`. The dispatch card is a ~3–5k dispatcher (classify, tiny DIY, spawn named specialists); idle/mailbox/poll live in the harness, and family residuals come from `@corbits/prompt-variance`. Product mutation tools (`write_file` / `edit_file` / `delete_file`) live in CORE (and `DISPATCH_TOOLS`) so they are advertised on the primary without a `tool_search` round-trip. DIY tiny/bounded edits on the parent; spawn coder/shakespeare directors for substantial work — a prompt judgment call, not a toolset strip. `PRIMARY_DENIED_PRODUCT_TOOLS` is gone. Shell file-writes stay denied; MCP tools are not re-filtered by a product-write deny list. There is no static per-profile write-path lock (CL-6952).
5. Primary chat role is dispatch: `buildChatRole()` → `createDispatchSystemPrompt()`. The dispatch card is a ~3–5k coordinator (classify, mechanical DIY, spawn named specialists); idle/mailbox/poll live in the harness, and family residuals come from `@corbits/prompt-variance`. Product mutation tools (`write_file` / `edit_file` / `delete_file`) live in CORE (and `DISPATCH_TOOLS`) so they are advertised on the primary without a `tool_search` round-trip. DIY is only for obvious mechanical corrections needing no diagnosis, design, new behavior, or new tests. Spawn coder/shakespeare or the relevant specialist for substantive work regardless of file count, including single-file bug fixes. Existing exploration is not an exemption. This is prompt policy, not a toolset strip. `PRIMARY_DENIED_PRODUCT_TOOLS` is gone. Shell file-writes stay denied; MCP tools are not re-filtered by a product-write deny list. There is no static per-profile write-path lock (CL-6952).

**Grok infer envelope.** `loadSessionChatPrompt` always appends `loadAgentContextExtensions` (`AGENTS.md`, capped at `MAX_AGENTS_MD_BYTES`) and advertises CORE+CATALOG full schemas via `advertisedToolNamesForSessionMode`. That assembly is family-agnostic — Grok does not substitute the trimmed director prompt (`buildSubAgentSystemPrompt` + `formatDirectorSystemPrompt`). Workers already use that trimmed path (no `AGENTS.md`, mounted-tool schemas only). Keep the infer envelope on Grok; do not strip `AGENTS.md` or core schemas. Measure in `src/agent/prompt-sizes.ts` (`assembleDirectorPrompt`).

Expand Down
40 changes: 25 additions & 15 deletions evals/capability/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,15 @@ comparison a bait whose baseline no longer exceeds its threshold is flagged
(`BAIT FLAG ... no longer reproduces its misbehavior`) rather than silently
passing.

The med/hard/xhard cases also check primary Dispatch ownership:
`spawnAgentToolCallCount >= 1` and `parentFileMutationToolCallCount == 0`.
These cover substantive single-file fixes and multi-file implementation.
The mutation metric counts primary file-edit calls (wire and engine names),
not worker edits. These routing bounds apply to default and explicit Dispatch
runs via `directors: ["dispatch"]`; leaf-director overlays skip them but retain
the general discipline bounds. An easy
case is not forced to spawn merely to raise a count.

### Graders are deterministic and cheat-resistant

No LLM judging. Every tier's `verify.sh` is shell plus `bun`, and each one was
Expand Down Expand Up @@ -100,20 +109,21 @@ turn stream — tool calls with arguments plus assistant content. Derivation is
pure (`behaviors.ts`); command analysis is a quote-aware token scan, not a full
shell parser.

| Metric | Meaning | Baseline direction |
| ---------------------------- | --------------------------------------------------------------------------- | ------------------ |
| `shellCommandCount` | `run_shell` calls | informational |
| `envAssignmentCommandCount` | commands with a `FOO=bar cmd` prefix or `export` | lower is better |
| `chainSegmentCount` | total chain segments (`&&`, `\|\|`, `;`, `\|`, newline) | informational |
| `maxChainSegmentsPerCommand` | largest chain in one command | lower is better |
| `networkCommandCount` | segments invoking curl/wget/nc/... | lower is better |
| `webFetchToolCallCount` | `web_fetch` tool calls (0 when the tool is absent or unused) | informational |
| `spawnAgentToolCallCount` | `spawn_agent` tool calls (0 when the tool is absent or unused) | informational |
| `editViaShellCount` | sed/perl/awk `-i` edits or heredoc writes | lower is better |
| `repeatedSearchCount` | tool calls repeating an earlier call's name with normalized-equal arguments | lower is better |
| `longestToolOnlyStreak` | longest run of assistant turns with tool calls and no text | lower is better |
| `maxTurnDurationMs` | slowest single turn (stall gap on slow commands) | lower is better |
| `toolCallsByName` | per-tool-name call counts | informational |
| Metric | Meaning | Baseline direction |
| --------------------------------- | --------------------------------------------------------------------------- | ------------------ |
| `shellCommandCount` | `run_shell` calls | informational |
| `envAssignmentCommandCount` | commands with a `FOO=bar cmd` prefix or `export` | lower is better |
| `chainSegmentCount` | total chain segments (`&&`, `\|\|`, `;`, `\|`, newline) | informational |
| `maxChainSegmentsPerCommand` | largest chain in one command | lower is better |
| `networkCommandCount` | segments invoking curl/wget/nc/... | lower is better |
| `webFetchToolCallCount` | `web_fetch` tool calls (0 when the tool is absent or unused) | informational |
| `spawnAgentToolCallCount` | `spawn_agent` tool calls (0 when the tool is absent or unused) | informational |
| `parentFileMutationToolCallCount` | primary file edit/write/delete calls, excluding worker edits | lower is better |
| `editViaShellCount` | sed/perl/awk `-i` edits or heredoc writes | lower is better |
| `repeatedSearchCount` | tool calls repeating an earlier call's name with normalized-equal arguments | lower is better |
| `longestToolOnlyStreak` | longest run of assistant turns with tool calls and no text | lower is better |
| `maxTurnDurationMs` | slowest single turn (stall gap on slow commands) | lower is better |
| `toolCallsByName` | per-tool-name call counts | informational |

## Prerequisites

Expand Down Expand Up @@ -218,7 +228,7 @@ verify.sh # objective grader (exit 0 = pass)
- `verify` — grader filename (default `verify.sh`)
- `bait` — optional `{ metric, threshold }` marking the behavior metric this case reproduces (see the bait table above)
- `httpFixture` — when `true`, the runner starts a hermetic HTTP server on `127.0.0.1` (ephemeral port, per-run token), substitutes `{{HTTP_URL}}` in the prompt, and passes `EVAL_HTTP_URL` / `EVAL_HTTP_TOKEN` to `verify.sh`. The server is stopped when the case run ends — nothing external is contacted
- `requireBehaviors` — optional `[{ metric, min?, max? }]`. After the run, each listed metric must fall in range or the case fails. Missing capture with a non-empty list fails closed
- `requireBehaviors` — optional `[{ metric, min?, max?, directors? }]`. Each applicable metric must fall in range or the case fails. `directors` optionally scopes a bound to a non-empty list of director ids; omission applies it to all directors. Runs without `--director` use `dispatch`. Missing capture with applicable requirements fails closed

## Results JSON (v3)

Expand Down
45 changes: 45 additions & 0 deletions evals/capability/behaviors.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,39 @@ describe("normalizeToolArguments", () => {
});

describe("deriveBehaviorMetrics", () => {
test("counts parent file mutations across wire and engine tool names", () => {
const metrics = deriveBehaviorMetrics(
summary([
turn({
toolCalls: [
{ name: "read" },
{ name: "edit" },
{ name: "write_file" },
{ name: "spawn_agent", arguments: { agent: "coder" } },
{ name: "delete" },
{ name: "edit_file" },
{ name: "write" },
{ name: "delete_file" },
],
}),
]),
);
expect(metrics.parentFileMutationToolCallCount).toBe(6);
expect(metrics.spawnAgentToolCallCount).toBe(1);
});

test("delegation without parent file mutations reports zero parent writes", () => {
const metrics = deriveBehaviorMetrics(
summary([
turn({ toolCalls: [{ name: "read" }, { name: "grep" }] }),
turn({
toolCalls: [{ name: "spawn_agent", arguments: { agent: "coder" } }],
}),
]),
);
expect(metrics.parentFileMutationToolCallCount).toBe(0);
});

test("counts env-prefix and export commands", () => {
const metrics = deriveBehaviorMetrics(
summary([
Expand Down Expand Up @@ -302,6 +335,18 @@ describe("parseCapturedRunSummary", () => {
});

describe("parseBehaviorMetrics", () => {
test("derives parent mutations from stored per-name counts when the field is absent", () => {
const metrics = deriveBehaviorMetrics(
summary([
turn({ toolCalls: [{ name: "edit" }, { name: "write_file" }] }),
]),
);
const { parentFileMutationToolCallCount: _dropped, ...stored } = metrics;
expect(parseBehaviorMetrics(stored)?.parentFileMutationToolCallCount).toBe(
2,
);
});

test("round-trips a derived metrics object", () => {
const metrics = deriveBehaviorMetrics(summary([shellTurn("ls")]));
expect(parseBehaviorMetrics(JSON.parse(JSON.stringify(metrics)))).toEqual(
Expand Down
24 changes: 24 additions & 0 deletions evals/capability/behaviors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ export interface BehaviorMetrics {
webFetchToolCallCount: number;
/** spawn_agent tool calls (0 when the tool is absent or unused). */
spawnAgentToolCallCount: number;
/** File-edit calls on the captured primary session, not its workers. */
parentFileMutationToolCallCount: number;
/** Segments editing files via sed/perl/awk in-place or heredoc redirection. */
editViaShellCount: number;
/** Tool calls repeating an earlier call's name with normalized-equal arguments. */
Expand All @@ -72,6 +74,7 @@ export const NUMERIC_BEHAVIOR_METRICS = [
"networkCommandCount",
"webFetchToolCallCount",
"spawnAgentToolCallCount",
"parentFileMutationToolCallCount",
"editViaShellCount",
"repeatedSearchCount",
"longestToolOnlyStreak",
Expand Down Expand Up @@ -102,6 +105,7 @@ export const BEHAVIOR_METRIC_DIRECTIONS: Record<
networkCommandCount: "lower",
webFetchToolCallCount: "neutral",
spawnAgentToolCallCount: "neutral",
parentFileMutationToolCallCount: "lower",
editViaShellCount: "lower",
repeatedSearchCount: "lower",
longestToolOnlyStreak: "lower",
Expand All @@ -124,6 +128,21 @@ const SHELL_TOOL_NAME = "run_shell";
const WEB_FETCH_TOOL_NAME = "web_fetch";
const SPAWN_AGENT_TOOL_NAME = "spawn_agent";
const LEGACY_TASK_TOOL_NAME = "task";
const FILE_MUTATION_TOOLS = [
"edit",
"write",
"delete",
"edit_file",
"write_file",
"delete_file",
];

function countParentFileMutations(callsByName: Record<string, number>): number {
return FILE_MUTATION_TOOLS.reduce(
(total, name) => total + (callsByName[name] ?? 0),
0,
);
}

/**
* Split a shell command into chain segments, using the same quote-aware
Expand Down Expand Up @@ -281,6 +300,7 @@ export function deriveBehaviorMetrics(
networkCommandCount,
webFetchToolCallCount: toolCallsByName[WEB_FETCH_TOOL_NAME] ?? 0,
spawnAgentToolCallCount: toolCallsByName[SPAWN_AGENT_TOOL_NAME] ?? 0,
parentFileMutationToolCallCount: countParentFileMutations(toolCallsByName),
editViaShellCount,
repeatedSearchCount,
longestToolOnlyStreak,
Expand All @@ -297,6 +317,7 @@ const BehaviorMetricsType = type({
networkCommandCount: "number",
webFetchToolCallCount: "number",
"spawnAgentToolCallCount?": "number",
"parentFileMutationToolCallCount?": "number",
"taskToolCallCount?": "number",
editViaShellCount: "number",
repeatedSearchCount: "number",
Expand All @@ -314,6 +335,9 @@ export function parseBehaviorMetrics(raw: unknown): BehaviorMetrics | null {
// Frozen baseline files predate this field; derive from the per-name map.
return {
...current,
parentFileMutationToolCallCount:
parsed.parentFileMutationToolCallCount ??
countParentFileMutations(parsed.toolCallsByName),
spawnAgentToolCallCount:
parsed.spawnAgentToolCallCount ??
parsed.toolCallsByName[SPAWN_AGENT_TOOL_NAME] ??
Expand Down
12 changes: 11 additions & 1 deletion evals/capability/cases/tier-hard/case.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@
"verify": "verify.sh",
"requireBehaviors": [
{ "metric": "editViaShellCount", "max": 0 },
{ "metric": "repeatedSearchCount", "max": 6 }
{ "metric": "repeatedSearchCount", "max": 6 },
{
"metric": "spawnAgentToolCallCount",
"min": 1,
"directors": ["dispatch"]
},
{
"metric": "parentFileMutationToolCallCount",
"max": 0,
"directors": ["dispatch"]
}
]
}
Loading
Loading