Skip to content

Commit 26f07f0

Browse files
TheGreatAxiosTheGreatAxios
committed
test: drop hollow prompt and span pins
1 parent 85fe9af commit 26f07f0

9 files changed

Lines changed: 41 additions & 853 deletions

File tree

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -346,7 +346,7 @@ The primary director is **dispatch** (`buildChatRole` → `createDispatchSystemP
346346
- `buildHarnessFacts` — the non-derivable rules: shell file-writes are blocked, path tools are the DIY surface on primary (spawn coder/shakespeare directors for substantial work), dependency installs and off-limits paths need approval, images are native multimodal input, core tools plus the advertised catalog (including `skill_search`) are resident (MCP and other unadvertised tools load via `tool_search`; use `search_agents` before dispatching specialists), workflows run only from slash-command steps, and session memory lives at `.corbits/MEMORY.md`.
347347
- `buildGuidelines` — be concise, prefer `spawn_agent` (mailbox mail on TUI; exec-primary `wait_agents`) for substantial product work with one focused task per worker and fan-out by independent lanes, DIY tiny/bounded edits on the parent, answer questions and diagnose visual/product feedback before editing, work autonomously for explicit coding tasks, use `lsp` for symbol work, and require implementation agents to run the repository-defined typecheck, relevant tests, and every defined full verification command. Agents report exact commands, outcomes, and exit statuses; a repository with no typecheck command produces an explicit Blocker backed by project-configuration evidence rather than an invented command or silent skip.
348348
- `buildPromptDisciplineBlock` — a shared, prohibition-form section appended exactly once to every built prompt (chat and sub-agent, every provider family). Primary vs worker wording differs for product writes: workers are told to use `read_file`/`edit_file`/`write_file`; dispatch is told to DIY tiny/bounded edits with those path tools and spawn directors for substantial work. Shared rules: never `cat`/`sed`/heredoc/`echo` for file work, no setting or exporting environment variables (recurring needs belong in project settings), `web_fetch`/`web_search` instead of `curl`/`wget`/hand-rolled queries, one operation per `run_shell` call, turn semantics (a tool-less reply is the final answer, no repeat searches, stop and change approach after three failed attempts, batch independent reads in parallel), and TTY output rules (short bold headers, one-line bullets, backticks for paths/commands, no wide tables).
349-
- `@corbits/prompt-fragments` — the only copy of director card prose. Cards compose the identity banner, report-envelope shape, DONE GATE / OUT OF LANE frames, and `DISPATCH_ROUTE_ROWS`; director modules call the composed card and do not restate it. Composed bytes are hash-pinned in `src/agent/directors/prompt-card.test.ts`; route rows are drift-pinned against `DIRECTOR_REGISTRY`.
349+
- `@corbits/prompt-fragments` — the only copy of director card prose. Cards compose the identity banner, report-envelope shape, DONE GATE / OUT OF LANE frames, and `DISPATCH_ROUTE_ROWS`; director modules call the composed card and do not restate it. Registry system prompts are the composed cards (`src/agent/directors/prompt-card.test.ts`); route rows are drift-pinned against `DIRECTOR_REGISTRY`.
350350

351351
**Provider-conditional residuals.** Per-family additions layer on top of the shared block via the same `ModelFamilyPolicy` mechanism the directors use (`src/subagent/provider-family.ts`, `src/agent/model-family-policy.ts`) — additive lines, never prompt forks. **Grok** leaves get `buildGrokLeafAntiThrashNote` (gated by `shouldApplyGrokAntiThrash` / `applyGrokFinishBias`, withheld from orchestrators): a compact finish-bias reinforcement plus a one-line reminder to route file/web work through the dedicated tools rather than `run_shell`, motivated by observed tool-routing thrash on the same harness. **Kimi** intentionally has no residual yet — `detectModelFamily` already resolves the family so callers can branch on it, but the prompt seam is left unfilled pending eval characterization of Kimi's behavior, mirroring the provisional (permissive-default) policy in `model-family-policy.ts`.
352352

‎packages/prompt-fragments/src/cards.ts‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -294,8 +294,7 @@ BLINDERS ON: measure what the brief's success_criteria ask for, on the harness b
294294
over \`evals/capability\`) — the product non-TUI path (\`loadConfig\` +
295295
\`runExec\`) against fixture copies with \`verify.sh\` graders.
296296
- Latency spans: the PerfTrace harness under \`src/perf\` — \`rollup.ts\`
297-
(TTFT vs stream split), \`assert-spans.ts\` (span assertions), and
298-
\`attribution-report.ts\` (per-turn attribution).
297+
(TTFT vs stream split) and \`attribution-report.ts\` (per-turn attribution).
299298
- Read how they invoke before running; extend neither. If the harness
300299
cannot answer the brief, report that under Blockers — do not hand-roll
301300
a replacement harness.

‎packages/prompt-variance/src/rows.test.ts‎

Lines changed: 0 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -6,17 +6,10 @@ import {
66
FAMILY_ROWS,
77
gptRow,
88
grokRow,
9-
grokToolBudgetResidual,
109
museRow,
1110
type PromptVarianceFamily,
1211
} from "./rows.js";
1312

14-
const CEREMONY_LINES = [
15-
"Never run git add, git commit, git stash, or any other state-changing git command unless the user asks.",
16-
"Do not narrate a plan before acting on a small task; act, then report.",
17-
"Verify with the test command once at the end, not after every edit.",
18-
];
19-
2013
describe("prompt-variance family rows", () => {
2114
test("ships exactly the default/muse/grok/claude/gpt families", () => {
2215
expect([...FAMILY_IDS]).toEqual([
@@ -61,55 +54,4 @@ describe("prompt-variance family rows", () => {
6154
"muse",
6255
]);
6356
});
64-
65-
test("muse row is the shipped CL-7869 tool-discipline text", () => {
66-
expect(museRow.residual).toContain("Tool discipline:");
67-
expect(museRow.residual).toContain("Batch independent tool calls");
68-
expect(museRow.residual).toContain("Never re-read a file");
69-
expect(museRow.residual).toContain("Do not narrate; act.");
70-
});
71-
72-
test("grok row is the single finish-bias + ceremony block (CL-8296)", () => {
73-
expect(grokRow.residual).toContain("Finish bias (xAI / Grok worker):");
74-
expect(grokRow.residual).toContain("prefer the structured report");
75-
expect(grokRow.residual).toContain("re-open paths you already read");
76-
expect(grokRow.residual).toContain("done-definition is met");
77-
expect(grokRow.residual).toContain("never bash");
78-
});
79-
80-
test("each ceremony line appears exactly once in the grok row (P2 invariant)", () => {
81-
for (const line of CEREMONY_LINES) {
82-
const occurrences = grokRow.residual.split(line).length - 1;
83-
expect(occurrences).toBe(1);
84-
}
85-
});
86-
87-
test("grok tool-budget residual is ceremony-free and budget text appears once (P1 invariant)", () => {
88-
expect(grokToolBudgetResidual).toContain("Tool budget:");
89-
for (const line of CEREMONY_LINES) {
90-
expect(grokToolBudgetResidual).not.toContain(line);
91-
}
92-
const budgetOccurrences =
93-
grokToolBudgetResidual.split("Tool budget:").length - 1;
94-
expect(budgetOccurrences).toBe(1);
95-
});
96-
97-
test("default row carries no residual", () => {
98-
expect(defaultRow.residual).toBe("");
99-
});
100-
101-
test("claude row is the single XML task_guidance block (CL-8309)", () => {
102-
expect(claudeRow.residual).toContain("<task_guidance>");
103-
expect(claudeRow.residual).toContain("</task_guidance>");
104-
expect(claudeRow.residual).toContain("Follow the dispatch brief exactly");
105-
expect(claudeRow.residual).toContain("Batch independent tool calls");
106-
});
107-
108-
test("gpt row is the narrate-before-tools nudge (CL-8310)", () => {
109-
expect(gptRow.residual).toContain("Narrate before tools (GPT worker):");
110-
expect(gptRow.residual).toContain(
111-
"one short line saying what you are doing",
112-
);
113-
expect(gptRow.residual).toContain("done-definition is met");
114-
});
11557
});

‎src/agent/canonical-tool-name.test.ts‎

Lines changed: 0 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
import { describe, expect, test } from "bun:test";
2-
import { join } from "node:path";
32
import { isSameTool } from "./canonical-tool-name.js";
43

54
describe("isSameTool", () => {
@@ -50,76 +49,3 @@ describe("isSameTool", () => {
5049
expect(isSameTool(b, a)).toBe(false);
5150
});
5251
});
53-
54-
// Callsites that compare incoming wire names must go through isSameTool (or
55-
// the canonical modules themselves). A new raw `name === "<tool>"` compare in
56-
// one of these files reintroduces the wire-rename breakage on the next rename,
57-
// so this test fails until the callsite is converted.
58-
describe("tool-name comparison regression guard", () => {
59-
const CALLSITES = [
60-
"src/agent/director.ts",
61-
"src/subagent/poll-exempt.ts",
62-
"src/tui/mcp-view.ts",
63-
"src/tui/runtime-bridge.ts",
64-
"src/tui/stall-watchdog.ts",
65-
"src/agent/tool-execution-watchdog.ts",
66-
"src/tui/turns-to-blocks.ts",
67-
];
68-
69-
const TOOL_LITERALS = new Set([
70-
"read_file",
71-
"write_file",
72-
"edit_file",
73-
"delete_file",
74-
"run_shell",
75-
"search_files",
76-
"manage_tasks",
77-
"use_skill",
78-
"web_fetch",
79-
"web_search",
80-
"ask_operator",
81-
"ask_director",
82-
"wait_agents",
83-
"spawn_agent",
84-
"submit_output",
85-
"submit_plan",
86-
"present",
87-
"list_agents",
88-
"read",
89-
"write",
90-
"edit",
91-
"delete",
92-
"bash",
93-
"shell",
94-
"glob",
95-
"todowrite",
96-
"skill",
97-
"webfetch",
98-
"websearch",
99-
"question",
100-
"update_plan",
101-
"wait",
102-
]);
103-
104-
test("no raw tool-name ===/!== compares outside the canonical modules", async () => {
105-
const root = join(import.meta.dirname, "..", "..");
106-
const violations: string[] = [];
107-
for (const file of CALLSITES) {
108-
const text = await Bun.file(join(root, file)).text();
109-
for (const [index, line] of text.split("\n").entries()) {
110-
const match = /(?:\?\.name|\.name)\s*(===|!==)\s*(["'])([^"']+)\2/.exec(
111-
line,
112-
);
113-
if (match?.[3] !== undefined && TOOL_LITERALS.has(match[3])) {
114-
violations.push(`${file}:${index + 1}: ${line.trim()}`);
115-
}
116-
const canonical =
117-
/canonicalToolName\(([^)]+)\)\s*(===|!==)\s*["']/.exec(line);
118-
if (canonical !== null) {
119-
violations.push(`${file}:${index + 1}: ${line.trim()}`);
120-
}
121-
}
122-
}
123-
expect(violations).toEqual([]);
124-
});
125-
});

‎src/agent/directors/prompt-card.test.ts‎

Lines changed: 0 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
import { createHash } from "node:crypto";
2-
import { readFileSync } from "node:fs";
31
import { describe, expect, test } from "bun:test";
42

53
import {
@@ -14,43 +12,13 @@ import {
1412
import { DIRECTOR_REGISTRY } from "./registry.js";
1513
import { DIRECTOR_IDS } from "./types.js";
1614

17-
/**
18-
* Composed-output pin (CL-10674). The fragment package is the only copy of
19-
* the card prose; these hashes fail when a fragment edit changes bytes.
20-
* Director modules must keep calling the composed card, not restating it.
21-
*/
22-
const CARD_SHA256: Record<(typeof DIRECTOR_IDS)[number], string> = {
23-
dispatch: "0ce0493f3119a02b999b87b5e5c51d61df01ae8042923710404cae6ee9758328",
24-
explorer: "d4cac32fe0703953bf4d68c68b02aa67e3e50d2172b8137c5692ccdb1afc76eb",
25-
planner: "6ce04bb2349e9a67cfab538c1cbcb08c4eaa7dfe2eb40dca94bd022e4608bc23",
26-
coder: "6e872830d642935e98407b1e45a23740e73c731abfb3e6297a242201abd09092",
27-
reviewer: "9de0237970a17a030db9bc4c387d46d5d25788e91a806a7e9713b93778d75d5b",
28-
designer: "d021f346f451eda0770d6580b7647441311c89faef919e670ac6ab4ae9aa297f",
29-
artist: "8ba1a961af72fabbaa6310b322e931872aa759e6572dda327f48d9deefe12ecb",
30-
warden: "585add005436fea88c84b66d2f0aaa0bc4454c8165b6951989678226ebd0e27c",
31-
shakespeare:
32-
"e993db0e6bee1090aa64c1a7f8501e6c87c15e6eed26ce686b7efba7fcfd9541",
33-
prober: "a34f956da1245a4f370eeedeb71e09f4dfe3d6df009b0083fc5220a30790ac59",
34-
"qa-lead": "5beb75426761fdf952fdd889fb06c3152d90e2ead9358071d747169b10518c0e",
35-
};
36-
37-
function sha256(text: string): string {
38-
return createHash("sha256").update(text).digest("hex");
39-
}
40-
4115
describe("director cards compose from prompt-fragments", () => {
4216
test("registry system prompts are the composed cards", () => {
4317
for (const id of DIRECTOR_IDS) {
4418
expect(DIRECTOR_REGISTRY[id].systemPrompt).toBe(DIRECTOR_CARDS[id]);
4519
}
4620
});
4721

48-
test("composed output hash pin", () => {
49-
for (const id of DIRECTOR_IDS) {
50-
expect(sha256(DIRECTOR_CARDS[id])).toBe(CARD_SHA256[id]);
51-
}
52-
});
53-
5422
test("shared frames are the fragment helpers, not a second wording", () => {
5523
expect(DIRECTOR_CARDS.explorer.startsWith(identityLine("Explorer"))).toBe(
5624
true,
@@ -72,17 +40,4 @@ describe("director cards compose from prompt-fragments", () => {
7240
DIRECTOR_IDS.filter((id) => id !== "dispatch").sort(),
7341
);
7442
});
75-
76-
test("director modules do not embed a second copy of the card", () => {
77-
for (const id of DIRECTOR_IDS) {
78-
const path =
79-
id === "qa-lead"
80-
? new URL("./qa-lead/package.ts", import.meta.url)
81-
: new URL(`../../../agents/${id}/src/prompt.ts`, import.meta.url);
82-
const text = readFileSync(path, "utf8");
83-
expect(text).not.toContain("PRIMARY INTENT");
84-
expect(text).not.toContain("OUT OF LANE:");
85-
expect(text).not.toContain("# Route");
86-
}
87-
});
8843
});

‎src/agent/model-family-policy.test.ts‎

Lines changed: 0 additions & 137 deletions
Original file line numberDiff line numberDiff line change
@@ -190,143 +190,6 @@ describe("resolveModelFamilyPolicy", () => {
190190
expect(gpt.advertisedToolDeny).toEqual([]);
191191
});
192192

193-
describe("astra repro trace (CL-9027)", () => {
194-
// Minimal failing session-trace fixture: 8 consecutive tool-only turns
195-
// from a gpt-6-astra leaf. The shared guard fires only on exact repeats,
196-
// so a loop varying trivial argument details escapes it — evasion, not
197-
// waste — and the residual forbids exactly this variation. Test-only
198-
// forensics; no signature normalization ships in first-party code.
199-
interface ReproTurn {
200-
tool: string;
201-
args: Record<string, unknown>;
202-
resultTokens: number;
203-
}
204-
const ASTRA_REPRO_TRACE: readonly ReproTurn[] = [
205-
{
206-
tool: "read",
207-
args: { path: "src/agent/prompts.ts" },
208-
resultTokens: 3200,
209-
},
210-
{
211-
tool: "read",
212-
args: { path: "./src/agent/prompts.ts" },
213-
resultTokens: 3200,
214-
},
215-
{
216-
tool: "read",
217-
args: { path: "src/agent/prompts.ts", offset: 1 },
218-
resultTokens: 3180,
219-
},
220-
{
221-
tool: "grep",
222-
args: { pattern: "narrate", path: "src/agent" },
223-
resultTokens: 420,
224-
},
225-
{
226-
tool: "read",
227-
args: { path: "src/agent/prompts.ts" },
228-
resultTokens: 3200,
229-
},
230-
{
231-
tool: "grep",
232-
args: { pattern: "narrate ", path: "src/agent" },
233-
resultTokens: 420,
234-
},
235-
{
236-
tool: "read",
237-
args: { path: "./src/agent/prompts.ts", limit: 2000 },
238-
resultTokens: 3200,
239-
},
240-
{
241-
tool: "read",
242-
args: { path: "src/agent/prompts.ts", offset: 0 },
243-
resultTokens: 3200,
244-
},
245-
];
246-
247-
function stableJson(value: unknown): string {
248-
if (value === null || typeof value !== "object")
249-
return JSON.stringify(value);
250-
if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`;
251-
const obj = value as Record<string, unknown>;
252-
const keys = Object.keys(obj).sort();
253-
return `{${keys.map((k) => `${JSON.stringify(k)}:${stableJson(obj[k])}`).join(",")}}`;
254-
}
255-
256-
function fingerprint(turn: ReproTurn): string {
257-
return `${turn.tool}:${stableJson(turn.args)}`;
258-
}
259-
260-
function maxExactRepeats(fps: readonly string[]): number {
261-
let best = 1;
262-
for (let period = 1; period <= 6; period++) {
263-
for (let end = 1; end <= fps.length; end++) {
264-
const prefix = fps.slice(0, end);
265-
let i = prefix.length - 1;
266-
let j = i - period;
267-
let matched = 0;
268-
while (j >= 0 && prefix[i] === prefix[j]) {
269-
matched++;
270-
i--;
271-
j--;
272-
}
273-
const reps = Math.floor((matched + period) / period);
274-
if (reps > best) best = reps;
275-
}
276-
}
277-
return best;
278-
}
279-
280-
function evasionKey(turn: ReproTurn): string {
281-
const args = { ...turn.args };
282-
// The trivial deltas this trace varies: a leading ./ prefix, default
283-
// offset/limit values, and padding whitespace.
284-
if (typeof args.path === "string")
285-
args.path = args.path.replace(/^\.\//, "");
286-
if (args.offset === 0 || args.offset === 1) delete args.offset;
287-
if (typeof args.limit === "number") delete args.limit;
288-
if (typeof args.pattern === "string") args.pattern = args.pattern.trim();
289-
return `${turn.tool}:${stableJson(args)}`;
290-
}
291-
292-
function classifyAstraTrace(trace: readonly ReproTurn[]): string {
293-
if (maxExactRepeats(trace.map(fingerprint)) >= 3) return "waste";
294-
const groups = new Map<string, number>();
295-
for (const turn of trace) {
296-
const key = evasionKey(turn);
297-
groups.set(key, (groups.get(key) ?? 0) + 1);
298-
}
299-
return Math.max(...groups.values()) >= 3 ? "evasion" : "waste";
300-
}
301-
302-
test("classifies as evasion: no exact repeat trips the shared guard", () => {
303-
expect(ASTRA_REPRO_TRACE).toHaveLength(8);
304-
expect(maxExactRepeats(ASTRA_REPRO_TRACE.map(fingerprint))).toBeLessThan(
305-
3,
306-
);
307-
expect(classifyAstraTrace(ASTRA_REPRO_TRACE)).toBe("evasion");
308-
});
309-
310-
test("pins the offending pattern and the wasted turn/token delta", () => {
311-
const groups = new Map<string, number>();
312-
for (const turn of ASTRA_REPRO_TRACE) {
313-
const key = evasionKey(turn);
314-
groups.set(key, (groups.get(key) ?? 0) + 1);
315-
}
316-
// Six re-reads of one file plus two re-greps of one pattern, each run
317-
// differing only by a trivial argument delta.
318-
expect(groups.get('read:{"path":"src/agent/prompts.ts"}')).toBe(6);
319-
expect(groups.get('grep:{"path":"src/agent","pattern":"narrate"}')).toBe(
320-
2,
321-
);
322-
const wastedTokens = ASTRA_REPRO_TRACE.reduce(
323-
(sum, turn) => sum + turn.resultTokens,
324-
0,
325-
);
326-
expect(wastedTokens).toBe(20020);
327-
});
328-
});
329-
330193
test("astra resolves to astra with the composed residual; thresholds stay default (CL-9027)", () => {
331194
const base = resolveModelFamilyPolicy({
332195
providerName: "unknown-provider",

0 commit comments

Comments
 (0)