test: add scripted-inference e2e layer and slim the unit suite (#1221)

* fix(session): classify persisted tool calls by engine name

Persisted tool_call blocks keep the wire name the model emitted
(read, glob, bash), but the compactor's replay/supersession index
and the subagent thrash tracker classify against engine-keyed sets
(read_file, search_files). After the one-wire migration that left
read supersession and thrash bookkeeping dead for the advertised
tools; route stored names through canonicalToolName before
classification.

* test(e2e): add scenario harness over the integration session

The e2e layer wraps the existing agent-loop harness (live tool
dispatch, chat director, posix tools, git-backed context store)
with fixture-repo seeding and a sendOperatorTurn driver that
surfaces terminal failures instead of asserting a reply. The
integration session gains the seams scenarios need: fleet
passthroughs, retry-policy and scheduler overrides, alternate
inference sources plus credential records, and access to the live
chat director. The toolset forwards the fleet's documented
retry-timing overrides to spawned runs.

* test(e2e): migrate orchestration suites and prune weak pins

Three orchestration scenarios now run through the real agent loop
on scripted inference: compaction read-paging (the wire-name
coverage that exposed the engine-name classification bug),
recoverable subagent failure (failed lane reports continuable
through wait_agents), and credential recovery (oauth refresh,
retry, terminal 401, accepted backup serves the replayed prompt).
The bespoke unit versions are deleted where the e2e covers the
same contract; constant-equal-literal tool-alias pins and a
decisions-diagnostics-only tail test are dropped for asserting
the same thing behavior already proves. AGENTS.md documents the
split and the v1 non-goals.

* test(subagent): pin wire-name classification in thrash tracking

The engine-name fix for persisted calls had no fixture proving
wire names reach the same evidence keys — regression coverage for
the read/search/mutation path that silently went dark after the
one-wire migration.

* test(e2e): harden collector teardown and per-test budgets

A stream error in the background collector would surface as an
unhandled rejection in the shared test process; the catch mirrors
runUntilSuspended. Explicit timeouts replace the 5s bun:test
default so a loaded CI worker fails a genuinely stuck scenario
rather than flaking at the default.

* refactor: drop write-only telemetry and consumer-less exports

* test: prune weak pins and collapse duplicate scaffolding

Cuts ~10.6k lines of co-located unit tests (~7%) plus ~1.4k of
assertion-level copy/literal/diagnostic pins, and folds ~3.4k lines of
copy-pasted arrange scaffolding into file-local helpers with zero
assertion loss. Moves the ask_director handshake and outer-retry
recovery scenarios into tests/e2e driven through the real agent loop.

* test: co-locate unit tests and lift e2e to the repo root

* test: prune low-value suites and extract shared fixtures

* test: extract shared harnesses and collapse remaining clone groups

* test: table-drive repeated cases and collapse module scaffolding

* test(permission): collapse clustered-option classification to one wiring smoke

* refactor(tui): drop dead index-based operatorResultFromSelection

* test(tui): drop wave6 palette re-runs covered by palette-paint and slash-popup-gate

* test(tui): drop sequential-operator paint re-run covered by gate-wire

* test(director): consolidate the split director suites into the co-located file

* test(director): drop the two recovery cases the merged file already pins

* test: drop a dead status assertion and prose-copy pins the flags already pin

* test(tui): table-drive the immediate-remove plugin origins

* test(e2e): table-drive the tool_search promotion schema matrix

* test(config): table-drive the skip-permissions precedence matrix

* test(session): drop the straggler-write re-run; state.test pins the fence

* test: table-drive exec status truth table and share the Codex limit body

* style: format touched test files

* test: replace personal identifiers with generic acme/alice examples

* docs: use generic acme/alice examples in provider and path comments

* test(agent): share one connectMCPServer mock kit across the two mcp fixture files

* test(mcp): extract the shared SDK mock kit and collapse the auth episode loops

* test(session): merge the duplicated approval-resume fixtures into one harness

* test(compaction): table-drive the handoff-arming and verify-pass cases

* test(subagent): share envelope fixtures and collapse repeated scaffolds

* test(config): extract shared settings-write and loadFor fixture helpers

* style: format worker-touched test files and drop an unused mock parameter

* refactor(testkit): move shared test helpers to the repo root

* test(e2e): pin the recoverable-failure successor round trip

Author: TheGreatAxios

.github/workflows/ci.yml

@@ -85,15 +85,15 @@ jobs:
 
   # The suite is sharded so the slowest slice, not the whole suite, sets the
   # wall clock. Four time-balanced shards split the same
-  # ./src ./tests ./evals ./scripts union via bun's --shard=k/4, balanced by
+  # ./src ./e2e ./evals ./scripts ./testkit union via bun's --shard=k/4, balanced by
   # the checked-in per-file durations in scripts/ci-timings.json (--timings).
   # Every shard still goes through check:projects-dir-guard: the guard
   # forwards the path union plus the shard flags to the suite it wraps, so the
   # gate covers the same tests as before, all of them sandboxed.
   #
   # Timings refresh policy: regenerate scripts/ci-timings.json by running the
   # full union locally with --update-timings (same seeded flags as `test`):
-  #   bun run check:projects-dir-guard ./src ./tests ./evals ./scripts \
+  #   bun run check:projects-dir-guard ./src ./e2e ./evals ./scripts ./testkit \
   #     --timings=./scripts/ci-timings.json --update-timings
   # Regen when the slowest shard's Test step skews more than ~20% above a
   # quarter of the one-process suite time (shards drifting apart means the
@@ -151,7 +151,7 @@ jobs:
       # `bun run test:paths <paths> --shard=k/4
       # --timings=./scripts/ci-timings.json`.
       - name: Test
-        run: bun run check:projects-dir-guard ./src ./tests ./evals ./scripts --shard=${{ matrix.shard }} --timings=./scripts/ci-timings.json
+        run: bun run check:projects-dir-guard ./src ./e2e ./evals ./scripts ./testkit --shard=${{ matrix.shard }} --timings=./scripts/ci-timings.json
 
   # Cross-shard pollution detector: the shards above split the path union,
   # but the union is not the isolation domain — a mock.module leak across

AGENTS.md

@@ -32,9 +32,11 @@ When refactoring replaces an old path, delete the old one. No back-compat shims,
 
 - Add or update tests with every behavior change.
 - Bug fixes start with a failing test that reproduces the bug. Do not start by patching.
-- `tests/unit/` shared unit tests and helpers · co-located `src/**/*.test.ts` for module logic · `tests/fixtures/` fixture repos · `tests/integration/` reactor/permission harness. Planned: `tests/e2e/` (fixture-repo runs).
-- A test must not depend on another file having run, or on the default file order. It must pass under `bun test ./src ./tests ./evals ./scripts --randomize`. If a test mutates module-level state or calls `mock.module`, it must restore that state itself (`afterEach`/`afterAll`), not rely on the process happening to reset it. When capturing a module's real exports to restore later, shallow-copy them (`{ ...moduleNamespace }`) at capture time, whether the namespace came from `await import(path)` or a static `import * as ns from "path"` — Bun mutates the live namespace object in place when the module is mocked, so holding a bare reference to it (either form) silently turns into the mocked exports.
-- Never call `mock.module` directly. Bun runs every test file in one process, so a `mock.module` call without its own teardown stays installed for the rest of the run and silently replaces the real module for other files — producing failures in files the change never touched, with no obvious link to the cause and no signal from `tsc` or a per-file run (CL-6967). Use `withMockedModule`/`withMockedModuleDuring` from `tests/helpers/mock-module.ts`, which capture the real module and register their own restore. The oxlint plugin (`corbits/no-bare-mock-module` in `.oxlintrc.json` / `scripts/oxlint-plugin-corbits.js`) rejects bare `mock.module` calls in `*.test.ts` files.
+- Co-located `src/**/*.test.ts` for module logic · `testkit/` shared test helpers (repo root, shared by unit and e2e) · `fixtures/` fixture repos · `e2e/` scenario runs over the production agent loop (fixture-repo seeding, scripted inference via `@intx/inference-testing`, `runUntilDone`/`runUntilSuspended`/`sendOperatorTurn` drivers — see `e2e/harness.ts`; the session plumbing lives in `e2e/integration-harness.ts`).
+- Unit vs e2e split: keep parser tables, race/atomicity tests, and small pure-function contracts co-located unit tests; put multi-step agent-loop orchestration (permission flows, compaction end-to-end, subagent lanes, credential recovery) in `e2e/` where a scripted model reply replaces pages of per-file fakes. Do not pin constants or `record.decisions` diagnostics — assert the behavior the contract exposes.
+- e2e v1 non-goals: TUI overlay/PTY driving, race tests, parser tables, crash-atomicity, and any second agent-loop stack — the integration harness already is one.
+- A test must not depend on another file having run, or on the default file order. It must pass under `bun test ./src ./e2e ./evals ./scripts --randomize`. If a test mutates module-level state or calls `mock.module`, it must restore that state itself (`afterEach`/`afterAll`), not rely on the process happening to reset it. When capturing a module's real exports to restore later, shallow-copy them (`{ ...moduleNamespace }`) at capture time, whether the namespace came from `await import(path)` or a static `import * as ns from "path"` — Bun mutates the live namespace object in place when the module is mocked, so holding a bare reference to it (either form) silently turns into the mocked exports.
+- Never call `mock.module` directly. Bun runs every test file in one process, so a `mock.module` call without its own teardown stays installed for the rest of the run and silently replaces the real module for other files — producing failures in files the change never touched, with no obvious link to the cause and no signal from `tsc` or a per-file run (CL-6967). Use `withMockedModule`/`withMockedModuleDuring` from `testkit/mock-module.ts`, which capture the real module and register their own restore. The oxlint plugin (`corbits/no-bare-mock-module` in `.oxlintrc.json` / `scripts/oxlint-plugin-corbits.js`) rejects bare `mock.module` calls in `*.test.ts` files.
 - A test earns its place only if a real behavior change can fail it. Document copy, brand colors, marketing assets, and splash text are not behavior: assertions that pin an asset's literal wording, an exact palette hex/ANSI value, or rendered copy fail on copy/design edits and catch no regressions — assert the contract instead (parsing, formatting, ranges, aliases, invariants). Tests are code too: pinning a source file's own text is the same trap. This bar is a review and authorship rule, not a linter shape match.
 
 ## Build & Validation
@@ -49,10 +51,10 @@ the projects-dir sandbox guard — in that order, matching CI.
 
 Run the full suite before declaring any task complete. Do not substitute individual targets. If a failure is pre-existing and unrelated to your change, say so explicitly.
 
-`bun run test` runs `bun test ./src ./tests ./evals ./scripts --randomize --seed 424242`
+`bun run test` runs `bun test ./src ./e2e ./evals ./scripts --randomize --seed 424242`
 as a single process. CI shards the same path union via `test:paths`
 (`.github/workflows/ci.yml`) for wall clock. Path-union is not the same
-isolation domain: a `mock.module` leak across `./src` vs `./tests` fails
+isolation domain: a `mock.module` leak across `./src` vs `./e2e` fails
 locally in the one-process suite but not in a CI shard (CL-6967). A bare
 `bun test` also scans `vendor/`, adding hundreds of unrelated results and
 making pass/fail counts meaningless to compare across branches — always use

CONTRIBUTING.md

@@ -34,7 +34,7 @@ These match the local development loop. CI splits the same path union into
 four time-balanced `--shard=k/4` slices via `test:paths` (balanced by the
 checked-in per-file durations in `scripts/ci-timings.json`) rather than
 running the one-process `bun run test` suite. Regenerate that file with
-`bun run check:projects-dir-guard ./src ./tests ./evals ./scripts
+`bun run check:projects-dir-guard ./src ./e2e ./evals ./scripts ./testkit
 --timings=./scripts/ci-timings.json --update-timings` when the slowest
 shard skews more than ~20% above a quarter of the one-process suite time,
 or proactively whenever slow files land — see `.github/workflows/ci.yml`

bunfig.toml

@@ -1,7 +1,7 @@
 [test]
-preload = ["./tests/preload.ts"]
+preload = ["./testkit/preload.ts"]
 pathIgnorePatterns = [
-  "tests/fixtures/**",
+  "fixtures/**",
   "eval/tasks/**",
   "tmp/**",
   # Bundles vendor/intx-storage-isogit's browser entry point with Bun.build,

docs/IMPLEMENTATION.md

@@ -74,7 +74,7 @@ src/
       registry.ts         DIRECTOR_REGISTRY, resolveDirector, packageToProfile
       tool-sets.ts        Shared allowlists (READ/IMPLEMENT/DOCS/REVIEW/…)
       <id>/package.ts     Per-director prompt, envelope, spawn, report
-    renderer.ts           Event-stream renderer (stderr + live cost; used by tests/utilities)
+    renderer.ts           Event-stream renderer (stderr + live cost; used by tests and utilities)
   session/
     index.ts              Session lifecycle
     state.ts              RunState JSON save/load
@@ -523,7 +523,7 @@ See `docs/PLUGINS.md` for the full design. Summary:
 
 ## Hardening wave — deferred and upstream-owned items
 
-Corbits Code v0.3 memory and stall hardening is implemented under `src/`, `tests/`, and `scripts/` only. The `vendor/` tree is out of scope for that wave (`scripts/verify-corbits-only-scope.sh` enforces this on landing branches). The items below were **not** closed in Corbits Code because they do not apply to the default CLI/TUI path or require upstream Interchange packages.
+Corbits Code v0.3 memory and stall hardening is implemented under `src/`, `e2e/`, and `scripts/` only. The `vendor/` tree is out of scope for that wave (`scripts/verify-corbits-only-scope.sh` enforces this on landing branches). The items below were **not** closed in Corbits Code because they do not apply to the default CLI/TUI path or require upstream Interchange packages.
 
 ### Child-supervisor IPC awaiter deadlines
 
@@ -560,12 +560,11 @@ Run all three before declaring work complete.
 
 ## Testing
 
-- **Unit tests** are co-located with source as `*.test.ts` (e.g. `config.test.ts`, `director.test.ts`, `prompts.test.ts`, `renderer.test.ts`, `permission/permission.test.ts`, each `plugins/*.test.ts`, and TUI tests under `tui/`).
-- **`tests/unit/`** holds shared unit helpers and focused packages (e.g. TUI geometry tests).
-- **`tests/fixtures/`** holds fixture repos and comparison assets (e.g. `demo-comparison/`, `multi-file-service/`).
-- **`tests/integration/`** holds the reactor permission / multi-turn harness (scripted models via `@intx/inference-testing`). **`tests/e2e/`** (fixture-repo runs) is still planned. Until e2e exists, broader harness coverage also lives in co-located `*.test.ts` files and `tests/unit/`.
+- **Unit tests** are co-located with source as `*.test.ts` (e.g. `config.test.ts`, `director.test.ts`, `prompts.test.ts`, `renderer.test.ts`, `permission/permission.test.ts`, each `plugins/*.test.ts`, and TUI tests under `tui/`). Shared test helpers live in `testkit/` (repo root, shared by unit and e2e).
+- **`fixtures/`** holds fixture repos and comparison assets (e.g. `demo-comparison/`, `multi-file-service/`).
+- **`e2e/`** holds scenario tests driven over the production agent loop with scripted models (`@intx/inference-testing`) — fixture-repo seeding, `runUntilDone`/`runUntilSuspended`/`sendOperatorTurn` drivers (see `e2e/harness.ts`), and the reactor permission / multi-turn exercises formerly under `tests/integration/`.
 - **Capability evals** (`evals/capability/`) are **not** the integration harness: they drive the product path (`corbits exec` / `runExec`) with real models against fixture copies and objective `verify.sh` graders. Case format + loader tests live under `evals/capability/`; run with `bun run eval:capability` (see `evals/capability/README.md`). Use `--baseline` to detect improve/regress across models or commits.
-- **TUI tests** are co-located `*.test.ts` files under `src/tui/` (e.g. `shell.test.ts`, `runner-host.test.ts`, `stream.test.ts`), run as part of `bun test` along with everything else; there is no separate `test:tui` script or test-setup preload.
+- **TUI tests** are co-located `*.test.ts` files under `src/tui/` (e.g. `shell.test.ts`, `runner-host.test.ts`, `stream.test.ts`), run as part of `bun test` along with everything else; there is no separate `test:tui` script. `bunfig.toml` preloads `testkit/preload.ts` for every test file, which strips ambient env (`COLORTERM`, `CORBITS_*`, telemetry) and hard-fails when `rg` is absent.
 - **Parallel local runs** — `bun run test:parallel [N]` (default 4 workers) runs the same seeded suite with `--parallel=N`, wrapped in an output-stall watchdog (`scripts/test-parallel.ts`). Bun 1.4.x intermittently livelocks under `--parallel` (one worker spins at 100 % CPU holding a zombie git child while the main process idles; no output, no summary — upstream oven-sh/bun#36235), and `bun test` has no run-level timeout, so a stalled run hangs forever. The watchdog kills the suite's own process group after 90 s of silence and retries up to 3 times; a child that exits on its own (pass or fail) is never retried. CI keeps sharded sequential runs (`test:paths`) and does not use `--parallel`.
 
 ## Deployment

docs/TELEMETRY.md

@@ -112,7 +112,7 @@ part of the provider's rejection message is sent.
 
 The mapping is `src/telemetry/classify.ts`, and the tests that feed each
 emission site a deliberately identifying name and assert it reaches no part of
-the payload are in `tests/unit/telemetry-product-events.test.ts`.
+the payload are in `src/telemetry/product-events.test.ts`.
 
 ## AI observability events
 

docs/VENDORING.md

@@ -289,7 +289,7 @@ so `grep -rn "Locally patched" vendor/*/src` finds every divergence.
 **Markers are navigation; the SHA-diff is proof.** Run
 `bin/vendor-patch-diff` against a pristine upstream checkout at the
 recorded SHA to print exactly the lines that are ours. A correspondence
-test (`tests/unit/vendor-patch-ledger.test.ts`) fails if a marker anchor
+test (`scripts/vendor-patch-ledger.test.ts`) fails if a marker anchor
 does not resolve to a ledger heading, or if a ledger heading has no marker.
 
 ## Re-syncing a vendored package to a newer upstream commit
@@ -320,7 +320,7 @@ does not resolve to a ledger heading, or if a ledger heading has no marker.
    `Locally patched` markers to reflect what actually landed, including
    any patches dropped as superseded and why. Run the full gate
    (`typecheck`/`build`/`test`, including
-   `tests/unit/vendor-patch-ledger.test.ts`) and do not consider the sync
+   `scripts/vendor-patch-ledger.test.ts`) and do not consider the sync
    complete until it passes clean.
 4. Because `@intx/inference`, `@intx/types`, and `@intx/storage-isogit` are
    coupled (see above), a re-sync that moves any one of their commit hashes

…integration/compaction-atomicity.test.ts e2e/compaction-atomicity.test.tstests/integration/compaction-atomicity.test.ts renamed to e2e/compaction-atomicity.test.ts tests/integration/compaction-atomicity.test.ts renamed to e2e/compaction-atomicity.test.ts

@@ -7,11 +7,11 @@ import type {
   ConversationTurn,
   StrategyContext,
 } from "@intx/types/runtime";
-import { createOptimizedContextStore } from "../../src/session/optimized-context-store.js";
+import { createOptimizedContextStore } from "../src/session/optimized-context-store.js";
 import {
   createCompactionArchive,
   wrapCompactorWithCompletenessGate,
-} from "../../src/session/compaction-archive.js";
+} from "../src/session/compaction-archive.js";
 
 function tempDir(): string {
   return fs.mkdtempSync(path.join(os.tmpdir(), "compact-atomic-"));
@@ -25,6 +25,54 @@ function texts(turns: ConversationTurn[]): string[] {
   return turns.map((t) => (t.content[0] as { text: string }).text);
 }
 
+// Map-backed blob store so the archive can be exercised without a real repo.
+function mapArchive(dir: string) {
+  const blobs = new Map<string, Uint8Array>();
+  return createCompactionArchive({
+    sessionId: "primary",
+    contextDir: dir,
+    writeBlob: async (key, bytes) => {
+      blobs.set(key, bytes);
+    },
+    readBlob: async (key) => {
+      const hit = blobs.get(key);
+      if (hit === undefined) throw new Error(`missing ${key}`);
+      return hit;
+    },
+  });
+}
+
+// A turn sequence carrying one tool_call/tool_result pair — the shape the
+// completeness gate checks for evidence coverage.
+function toolExchangeHistory(): ConversationTurn[] {
+  return [
+    turn("fact-a"),
+    {
+      role: "assistant" as const,
+      content: [
+        {
+          type: "tool_call" as const,
+          id: "c1",
+          name: "read_file",
+          arguments: { path: "x" },
+        },
+      ],
+      timestamp: 2,
+    },
+    {
+      role: "user" as const,
+      content: [
+        {
+          type: "tool_result" as const,
+          callId: "c1",
+          content: [{ type: "text" as const, text: "body" }],
+        },
+      ],
+      timestamp: 3,
+    },
+  ];
+}
+
 const EMPTY_META = {
   pendingOperations: [],
   tokenUsage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, thinking: 0 },
@@ -81,49 +129,12 @@ describe("compaction atomicity", () => {
 
   test("primary incomplete certifyRange refuses destructive compact", async () => {
     const dir = tempDir();
-    const blobs = new Map<string, Uint8Array>();
-    const archive = createCompactionArchive({
-      sessionId: "primary",
-      contextDir: dir,
-      writeBlob: async (key, bytes) => {
-        blobs.set(key, bytes);
-      },
-      readBlob: async (key) => {
-        const hit = blobs.get(key);
-        if (hit === undefined) throw new Error(`missing ${key}`);
-        return hit;
-      },
-    });
+    const archive = mapArchive(dir);
     const wrapped = wrapCompactorWithCompletenessGate(
       truncatingCompactor(),
       archive,
     );
-    const history = [
-      turn("fact-a"),
-      {
-        role: "assistant" as const,
-        content: [
-          {
-            type: "tool_call" as const,
-            id: "c1",
-            name: "read_file",
-            arguments: { path: "x" },
-          },
-        ],
-        timestamp: 2,
-      },
-      {
-        role: "user" as const,
-        content: [
-          {
-            type: "tool_result" as const,
-            callId: "c1",
-            content: [{ type: "text" as const, text: "body" }],
-          },
-        ],
-        timestamp: 3,
-      },
-    ];
+    const history = toolExchangeHistory();
     const result = await wrapped.apply(history, ctx);
     expect(result.output).toBe(history);
     expect(result.blobs).toBeUndefined();
@@ -132,19 +143,7 @@ describe("compaction atomicity", () => {
 
   test("adopted handoff is recorded so the next fold can drop the spine", async () => {
     const dir = tempDir();
-    const blobs = new Map<string, Uint8Array>();
-    const archive = createCompactionArchive({
-      sessionId: "primary",
-      contextDir: dir,
-      writeBlob: async (key, bytes) => {
-        blobs.set(key, bytes);
-      },
-      readBlob: async (key) => {
-        const hit = blobs.get(key);
-        if (hit === undefined) throw new Error(`missing ${key}`);
-        return hit;
-      },
-    });
+    const archive = mapArchive(dir);
     const foldingCompactor = (
       spine: string,
       keep: ConversationTurn[],
@@ -213,32 +212,7 @@ describe("compaction atomicity", () => {
   test("primary complete rewrite publishes turns and evidence together", async () => {
     const dir = tempDir();
     const store = await createOptimizedContextStore(dir);
-    const history = [
-      turn("fact-a"),
-      {
-        role: "assistant" as const,
-        content: [
-          {
-            type: "tool_call" as const,
-            id: "c1",
-            name: "read_file",
-            arguments: { path: "x" },
-          },
-        ],
-        timestamp: 2,
-      },
-      {
-        role: "user" as const,
-        content: [
-          {
-            type: "tool_result" as const,
-            callId: "c1",
-            content: [{ type: "text" as const, text: "body" }],
-          },
-        ],
-        timestamp: 3,
-      },
-    ];
+    const history = toolExchangeHistory();
     await store.writeTurns(history);
     await store.writeMetadata(EMPTY_META);
     const oldCommit = await store.commit({ message: "primary-old" });