Skip to content

Commit b1a6092

Browse files
committed
refactor(testkit): move shared test helpers to the repo root
1 parent ca88bcf commit b1a6092

199 files changed

Lines changed: 226 additions & 225 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -85,15 +85,15 @@ jobs:
8585

8686
# The suite is sharded so the slowest slice, not the whole suite, sets the
8787
# wall clock. Four time-balanced shards split the same
88-
# ./src ./e2e ./evals ./scripts union via bun's --shard=k/4, balanced by
88+
# ./src ./e2e ./evals ./scripts ./testkit union via bun's --shard=k/4, balanced by
8989
# the checked-in per-file durations in scripts/ci-timings.json (--timings).
9090
# Every shard still goes through check:projects-dir-guard: the guard
9191
# forwards the path union plus the shard flags to the suite it wraps, so the
9292
# gate covers the same tests as before, all of them sandboxed.
9393
#
9494
# Timings refresh policy: regenerate scripts/ci-timings.json by running the
9595
# full union locally with --update-timings (same seeded flags as `test`):
96-
# bun run check:projects-dir-guard ./src ./e2e ./evals ./scripts \
96+
# bun run check:projects-dir-guard ./src ./e2e ./evals ./scripts ./testkit \
9797
# --timings=./scripts/ci-timings.json --update-timings
9898
# Regen when the slowest shard's Test step skews more than ~20% above a
9999
# quarter of the one-process suite time (shards drifting apart means the
@@ -151,7 +151,7 @@ jobs:
151151
# `bun run test:paths <paths> --shard=k/4
152152
# --timings=./scripts/ci-timings.json`.
153153
- name: Test
154-
run: bun run check:projects-dir-guard ./src ./e2e ./evals ./scripts --shard=${{ matrix.shard }} --timings=./scripts/ci-timings.json
154+
run: bun run check:projects-dir-guard ./src ./e2e ./evals ./scripts ./testkit --shard=${{ matrix.shard }} --timings=./scripts/ci-timings.json
155155

156156
# Cross-shard pollution detector: the shards above split the path union,
157157
# but the union is not the isolation domain — a mock.module leak across

‎AGENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,11 +32,11 @@ When refactoring replaces an old path, delete the old one. No back-compat shims,
3232

3333
- Add or update tests with every behavior change.
3434
- Bug fixes start with a failing test that reproduces the bug. Do not start by patching.
35-
- Co-located `src/**/*.test.ts` for module logic · `src/testkit/` shared test helpers · `fixtures/` fixture repos · `e2e/` scenario runs over the production agent loop (fixture-repo seeding, scripted inference via `@intx/inference-testing`, `runUntilDone`/`runUntilSuspended`/`sendOperatorTurn` drivers — see `e2e/harness.ts`; the session plumbing lives in `e2e/integration-harness.ts`).
35+
- Co-located `src/**/*.test.ts` for module logic · `testkit/` shared test helpers (repo root, shared by unit and e2e) · `fixtures/` fixture repos · `e2e/` scenario runs over the production agent loop (fixture-repo seeding, scripted inference via `@intx/inference-testing`, `runUntilDone`/`runUntilSuspended`/`sendOperatorTurn` drivers — see `e2e/harness.ts`; the session plumbing lives in `e2e/integration-harness.ts`).
3636
- Unit vs e2e split: keep parser tables, race/atomicity tests, and small pure-function contracts co-located unit tests; put multi-step agent-loop orchestration (permission flows, compaction end-to-end, subagent lanes, credential recovery) in `e2e/` where a scripted model reply replaces pages of per-file fakes. Do not pin constants or `record.decisions` diagnostics — assert the behavior the contract exposes.
3737
- e2e v1 non-goals: TUI overlay/PTY driving, race tests, parser tables, crash-atomicity, and any second agent-loop stack — the integration harness already is one.
3838
- A test must not depend on another file having run, or on the default file order. It must pass under `bun test ./src ./e2e ./evals ./scripts --randomize`. If a test mutates module-level state or calls `mock.module`, it must restore that state itself (`afterEach`/`afterAll`), not rely on the process happening to reset it. When capturing a module's real exports to restore later, shallow-copy them (`{ ...moduleNamespace }`) at capture time, whether the namespace came from `await import(path)` or a static `import * as ns from "path"` — Bun mutates the live namespace object in place when the module is mocked, so holding a bare reference to it (either form) silently turns into the mocked exports.
39-
- Never call `mock.module` directly. Bun runs every test file in one process, so a `mock.module` call without its own teardown stays installed for the rest of the run and silently replaces the real module for other files — producing failures in files the change never touched, with no obvious link to the cause and no signal from `tsc` or a per-file run (CL-6967). Use `withMockedModule`/`withMockedModuleDuring` from `src/testkit/mock-module.ts`, which capture the real module and register their own restore. The oxlint plugin (`corbits/no-bare-mock-module` in `.oxlintrc.json` / `scripts/oxlint-plugin-corbits.js`) rejects bare `mock.module` calls in `*.test.ts` files.
39+
- Never call `mock.module` directly. Bun runs every test file in one process, so a `mock.module` call without its own teardown stays installed for the rest of the run and silently replaces the real module for other files — producing failures in files the change never touched, with no obvious link to the cause and no signal from `tsc` or a per-file run (CL-6967). Use `withMockedModule`/`withMockedModuleDuring` from `testkit/mock-module.ts`, which capture the real module and register their own restore. The oxlint plugin (`corbits/no-bare-mock-module` in `.oxlintrc.json` / `scripts/oxlint-plugin-corbits.js`) rejects bare `mock.module` calls in `*.test.ts` files.
4040
- A test earns its place only if a real behavior change can fail it. Document copy, brand colors, marketing assets, and splash text are not behavior: assertions that pin an asset's literal wording, an exact palette hex/ANSI value, or rendered copy fail on copy/design edits and catch no regressions — assert the contract instead (parsing, formatting, ranges, aliases, invariants). Tests are code too: pinning a source file's own text is the same trap. This bar is a review and authorship rule, not a linter shape match.
4141

4242
## Build & Validation

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ These match the local development loop. CI splits the same path union into
3434
four time-balanced `--shard=k/4` slices via `test:paths` (balanced by the
3535
checked-in per-file durations in `scripts/ci-timings.json`) rather than
3636
running the one-process `bun run test` suite. Regenerate that file with
37-
`bun run check:projects-dir-guard ./src ./e2e ./evals ./scripts
37+
`bun run check:projects-dir-guard ./src ./e2e ./evals ./scripts ./testkit
3838
--timings=./scripts/ci-timings.json --update-timings` when the slowest
3939
shard skews more than ~20% above a quarter of the one-process suite time,
4040
or proactively whenever slow files land — see `.github/workflows/ci.yml`

‎bunfig.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[test]
2-
preload = ["./src/testkit/preload.ts"]
2+
preload = ["./testkit/preload.ts"]
33
pathIgnorePatterns = [
44
"fixtures/**",
55
"eval/tasks/**",

‎docs/IMPLEMENTATION.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -560,11 +560,11 @@ Run all three before declaring work complete.
560560

561561
## Testing
562562

563-
- **Unit tests** are co-located with source as `*.test.ts` (e.g. `config.test.ts`, `director.test.ts`, `prompts.test.ts`, `renderer.test.ts`, `permission/permission.test.ts`, each `plugins/*.test.ts`, and TUI tests under `tui/`). Shared test helpers live in `src/testkit/`.
563+
- **Unit tests** are co-located with source as `*.test.ts` (e.g. `config.test.ts`, `director.test.ts`, `prompts.test.ts`, `renderer.test.ts`, `permission/permission.test.ts`, each `plugins/*.test.ts`, and TUI tests under `tui/`). Shared test helpers live in `testkit/` (repo root, shared by unit and e2e).
564564
- **`fixtures/`** holds fixture repos and comparison assets (e.g. `demo-comparison/`, `multi-file-service/`).
565565
- **`e2e/`** holds scenario tests driven over the production agent loop with scripted models (`@intx/inference-testing`) — fixture-repo seeding, `runUntilDone`/`runUntilSuspended`/`sendOperatorTurn` drivers (see `e2e/harness.ts`), and the reactor permission / multi-turn exercises formerly under `tests/integration/`.
566566
- **Capability evals** (`evals/capability/`) are **not** the integration harness: they drive the product path (`corbits exec` / `runExec`) with real models against fixture copies and objective `verify.sh` graders. Case format + loader tests live under `evals/capability/`; run with `bun run eval:capability` (see `evals/capability/README.md`). Use `--baseline` to detect improve/regress across models or commits.
567-
- **TUI tests** are co-located `*.test.ts` files under `src/tui/` (e.g. `shell.test.ts`, `runner-host.test.ts`, `stream.test.ts`), run as part of `bun test` along with everything else; there is no separate `test:tui` script. `bunfig.toml` preloads `src/testkit/preload.ts` for every test file, which strips ambient env (`COLORTERM`, `CORBITS_*`, telemetry) and hard-fails when `rg` is absent.
567+
- **TUI tests** are co-located `*.test.ts` files under `src/tui/` (e.g. `shell.test.ts`, `runner-host.test.ts`, `stream.test.ts`), run as part of `bun test` along with everything else; there is no separate `test:tui` script. `bunfig.toml` preloads `testkit/preload.ts` for every test file, which strips ambient env (`COLORTERM`, `CORBITS_*`, telemetry) and hard-fails when `rg` is absent.
568568
- **Parallel local runs** — `bun run test:parallel [N]` (default 4 workers) runs the same seeded suite with `--parallel=N`, wrapped in an output-stall watchdog (`scripts/test-parallel.ts`). Bun 1.4.x intermittently livelocks under `--parallel` (one worker spins at 100 % CPU holding a zombie git child while the main process idles; no output, no summary — upstream oven-sh/bun#36235), and `bun test` has no run-level timeout, so a stalled run hangs forever. The watchdog kills the suite's own process group after 90 s of silence and retries up to 3 times; a child that exits on its own (pass or fail) is never retried. CI keeps sharded sequential runs (`test:paths`) and does not use `--parallel`.
569569

570570
## Deployment

‎e2e/crash-finalize.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ import { describe, expect, test } from "bun:test";
55

66
import { generateSessionId, sessionDir } from "../src/session/index.js";
77
import type { RunState } from "../src/session/state.js";
8-
import { createTempDirs } from "../src/testkit/temporary-dirs.js";
8+
import { createTempDirs } from "../testkit/temporary-dirs.js";
99

1010
const FIXTURE = join(
1111
import.meta.dirname,

‎e2e/git-push-scoped.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ import { tmpdir } from "node:os";
1818
import { join } from "node:path";
1919

2020
import { afterEach, beforeEach, describe, expect, it } from "bun:test";
21-
import { initTemporaryGitRepo } from "../src/testkit/temporary-git-repo.js";
21+
import { initTemporaryGitRepo } from "../testkit/temporary-git-repo.js";
2222

2323
const SCRIPT = join(import.meta.dir, "../bin/git-push-scoped");
2424

‎e2e/harness.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ import {
2222
createPermissionGate,
2323
type PermissionGate,
2424
} from "../src/permission/gate.js";
25-
import { initTemporaryGitRepo } from "../src/testkit/temporary-git-repo.js";
25+
import { initTemporaryGitRepo } from "../testkit/temporary-git-repo.js";
2626
import {
2727
closeIntegrationSession,
2828
openIntegrationSession,

‎e2e/reactor-approval-suspend.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ import {
1515
runUntilSuspended,
1616
toolDoneEvents,
1717
} from "./integration-harness.js";
18-
import { defined } from "../src/testkit/defined.js";
18+
import { defined } from "../testkit/defined.js";
1919

2020
const CURL_CALL = {
2121
name: "run_shell",

‎e2e/signal-helpers.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { createTempDirs } from "../src/testkit/temporary-dirs.js";
1+
import { createTempDirs } from "../testkit/temporary-dirs.js";
22

33
/**
44
* Reads a spawned fixture's stdout pipe until the first chunk containing

0 commit comments

Comments
 (0)