Skip to content

Commit a3a8271

Browse files
TheGreatAxiosTheGreatAxios
committed
ci(test): shard suite into four time-balanced slices
1 parent 7c41ba8 commit a3a8271

3 files changed

Lines changed: 596 additions & 22 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 71 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -78,22 +78,29 @@ jobs:
7878
run: bun run build
7979

8080
# The suite is sharded so the slowest slice, not the whole suite, sets the
81-
# wall clock. Every shard still goes through check:projects-dir-guard: the
82-
# guard forwards these path filters to the suite it wraps, and the union of
83-
# the shards' filters is exactly ./src ./tests ./evals ./scripts, so the gate covers
84-
# the same tests as before, all of them sandboxed.
81+
# wall clock. Four time-balanced shards split the same
82+
# ./src ./tests ./evals ./scripts union via bun's --shard=k/4, balanced by
83+
# the checked-in per-file durations in scripts/ci-timings.json (--timings).
84+
# Every shard still goes through check:projects-dir-guard: the guard
85+
# forwards the path union plus the shard flags to the suite it wraps, so the
86+
# gate covers the same tests as before, all of them sandboxed.
87+
#
88+
# Timings refresh policy: regenerate scripts/ci-timings.json by running the
89+
# full union locally with --update-timings (same seeded flags as `test`):
90+
# bun run check:projects-dir-guard ./src ./tests ./evals ./scripts \
91+
# --timings=./scripts/ci-timings.json --update-timings
92+
# Regen when the slowest shard's Test step skews more than ~20% above a
93+
# quarter of the one-process suite time (shards drifting apart means the
94+
# timings no longer describe the suite), or proactively whenever slow files
95+
# land. A scheduled refresh artifact is a future option, not current setup.
8596
test:
8697
runs-on: ubuntu-latest
8798
strategy:
88-
# A red shard must not cancel the other; both results are the signal.
99+
# A red shard must not cancel the others; all results are the signal.
89100
fail-fast: false
90101
matrix:
91-
shard:
92-
- name: src
93-
paths: ./src
94-
- name: tests-evals-and-scripts
95-
paths: ./tests ./evals ./scripts
96-
name: test (${{ matrix.shard.name }})
102+
shard: ["1/4", "2/4", "3/4", "4/4"]
103+
name: test (${{ matrix.shard }})
97104
steps:
98105
- name: Checkout
99106
uses: actions/checkout@v4
@@ -122,19 +129,63 @@ jobs:
122129
- name: Install dependencies
123130
run: bun install --frozen-lockfile
124131

125-
# The same script the local `bun run check` gate runs, with the shard's
126-
# path filters forwarded through the guard to the suite. The guard
127-
# routes a filtered run through test:paths, which carries the same
128-
# seeded flags as the `test` script; bun test filters are additive, so
129-
# appending filters to `bun run test` could not narrow it. Randomized
130-
# order catches tests that only pass in the default file order (shared
131-
# module-level state, an unrestored global mock, a leaked env var).
132+
# The same script the local `bun run check` gate runs, with the full
133+
# path union plus the shard's --shard/--timings flags forwarded through
134+
# the guard to the suite. The guard routes a filtered run through
135+
# test:paths, which carries the same seeded flags as the `test` script;
136+
# bun test filters are additive, so appending filters to `bun run test`
137+
# could not narrow it. --shard splits by file (balanced by --timings),
138+
# so every shard covers the same union and the four shards together
139+
# cover the whole suite. Randomized order catches tests that only pass
140+
# in the default file order (shared module-level state, an unrestored
141+
# global mock, a leaked env var).
132142
# The seed stays 424242 in every shard rather than varying per shard:
133143
# the shards already run disjoint file sets, and a fixed seed keeps
134144
# any failure reproducible locally with the same
135-
# `bun run test:paths <paths>`.
145+
# `bun run test:paths <paths> --shard=k/4
146+
# --timings=./scripts/ci-timings.json`.
136147
- name: Test
137-
run: bun run check:projects-dir-guard ${{ matrix.shard.paths }}
148+
run: bun run check:projects-dir-guard ./src ./tests ./evals ./scripts --shard=${{ matrix.shard }} --timings=./scripts/ci-timings.json
149+
150+
# Cross-shard pollution detector: the shards above split the path union,
151+
# but the union is not the isolation domain — a mock.module leak across
152+
# files fails in the one-process suite yet passes when the files land in
153+
# different shards (CL-6967). This job reruns the whole union in one
154+
# process with no filters, exactly like the local `bun run check` gate.
155+
# Non-blocking (continue-on-error) so a slow or flaky full run cannot hold
156+
# the gate; a real pollution failure still shows up red for triage.
157+
test-one-process:
158+
name: test (one-process pollution detector)
159+
continue-on-error: true
160+
runs-on: ubuntu-latest
161+
steps:
162+
- name: Checkout
163+
uses: actions/checkout@v4
164+
165+
- name: Setup Node
166+
uses: actions/setup-node@v4
167+
with:
168+
node-version: "24"
169+
170+
- name: Setup Bun
171+
uses: oven-sh/setup-bun@v2
172+
with:
173+
bun-version: "1.3.14"
174+
175+
- name: Install ripgrep
176+
run: sudo apt-get install -y ripgrep
177+
178+
- name: Cache dependencies
179+
uses: actions/cache@v4
180+
with:
181+
path: node_modules
182+
key: bun-${{ hashFiles('bun.lock') }}
183+
184+
- name: Install dependencies
185+
run: bun install --frozen-lockfile
186+
187+
- name: Test
188+
run: bun run check:projects-dir-guard
138189

139190
# protect-main still requires the pre-restructure check names. These jobs
140191
# exist only to publish those contexts after the real work succeeds.

‎CONTRIBUTING.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,15 @@ bun run build
3030
bun run test
3131
```
3232

33-
These match the local development loop. CI shards the same path union via
34-
`test:paths` rather than running the one-process `bun run test` suite.
33+
These match the local development loop. CI splits the same path union into
34+
four time-balanced `--shard=k/4` slices via `test:paths` (balanced by the
35+
checked-in per-file durations in `scripts/ci-timings.json`) rather than
36+
running the one-process `bun run test` suite. Regenerate that file with
37+
`bun run check:projects-dir-guard ./src ./tests ./evals ./scripts
38+
--timings=./scripts/ci-timings.json --update-timings` when the slowest
39+
shard skews more than ~20% above a quarter of the one-process suite time,
40+
or proactively whenever slow files land — see `.github/workflows/ci.yml`
41+
for the full policy.
3542
Run `bun run check`
3643
(lint, typecheck, build, and the guarded test suite) before opening a PR —
3744
`bun run test` alone skips the projects-dir sandbox guard, which only runs

0 commit comments

Comments
 (0)