From 50b38a5ba433733884bb64f6fe20dd74d20028c7 Mon Sep 17 00:00:00 2001 From: JUN Date: Thu, 17 Sep 2026 10:46:24 +0900 Subject: [PATCH 1/3] ci(windows): restore the margin the six-shard leg lost, and make a breach legible Every Windows dispatch had become a coin flip against the 30-minute job wall. Measured wall time per shard over the last seven lane=all dispatches, in minutes: run 35168946544 30.2 CANCELLED 20.8 20.3 21.3 13.4 21.0 run 35164979005 23.6 22.5 20.4 23.3 13.8 16.6 run 35161399172 23.5 24.7 23.7 26.8 17.2 17.3 run 35152226272 16.5 18.6 20.3 13.7 20.4 24.8 run 35148850553 18.6 20.0 24.8 14.2 23.9 22.8 run 35139132889 21.7 18.8 23.8 18.9 24.7 20.5 run 35134620067 20.5 20.3 19.9 16.8 24.0 28.3 13.4 to 30.2 against a 30-minute ceiling. A shard killed at the wall reports cancelled - neither a pass nor a fail, and with no indication of which file was running when it died. This is the third time this leg has grown into its ceiling; ci.yml already records the first two. One leg reached 30 minutes and died in cleanup, four shards then ran 17-25 minutes with a green 3/4 cancelled at 25m12s, and six were chosen to put each leg at two-thirds of that. Six has now done the same, helped by a suite that keeps growing and by #4835 re-enabling a family that had been skipped. Nine shards, arithmetic in the workflow: total observed work is about 133 minutes, so nine legs project to 26.5 minutes including the ~1.43 slowest-shard skew and the 25% run-to-run variance this file already documents; eight projects to 29.8, which is not margin. The ceiling stays 30 minutes, because raising it is the masking answer and the number is supposed to mean something. The cost is three more concurrent Windows runners and their fixed setup. Cutting work per shard buys time but does not make a wedge readable, so this leg now runs through the same batch runner Linux uses: at-most-12-file processes with a 120-second bound. A timeout or crash fixes the shard red immediately and names the batch; the singleton sweep that follows is diagnosis only and cannot turn it green, exactly as #4837 established. scope=all keeps all 1327 Windows files - Linux alone excludes the storage-policy and api-usage families because separate jobs own them. The aggregate gate counts the nine legs by name through the Actions API. A matrix rolls up to success when a leg never starts, so counting is the only way to know the dispatch produced the evidence it was run to produce. No local suite, focused test, typecheck, build, or install was run. --- .github/workflows/ci.yml | 55 ++++++++-------- scripts/ci/run-bun-test-batches.sh | 25 +++++--- structure/ops/docs-and-release.md | 15 +++-- structure/overview.md | 6 ++ .../ci-bun-crash-classifier.test.ts | 13 ++-- .../ci-workflows/ci-crash-disposition.test.ts | 28 ++++++-- tests/ci-workflows/ci-workflows.test.ts | 64 +++++++++---------- 7 files changed, 115 insertions(+), 91 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8450a7c689..77fe2fae7e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -720,7 +720,7 @@ jobs: # now means Linux + macOS + the gates; Windows re-enters the gate when the # tracked failures are fixed, not before. platform-windows: - name: windows ${{ matrix.shard }}/6 + name: windows ${{ matrix.shard }}/9 needs: select-windows-runner if: >- github.event_name == 'workflow_dispatch' && (github.event.inputs.lane == '' || github.event.inputs.lane == 'all') @@ -747,13 +747,23 @@ jobs: # inside the margin 25 was chosen to provide. # Shard 1 of run 34036848646 then reached that wall with 2736 passing tests # and no test failures. The matched tests were 25% slower than the prior - # complete run; about one minute of tests remained. Keep every test deadline - # and all six shards, but leave the whole batch and cleanup a 30-minute bound. + # complete run; about one minute of tests remained. That change kept every + # test deadline and all six shards, but left the whole batch and cleanup a + # 30-minute bound. + # + # Six shards then grew into the 30-minute ceiling too. Across seven lane=all + # dispatches the six shard totals were 114.3-133.2 minutes. The worst observed + # shard imbalance was 1.43x its dispatch's per-shard average. Eight shards leave + # no margin: 133.2 / 8 * 1.43 * 1.25 = 29.8 minutes after the already-observed + # 25% run-to-run slowdown. Nine gives 133.2 / 9 * 1.43 * 1.25 = 26.5 minutes. + # Keep the 30-minute bound and pay for three more concurrent runners plus their + # repeated checkout/install/build setup so test work, rather than the ceiling, + # shrinks. Per-batch bounds below make a future wedge a named failure. timeout-minutes: 30 strategy: fail-fast: false matrix: - shard: [1, 2, 3, 4, 5, 6] + shard: [1, 2, 3, 4, 5, 6, 7, 8, 9] steps: - name: Show selected runner shell: bash @@ -804,34 +814,21 @@ jobs: cd gui bun run build - - name: Test + - name: Test in fresh-process batches # --timeout: the Linux batches and the macOS control both pass 60000; this leg was # the only one left on Bun's 5s default, and it is the slowest hardware on the board. # Three of its failures were the default firing on tests that had not hung — the # composed-acceptance cases spawn a real `ocx start` and were still working at 41s. # - # Nothing is retried. This leg was the only one that reported a Bun panic honestly on - # BOTH attempts, which is the single reason the 1.4.2 preload segfault was visible at - # all while Linux swept it and macOS retried it. A crash fails the shard on the first - # occurrence; the shared classifier only chooses the message. + # Nothing is retried. The shared runner gives each at-most-12-file process 120 seconds; + # a timeout or crash makes the shard irreversibly red before the diagnostic singleton + # sweep starts. scope=all preserves the full Windows suite: Linux alone excludes the + # storage-policy and api-usage families because separate Linux jobs own them. shell: bash - run: | - set +e - set -uo pipefail - # One shared classifier for every lane; see scripts/ci/bun-crash-signatures.sh. - source scripts/ci/bun-crash-signatures.sh - suite_log="$(mktemp -t ocx-windows-suite.XXXXXX)" - bun test --isolate --timeout 60000 tests --shard=${{ matrix.shard }}/6 2>&1 | tee "$suite_log" - suite_status="${PIPESTATUS[0]}" - if [ "$suite_status" -eq 0 ]; then - exit 0 - fi - if is_bun_runtime_crash "$suite_status" "$suite_log"; then - echo "::error::Bun runtime crash in Windows shard ${{ matrix.shard }}/6 (exit ${suite_status}); a crash is process death, not a test result, and it fails this shard on the first occurrence." - else - echo "::error::Windows shard ${{ matrix.shard }}/6 failed (exit ${suite_status})." - fi - exit "$suite_status" + env: + TEST_SHARD: ${{ matrix.shard }}/9 + BUN_TEST_FILE_SCOPE: all + run: bash scripts/ci/run-bun-test-batches.sh "$TEST_SHARD" - name: CLI help smoke run: bun run src/cli/index.ts help @@ -1099,10 +1096,10 @@ jobs: # started: five successes and one skipped leg roll up to `success`. No matrix here # carries a per-leg `if:`, so a leg can only be skipped when its whole job is, which # the expectation table above already catches. What that table cannot catch is a - # matrix that produced FEWER legs than the six a dispatch is run to read, so count - # them by name. The six Windows legs are the entire output of that dispatch. + # matrix that produced FEWER legs than the nine a dispatch is run to read, so count + # them by name. The nine Windows legs are the entire output of that dispatch. if [ "$windows" = requested ]; then - shards=6 + shards=9 # `filter=latest` (the default) is the latest execution of each job in the run, # which is what a human reading the run sees. Asking for one ATTEMPT instead would # fail every partial re-run: "Re-run failed jobs" puts only the repaired shard in diff --git a/scripts/ci/run-bun-test-batches.sh b/scripts/ci/run-bun-test-batches.sh index 82c2403f20..bee228bbd5 100644 --- a/scripts/ci/run-bun-test-batches.sh +++ b/scripts/ci/run-bun-test-batches.sh @@ -5,6 +5,7 @@ readonly SHARD_SPEC="${1:-}" readonly BATCH_SIZE="${BUN_TEST_BATCH_SIZE:-12}" readonly BATCH_TIMEOUT_SECONDS="${BUN_TEST_BATCH_TIMEOUT_SECONDS:-120}" readonly BATCH_KILL_GRACE_SECONDS="${BUN_TEST_BATCH_KILL_GRACE_SECONDS:-15}" +readonly TEST_FILE_SCOPE="${BUN_TEST_FILE_SCOPE:-general}" # Runtime under test. Defaults to whatever `bun` PATH resolves to; the Bun 1.4 # qualification lane sets OPENCODEX_BUN_PATH so the batches actually execute on # the candidate binary. Without this the lane would export an override, run the @@ -42,6 +43,10 @@ if [[ ! "$BATCH_KILL_GRACE_SECONDS" =~ ^[1-9][0-9]*$ ]]; then echo "BUN_TEST_BATCH_KILL_GRACE_SECONDS must be a positive integer, got: $BATCH_KILL_GRACE_SECONDS" >&2 exit 64 fi +if [[ "$TEST_FILE_SCOPE" != "general" && "$TEST_FILE_SCOPE" != "all" ]]; then + echo "BUN_TEST_FILE_SCOPE must be general or all, got: $TEST_FILE_SCOPE" >&2 + exit 64 +fi if ! command -v timeout >/dev/null 2>&1; then echo "GNU timeout is required to bound Bun test batches." >&2 exit 69 @@ -50,13 +55,17 @@ fi is_general_test_file() { local path="$1" - case "$path" in - # Dedicated CI jobs run these in their own Bun process (ci.yml storage-policy / api-usage). - # Match by basename at any depth so the exclusion survives the tests/ domain layout. - */api-storage-policy*.test.ts|*/api-storage.test.ts|*/api-usage.test.ts) - return 1 - ;; - esac + if [[ "$TEST_FILE_SCOPE" == "general" ]]; then + case "$path" in + # Dedicated Linux CI jobs run these in their own Bun process (ci.yml storage-policy / + # api-usage). Windows sets scope=all because its manual platform leg has always covered + # the full suite and batching must not silently shrink that platform contract. + # Match by basename at any depth so the exclusion survives the tests/ domain layout. + */api-storage-policy*.test.ts|*/api-storage.test.ts|*/api-usage.test.ts) + return 1 + ;; + esac + fi case "$path" in *.test.js|*.test.jsx|*.test.ts|*.test.tsx|*_test.js|*_test.jsx|*_test.ts|*_test.tsx|*.spec.js|*.spec.jsx|*.spec.ts|*.spec.tsx|*_spec.js|*_spec.jsx|*_spec.ts|*_spec.tsx) @@ -186,7 +195,7 @@ if (( ${#SELECTED_FILES[@]} == 0 )); then fi readonly TOTAL_BATCHES=$(( (${#SELECTED_FILES[@]} + BATCH_SIZE - 1) / BATCH_SIZE )) -echo "Shard ${SHARD_SPEC}: ${#SELECTED_FILES[@]} files in ${TOTAL_BATCHES} primary Bun processes (batch size <= ${BATCH_SIZE}, timeout ${BATCH_TIMEOUT_SECONDS}s)." +echo "Shard ${SHARD_SPEC}: ${#SELECTED_FILES[@]} files in ${TOTAL_BATCHES} primary Bun processes (scope ${TEST_FILE_SCOPE}, batch size <= ${BATCH_SIZE}, timeout ${BATCH_TIMEOUT_SECONDS}s)." echo "Nothing here is retried. A test failure, a process timeout and a Bun runtime crash each fail this shard on their first occurrence." echo "A timeout or a crash is additionally swept one file per process for attribution, after the shard has already failed; that sweep cannot turn it green." diff --git a/structure/ops/docs-and-release.md b/structure/ops/docs-and-release.md index e6c2f66cce..22e3f6d573 100644 --- a/structure/ops/docs-and-release.md +++ b/structure/ops/docs-and-release.md @@ -85,7 +85,7 @@ Those controls still have no owner, so there is no image-publish workflow or off | Workflow | Trigger | Purpose | | --- | --- | --- | -| `.github/workflows/ci.yml` | Any `pull_request`; runtime/package `push` to `main`/`preview`/`dev`; manual dispatch | Linux runs four suite shards plus `gates`; macOS runs two shards. Windows runs six shards only on manual dispatch with `lane=all` (or empty), not on push events. No lane retries: a test failure, a process timeout and a Bun runtime crash each fail their job on the first occurrence. Aggregate `ci` is event-aware — it derives which jobs this event requested and requires `success` from each of them and `skipped` from the rest, and on a `lane=all` dispatch it reads the run's own job list and requires six concrete successful `windows N/6` results. `npm-global-smoke` remains GitHub-hosted because it mutates the global package prefix. | +| `.github/workflows/ci.yml` | Any `pull_request`; runtime/package `push` to `main`/`preview`/`dev`; manual dispatch | Linux runs four suite shards plus `gates`; macOS runs two shards. Windows runs nine shards only on manual dispatch with `lane=all` (or empty), not on push events. Linux and Windows run at-most-12-file processes with a 120-second process bound; Windows uses all-file scope so its full-suite contract is unchanged. No lane retries: a test failure, a process timeout and a Bun runtime crash each fail their job on the first occurrence. Aggregate `ci` is event-aware — it derives which jobs this event requested and requires `success` from each of them and `skipped` from the rest, and on a `lane=all` dispatch it reads the run's own job list and requires nine concrete successful `windows N/9` results. `npm-global-smoke` remains GitHub-hosted because it mutates the global package prefix. | | `.github/workflows/dev-version-bump.yml` | Manual dispatch with an intended version and `pre-move` or `repair` mode | Opens the reviewed pull request that moves `dev` past a release target. The default `pre-move` mode runs before promotion and publication; explicit `repair` mode retains the post-publish catch-up path. It is neither called by `release.yml` nor triggered by publication. | | `.github/workflows/release.yml` | Manual dispatch only | npm publish/dry-run workflow. It requires successful Cross-platform CI for the exact `GITHUB_SHA`, requires `dev` to outrank the target, then checks the target against the freshly fetched global tag set before publish or dry-run. | | `.github/workflows/deploy-docs.yml` | `push` to `main` touching `docs-site/**` or the workflow, or manual dispatch | Build and publish the Astro/Starlight docs site to GitHub Pages. | @@ -288,19 +288,22 @@ The [desktop membership contract](../runtime.md#codex-desktop-process-membership `.github/workflows/ci.yml` is the ordinary quality gate for runtime/package changes. Linux runs the suite in four shards with a separate `gates` job, and macOS runs it in two shards. Windows -runs the full suite in six shards only on manual `workflow_dispatch` with `lane=all` (or an +runs the full suite in nine shards only on manual `workflow_dispatch` with `lane=all` (or an empty lane). Pushes to `dev`, `main` and `preview` do not activate that Windows matrix, and an aggregate green `ci` check on those events legitimately includes a deliberate Windows skip. -Nothing in the workflow retries. A test failure, a process timeout and a Bun runtime crash each -fail their job on the first occurrence; `scripts/ci/run-bun-test-batches.sh` still sweeps a +Nothing in the workflow retries. Linux and Windows use `scripts/ci/run-bun-test-batches.sh` to +run at most twelve files per Bun process with a 120-second process bound. Windows selects all +test families, while Linux leaves the storage-policy and api-usage families to its dedicated +jobs. A test failure, a process timeout and a Bun runtime crash each fail their job on the first +occurrence; the batch runner still sweeps a crashed or timed-out batch one file per process, but only to attribute a failure the shard has already taken. The aggregate `ci` gate derives, from the event and the `changes` outputs, which jobs this run actually requested, then requires `success` from every one of them and `skipped` from every job the event did not request — so a job that was requested and never started can no longer report as a deliberate skip. On a `lane=all` dispatch the gate additionally reads the -run's own job list through the Actions API and requires six concrete successful `windows N/6` -results, because a matrix rollup reports `success` for five successes and one skipped leg. A +run's own job list through the Actions API and requires nine concrete successful `windows N/9` +results, because a matrix rollup can report `success` when one matrix leg is skipped. A release that requires Windows proof still dispatches it for the exact publish SHA. Across the jobs, the workflow runs: diff --git a/structure/overview.md b/structure/overview.md index d8f01d4bbc..2e8f096a1c 100644 --- a/structure/overview.md +++ b/structure/overview.md @@ -106,6 +106,12 @@ still cover the rule, which is a judgement only review makes. through `tests/helpers/repo-root.ts`, never `import.meta.dir + "/.."`. Enforced by `tests/test-layout.test.ts`. +CI enumerates that domain layout through `scripts/ci/run-bun-test-batches.sh`. Its default general +scope leaves the dedicated Linux storage-policy and api-usage jobs out of the general shards; the +manual Windows matrix selects all-file scope so batching changes process size without changing +the platform suite's file set. The workflow contract and process bounds live in +[`ops/docs-and-release.md`](ops/docs-and-release.md#cross-platform-ci). + Two invariants are stated here without a binding, and `grace.unboundInvariants` in [`manifest.json`](manifest.json) carries the reason for each. They are true statements about the system; no test in this repository currently pins them, and saying so is more useful than naming a test that diff --git a/tests/ci-workflows/ci-bun-crash-classifier.test.ts b/tests/ci-workflows/ci-bun-crash-classifier.test.ts index 68183827ef..62c95ed0c2 100644 --- a/tests/ci-workflows/ci-bun-crash-classifier.test.ts +++ b/tests/ci-workflows/ci-bun-crash-classifier.test.ts @@ -1,5 +1,6 @@ /** - * The Bun crash classifier is one definition, every lane sources it, and a crash fails the shard. + * The Bun crash classifier is one definition, every direct lane or shared runner uses it, and a + * crash fails the shard. * * Two separate defects are pinned here. * @@ -15,9 +16,9 @@ * success when that sweep passed. The sweep is not a retry of a flaky test: one file per process * is a configuration in which this class of defect cannot occur, so it was guaranteed to pass and * guaranteed to report nothing. Linux CI segfaulted twelve to fourteen times per run from - * 2026-09-08 while reporting green, and the Windows lane -- which has no sweep -- was the only - * place the Bun 1.4.2 regression was visible at all. The sweep is kept for attribution; the shard - * now fails regardless of its result. + * 2026-09-08 while reporting green, and the then-unbatched Windows lane was the only place the + * Bun 1.4.2 regression was visible at all. The sweep is kept for attribution; the shard now fails + * regardless of its result. * * What this file may and may not assert. Reading shell SOURCE TEXT proves only that a string is * present, which is why the disposition contract does NOT live here any more: the old @@ -65,7 +66,6 @@ describe("the Bun crash classifier is shared", () => { const workflow = read(".github", "workflows", "ci.yml"); const lanes = { - windows: runBlockContaining(workflow, "bun test --isolate --timeout 60000 tests --shard=${{ matrix.shard }}/6"), "macos-shard": runBlockContaining(workflow, "run_macos_suite tests"), "macos-control": runBlockContaining(workflow, "bun test --isolate --timeout 60000 tests 2>&1"), }; @@ -89,7 +89,7 @@ describe("the Bun crash classifier is shared", () => { } }); - test("every lane sources the classifier and calls the shared predicate", () => { + test("every direct lane and the shared batch runner use the classifier", () => { for (const [name, text] of Object.entries(lanes)) { expect(`${name}:sources:${text.includes(SOURCE_LINE)}`).toBe(`${name}:sources:true`); expect(`${name}:calls:${text.includes("is_bun_runtime_crash \"$suite_status\" \"$suite_log\"")}`) @@ -97,6 +97,7 @@ describe("the Bun crash classifier is shared", () => { } expect(batchScript).toContain("bun-crash-signatures.sh"); expect(batchScript).toContain('is_bun_runtime_crash "$status" "$log_file"'); + expect(workflow.match(/run: bash scripts\/ci\/run-bun-test-batches\.sh/g)).toHaveLength(2); }); test("the thread-numbered panic form is the anchor nowhere", () => { diff --git a/tests/ci-workflows/ci-crash-disposition.test.ts b/tests/ci-workflows/ci-crash-disposition.test.ts index 801bc9029b..5034e27f14 100644 --- a/tests/ci-workflows/ci-crash-disposition.test.ts +++ b/tests/ci-workflows/ci-crash-disposition.test.ts @@ -16,12 +16,11 @@ * `timeout` that reproduce a crash, a hang and an assertion failure on demand. The assertion * in every runner case is the process exit status, which is the only thing GitHub reads. * - * The two halves have different platform reach on purpose. The classifier is portable shell, so - * it runs wherever a POSIX shell exists. The batch runner is Linux-only CI tooling -- - * `mapfile -d ''` needs Bash 4.4 and macOS ships Bash 3.2 -- and its job is - * `runs-on: ubuntu-latest`, so it is executed on the platform that actually runs it. That is - * coverage matched to where the code lives rather than a platform skip that dodges a failure: - * the Linux shards run this file on every push and every pull request. + * The two halves have different local harness reach on purpose. The classifier is portable shell, + * so it runs wherever a POSIX shell exists. The batch runner executes in Linux and Windows CI; + * this fake-toolchain harness stays Linux-only because it synthesizes GNU `timeout` and POSIX + * process statuses. The manual Windows matrix exercises the real Git-for-Windows Bash/coreutils + * path. That is platform evidence matched to the actual runner rather than a local emulation. */ import { describe, expect, test } from "bun:test"; import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; @@ -103,6 +102,7 @@ describe.skipIf(process.platform === "win32")("the shared Bun crash classifier, // stops rather than continuing to collect batches it can no longer pass. const FIXTURE_FILES = ["alpha", "bravo", "charlie", "delta", "echo", "foxtrot"] .map(name => `${name}.test.ts`); +const DEDICATED_FILE = "api-usage.test.ts"; const FIRST_BATCH = FIXTURE_FILES.slice(0, 3); const SECOND_BATCH = FIXTURE_FILES.slice(3); @@ -159,7 +159,10 @@ const FAKE_BUN = [ type RunnerResult = { status: number | null; output: string; calls: string[] }; -function runBatches(mode: "green" | "crash" | "timeout" | "assert"): RunnerResult { +function runBatches( + mode: "green" | "crash" | "timeout" | "assert", + fileScope: "general" | "all" = "general", +): RunnerResult { const directory = mkdtempSync(join(tmpdir(), "ocx-batch-disposition-")); try { const binDirectory = join(directory, "bin"); @@ -167,6 +170,7 @@ function runBatches(mode: "green" | "crash" | "timeout" | "assert"): RunnerResul mkdirSync(join(directory, "tmp")); mkdirSync(join(directory, "tests")); for (const file of FIXTURE_FILES) writeFileSync(join(directory, "tests", file), ""); + writeFileSync(join(directory, "tests", DEDICATED_FILE), ""); writeFileSync(join(binDirectory, "timeout"), FAKE_TIMEOUT, { mode: 0o755 }); writeFileSync(join(binDirectory, "bun"), FAKE_BUN, { mode: 0o755 }); const calls = join(directory, "calls.log"); @@ -180,6 +184,7 @@ function runBatches(mode: "green" | "crash" | "timeout" | "assert"): RunnerResul TMPDIR: join(directory, "tmp"), CI: "true", BUN_TEST_BATCH_SIZE: "3", + BUN_TEST_FILE_SCOPE: fileScope, OPENCODEX_BUN_PATH: join(binDirectory, "bun"), FIXTURE_MODE: mode, FIXTURE_CALLS: calls, @@ -209,6 +214,15 @@ describe.skipIf(process.platform !== "linux")("the Linux batch runner, executed" expect(`status:${run.status}`, run.output).toBe("status:0"); expect(batchCalls(run)).toHaveLength(2); expect(singletonCalls(run)).toEqual([]); + expect(run.calls.some(call => call.includes(DEDICATED_FILE))).toBe(false); + }, SPAWN_BUDGET_MS); + + test("all scope preserves the dedicated families in the Windows suite", () => { + const run = runBatches("green", "all"); + expect(`status:${run.status}`, run.output).toBe("status:0"); + expect(batchCalls(run)).toHaveLength(3); + expect(run.calls.some(call => call.includes(DEDICATED_FILE))).toBe(true); + expect(singletonCalls(run)).toEqual([]); }, SPAWN_BUDGET_MS); test("a runtime crash fails the shard even though every file passes alone", () => { diff --git a/tests/ci-workflows/ci-workflows.test.ts b/tests/ci-workflows/ci-workflows.test.ts index d376fb5621..a8ec9f1161 100644 --- a/tests/ci-workflows/ci-workflows.test.ts +++ b/tests/ci-workflows/ci-workflows.test.ts @@ -193,20 +193,23 @@ describe("GitHub Actions hardening", () => { expect(`${jobName}:${String(checkout?.with?.["fetch-tags"])}`).toBe(`${jobName}:true`); } - // Windows shards more finely than Linux: the same suite takes 17-25 minutes per - // quarter on windows-latest, which is the leg's own 25-minute ceiling (run - // 33934756997 cancelled a green 3/4 at 25m12s). The invariant that matters is the - // one above — the matrix and the divisor tile the suite exactly — so pin the - // Windows matrix to its own divisor rather than to Linux's, and pin it to be - // contiguous from 1 so a dropped entry cannot leave a slice of the suite unrun. + // Windows shards more finely than Linux. Six shards grew to 13-30 minutes against + // the 30-minute wall; nine restores margin while keeping the bound unchanged. The + // matrix, runner shard spec, job name and aggregate leg count must move together. const windowsShards = (ci.jobs?.["platform-windows"] as { strategy?: { matrix?: { shard?: number[] } }; })?.strategy?.matrix?.shard ?? []; - expect(windowsShards).toEqual([1, 2, 3, 4, 5, 6]); + expect(windowsShards).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9]); expect(windowsShards).toEqual(windowsShards.map((_, i) => i + 1)); - const windowsSteps = (ci.jobs?.["platform-windows"] as { steps?: Array<{ run?: string }> })?.steps ?? []; - expect(windowsSteps.some(step => step.run?.includes(`--shard=\${{ matrix.shard }}/${windowsShards.length}`))).toBe(true); + const windowsSteps = (ci.jobs?.["platform-windows"] as { + steps?: Array<{ name?: string; env?: Record; run?: string }>; + })?.steps ?? []; + const windowsTest = windowsSteps.find(step => step.name === "Test in fresh-process batches"); + expect(windowsTest?.env?.TEST_SHARD).toBe(`\${{ matrix.shard }}/${windowsShards.length}`); + expect(windowsTest?.env?.BUN_TEST_FILE_SCOPE).toBe("all"); + expect(windowsTest?.run).toBe('bash scripts/ci/run-bun-test-batches.sh "$TEST_SHARD"'); expect(ci.jobs?.["platform-windows"]?.name).toBe(`windows \${{ matrix.shard }}/${windowsShards.length}`); + expect(workflow).toContain(`shards=${windowsShards.length}`); // The aggregate gate is the check a human trusts. Three ways to break it // silently: drop `if: always()` so it skips (and a skipped job reports @@ -335,35 +338,26 @@ describe("GitHub Actions hardening", () => { // self-hosted workspace wipe. Without the wipe a deleted file survives on // the runner's disk and the suite passes against a tree that no longer // exists in git. - const winSteps = (ci.jobs?.["platform-windows"] as { steps?: { if?: string; run?: string }[] })?.steps ?? []; - // --timeout is part of the contract, not incidental: this leg ran on Bun's 5s default - // while Linux and macOS both pass 60000, and it is the slowest hardware on the board. - // Three composed-acceptance failures were that default firing on tests still working - // at 41s. Pin the flag so the leg cannot silently drift back to the default. - const windowsTestCommand = `bun test --isolate --timeout 60000 tests --shard=\${{ matrix.shard }}/${windowsShards.length}`; - expect(hasShellCommandHead(`echo ${windowsTestCommand}`, windowsTestCommand)).toBe(false); - // Binding the assertion to an executable line is only half the guarantee: a - // step carrying the exact command still runs nothing under `if: false`, and - // the suite would stay green against a Windows leg that never tests. Require - // the matching step to be unconditional. - const windowsTestSteps = winSteps.filter(step => hasShellCommandHead(step.run, windowsTestCommand)); - expect(windowsTestSteps.length).toBeGreaterThan(0); - expect(windowsTestSteps.every(step => step.if === undefined)).toBe(true); + const winSteps = (ci.jobs?.["platform-windows"] as { + steps?: { if?: string; name?: string; run?: string }[]; + })?.steps ?? []; + const windowsBatchStep = winSteps.find(step => step.name === "Test in fresh-process batches"); + expect(windowsBatchStep?.run).toBe('bash scripts/ci/run-bun-test-batches.sh "$TEST_SHARD"'); + // A step carrying the runner still runs nothing under `if: false`; require the + // suite step to be unconditional. + expect(windowsBatchStep?.if).toBeUndefined(); expect(winSteps.some(step => step.if === "runner.environment == 'self-hosted'" && step.run?.includes("git clean -xffd"))).toBe(true); - // The crash-signature list lives in exactly one file now, and every lane sources it. - // ci-bun-crash-classifier.test.ts owns that contract, including the rule that no lane may - // reintroduce an inline copy and that a runtime crash fails the shard instead of being swept. - const windowsTestRun = windowsTestSteps[0]?.run ?? ""; - - // Windows carries the same single-attempt disposition as macOS. It is also the lane - // that proved why: it was the only one reporting a Bun panic honestly on both of its - // attempts, which is the single reason the 1.4.2 preload segfault was visible at all. - expect(hasExactShellCommand(windowsTestRun, "set +e")).toBe(true); - expect(windowsTestRun).not.toContain("for attempt in"); - expect(windowsTestRun).not.toContain("while true"); - expect(windowsTestRun).toContain("it fails this shard on the first occurrence"); + // Windows now shares Linux's bounded process runner. Pin the two bounds and the + // 60-second per-test deadline at their executable owner, and keep retries absent. + const batchRunner = await readText("scripts/ci/run-bun-test-batches.sh"); + expect(batchRunner).toContain('readonly BATCH_SIZE="${BUN_TEST_BATCH_SIZE:-12}"'); + expect(batchRunner).toContain('readonly BATCH_TIMEOUT_SECONDS="${BUN_TEST_BATCH_TIMEOUT_SECONDS:-120}"'); + expect(batchRunner).toContain('"$BUN_BIN" test --isolate --timeout 60000 "${files[@]}"'); + expect(batchRunner).not.toContain("for attempt in"); + expect(batchRunner).not.toContain("while true"); + expect(batchRunner).toContain("fail this shard on their first occurrence"); // Every job that runs the root suite must build the GUI first, unconditionally. // Tests that fetch the served dashboard read their session bootstrap out of From c8a9631b1dc117fd2ece8a976193b650f48fc42f Mon Sep 17 00:00:00 2001 From: JUN Date: Thu, 17 Sep 2026 11:22:02 +0900 Subject: [PATCH 2/3] ci(windows): size the batch bound from Windows data, not Linux's The first attempt gave this leg Linux's batch settings unchanged - 12 files, 120 seconds - and 7 of 9 shards went red on dispatch 35171877721. The runner reported it precisely: "batch 5 timeout failure (exit 124)" followed by "every file passed alone, so the timeout lives in multi-file process state". That second line is the report you get when a bound is simply too small, not when something is wedged. Windows is the slowest hardware on the board, which is the whole reason this leg needed nine shards; a bound copied from the fastest one was never going to hold. Measured across 58 completed batches in that dispatch: median 39.1s, p90 92.6s, p95 100.1s, max 105.8s, and seven batches reached the 120s ceiling. The bound sat at roughly the mean, so about half of all batches were always going to breach it. Six files per batch with a 480-second bound. The sizing case is one naturally slow file: codex-inject-integration.test.ts passes in 312.0s and 317.6s in green runs, so its six-file batch projects to 337.4s, and 421.8s with the 25% run-to-run variance this workflow already documents. 480 leaves 58.2s over that. Six-file attribution halves topped out at 148.0s, so every other batch has an enormous margin. Linux keeps 12 files and 120 seconds. That number is correctly sized for that hardware and sharing one constant across two very different machines is what caused this. The two numbers are independent. Batch size and bound decide how quickly a wedge is named; the nine-shard split decides total wall time. Six-file batches add 12 processes per shard at a measured 0.106-0.168s of wrapper overhead each, about 2.1s per shard, so the margin arithmetic in the shard comment is unchanged. A real wedge now fails within eight minutes naming at most six files, with singleton attribution after the shard is already red. No local suite, focused test, typecheck, build, or install was run. --- .github/workflows/ci.yml | 21 ++++++++++++++++----- structure/ops/docs-and-release.md | 12 ++++++------ structure/overview.md | 7 ++++--- tests/ci-workflows/ci-workflows.test.ts | 7 +++++-- 4 files changed, 31 insertions(+), 16 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 77fe2fae7e..5fc5188498 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -758,7 +758,9 @@ jobs: # 25% run-to-run slowdown. Nine gives 133.2 / 9 * 1.43 * 1.25 = 26.5 minutes. # Keep the 30-minute bound and pay for three more concurrent runners plus their # repeated checkout/install/build setup so test work, rather than the ceiling, - # shrinks. Per-batch bounds below make a future wedge a named failure. + # shrinks. The per-batch timeout below is independent: it aborts a stuck Bun + # process but adds no delay to a healthy one, so calibrating it does not change + # this total-work projection. timeout-minutes: 30 strategy: fail-fast: false @@ -820,14 +822,23 @@ jobs: # Three of its failures were the default firing on tests that had not hung — the # composed-acceptance cases spawn a real `ocx start` and were still working at 41s. # - # Nothing is retried. The shared runner gives each at-most-12-file process 120 seconds; - # a timeout or crash makes the shard irreversibly red before the diagnostic singleton - # sweep starts. scope=all preserves the full Windows suite: Linux alone excludes the - # storage-policy and api-usage families because separate Linux jobs own them. + # Nothing is retried. Run 35171877721 proved Linux's 12-file/120-second defaults are + # not Windows defaults: 58 completed primary batches took 4.6-105.8s and seven more + # hit 120s. Six of those passed every file alone; splitting their attribution times + # into six-file halves gives a 148.0s maximum. The seventh carried + # codex-inject-integration.test.ts, which passed in 312.0s and 317.6s in green runs + # 35164979005 and 35161399172. Replacing its censored 120s attribution with 317.6s projects + # that six-file half at 337.4s; 25% run variance makes 421.8s, so 480s leaves 58.2s. + # Six-file batches add twelve Bun processes per shard, but the two green shards measured + # only 0.106-0.168s of wrapper overhead per process: at most ~2.1s against the margin. + # A timeout or crash still fixes the shard red before singleton attribution. scope=all + # preserves the full Windows suite; Linux keeps its correctly sized 12-file/120s defaults. shell: bash env: TEST_SHARD: ${{ matrix.shard }}/9 BUN_TEST_FILE_SCOPE: all + BUN_TEST_BATCH_SIZE: "6" + BUN_TEST_BATCH_TIMEOUT_SECONDS: "480" run: bash scripts/ci/run-bun-test-batches.sh "$TEST_SHARD" - name: CLI help smoke diff --git a/structure/ops/docs-and-release.md b/structure/ops/docs-and-release.md index 22e3f6d573..2f37944405 100644 --- a/structure/ops/docs-and-release.md +++ b/structure/ops/docs-and-release.md @@ -85,7 +85,7 @@ Those controls still have no owner, so there is no image-publish workflow or off | Workflow | Trigger | Purpose | | --- | --- | --- | -| `.github/workflows/ci.yml` | Any `pull_request`; runtime/package `push` to `main`/`preview`/`dev`; manual dispatch | Linux runs four suite shards plus `gates`; macOS runs two shards. Windows runs nine shards only on manual dispatch with `lane=all` (or empty), not on push events. Linux and Windows run at-most-12-file processes with a 120-second process bound; Windows uses all-file scope so its full-suite contract is unchanged. No lane retries: a test failure, a process timeout and a Bun runtime crash each fail their job on the first occurrence. Aggregate `ci` is event-aware — it derives which jobs this event requested and requires `success` from each of them and `skipped` from the rest, and on a `lane=all` dispatch it reads the run's own job list and requires nine concrete successful `windows N/9` results. `npm-global-smoke` remains GitHub-hosted because it mutates the global package prefix. | +| `.github/workflows/ci.yml` | Any `pull_request`; runtime/package `push` to `main`/`preview`/`dev`; manual dispatch | Linux runs four suite shards plus `gates`; macOS runs two shards. Windows runs nine shards only on manual dispatch with `lane=all` (or empty), not on push events. Linux runs at-most-12-file processes with a 120-second process bound; Windows uses measured six-file/480-second processes and all-file scope so its full-suite contract is unchanged. No lane retries: a test failure, a process timeout and a Bun runtime crash each fail their job on the first occurrence. Aggregate `ci` is event-aware — it derives which jobs this event requested and requires `success` from each of them and `skipped` from the rest, and on a `lane=all` dispatch it reads the run's own job list and requires nine concrete successful `windows N/9` results. `npm-global-smoke` remains GitHub-hosted because it mutates the global package prefix. | | `.github/workflows/dev-version-bump.yml` | Manual dispatch with an intended version and `pre-move` or `repair` mode | Opens the reviewed pull request that moves `dev` past a release target. The default `pre-move` mode runs before promotion and publication; explicit `repair` mode retains the post-publish catch-up path. It is neither called by `release.yml` nor triggered by publication. | | `.github/workflows/release.yml` | Manual dispatch only | npm publish/dry-run workflow. It requires successful Cross-platform CI for the exact `GITHUB_SHA`, requires `dev` to outrank the target, then checks the target against the freshly fetched global tag set before publish or dry-run. | | `.github/workflows/deploy-docs.yml` | `push` to `main` touching `docs-site/**` or the workflow, or manual dispatch | Build and publish the Astro/Starlight docs site to GitHub Pages. | @@ -292,11 +292,11 @@ runs the full suite in nine shards only on manual `workflow_dispatch` with `lane empty lane). Pushes to `dev`, `main` and `preview` do not activate that Windows matrix, and an aggregate green `ci` check on those events legitimately includes a deliberate Windows skip. -Nothing in the workflow retries. Linux and Windows use `scripts/ci/run-bun-test-batches.sh` to -run at most twelve files per Bun process with a 120-second process bound. Windows selects all -test families, while Linux leaves the storage-policy and api-usage families to its dedicated -jobs. A test failure, a process timeout and a Bun runtime crash each fail their job on the first -occurrence; the batch runner still sweeps a +Nothing in the workflow retries. Linux and Windows use `scripts/ci/run-bun-test-batches.sh`, but +each lane owns its measured process shape: Linux keeps the default twelve files and 120 seconds; +Windows uses six files and 480 seconds. Windows selects all test families, while Linux leaves the +storage-policy and api-usage families to its dedicated jobs. A test failure, a process timeout +and a Bun runtime crash each fail their job on the first occurrence; the batch runner still sweeps a crashed or timed-out batch one file per process, but only to attribute a failure the shard has already taken. The aggregate `ci` gate derives, from the event and the `changes` outputs, which jobs this run actually requested, then requires `success` from every one of them and `skipped` diff --git a/structure/overview.md b/structure/overview.md index 2e8f096a1c..fb5d97ba2e 100644 --- a/structure/overview.md +++ b/structure/overview.md @@ -107,9 +107,10 @@ still cover the rule, which is a judgement only review makes. Enforced by `tests/test-layout.test.ts`. CI enumerates that domain layout through `scripts/ci/run-bun-test-batches.sh`. Its default general -scope leaves the dedicated Linux storage-policy and api-usage jobs out of the general shards; the -manual Windows matrix selects all-file scope so batching changes process size without changing -the platform suite's file set. The workflow contract and process bounds live in +scope and 12-file/120-second process shape leave the dedicated Linux storage-policy and api-usage +jobs out of the general shards. The manual Windows matrix selects all-file scope and overrides the +process shape to six files and 480 seconds, so batching changes process size without changing the +platform suite's file set. The workflow contract and process bounds live in [`ops/docs-and-release.md`](ops/docs-and-release.md#cross-platform-ci). Two invariants are stated here without a binding, and `grace.unboundInvariants` in diff --git a/tests/ci-workflows/ci-workflows.test.ts b/tests/ci-workflows/ci-workflows.test.ts index a8ec9f1161..0eb242705f 100644 --- a/tests/ci-workflows/ci-workflows.test.ts +++ b/tests/ci-workflows/ci-workflows.test.ts @@ -207,6 +207,8 @@ describe("GitHub Actions hardening", () => { const windowsTest = windowsSteps.find(step => step.name === "Test in fresh-process batches"); expect(windowsTest?.env?.TEST_SHARD).toBe(`\${{ matrix.shard }}/${windowsShards.length}`); expect(windowsTest?.env?.BUN_TEST_FILE_SCOPE).toBe("all"); + expect(windowsTest?.env?.BUN_TEST_BATCH_SIZE).toBe("6"); + expect(windowsTest?.env?.BUN_TEST_BATCH_TIMEOUT_SECONDS).toBe("480"); expect(windowsTest?.run).toBe('bash scripts/ci/run-bun-test-batches.sh "$TEST_SHARD"'); expect(ci.jobs?.["platform-windows"]?.name).toBe(`windows \${{ matrix.shard }}/${windowsShards.length}`); expect(workflow).toContain(`shards=${windowsShards.length}`); @@ -349,8 +351,9 @@ describe("GitHub Actions hardening", () => { expect(winSteps.some(step => step.if === "runner.environment == 'self-hosted'" && step.run?.includes("git clean -xffd"))).toBe(true); - // Windows now shares Linux's bounded process runner. Pin the two bounds and the - // 60-second per-test deadline at their executable owner, and keep retries absent. + // Windows shares Linux's bounded process runner but overrides the process shape with + // Windows measurements above. Pin Linux's 12-file/120s defaults at their owner so the + // Windows calibration cannot silently widen the correctly sized Linux lane. const batchRunner = await readText("scripts/ci/run-bun-test-batches.sh"); expect(batchRunner).toContain('readonly BATCH_SIZE="${BUN_TEST_BATCH_SIZE:-12}"'); expect(batchRunner).toContain('readonly BATCH_TIMEOUT_SECONDS="${BUN_TEST_BATCH_TIMEOUT_SECONDS:-120}"'); From a74a897343a32f054f12483640eac56353df03a1 Mon Sep 17 00:00:00 2001 From: JUN Date: Thu, 17 Sep 2026 12:01:12 +0900 Subject: [PATCH 3/3] test(ci): stop the batch oracle from discarding a one-file primary batch The new scope=all case failed expecting three batches and seeing two, and the interesting part is that the runner was right and the test was wrong. batchCalls() classified every invocation beginning with "1|" as singleton attribution. Seven fixture files at batch size three is a valid primary sequence of 3, 3, 1 - so the oracle threw away the last real batch and then reported the count it had just corrupted. A test that miscounts and then asserts its own miscount is the same false confidence this branch has been removing elsewhere, so the fix is the oracle, not the number. It now asserts the exact primary sequence 3, 3, 1, checks the runner's own summary line for seven files in three processes, and still requires the dedicated file to appear. Windows coverage was verified independently rather than assumed, because a scope that silently dropped the dedicated families would be exactly the silent loss this round exists to prevent. From dispatch 35174148018: 1327 test files in the repository, 1320 in general scope, 7 dedicated; the Windows legs ran 148x4 + 147x5 = 1327, and the logs show all seven - tests/server/api-usage.test.ts and the six storage-policy files - executing across shards 3 through 8. That dispatch also carried the calibration result: nine Windows shards, all green, at 10.3 12.2 12.5 13.1 13.6 13.8 15.0 15.1 17.1 minutes against the 30-minute wall, against a six-shard spread of 13.4 to 30.2. No local suite, focused test, typecheck, build, or install was run. --- tests/ci-workflows/ci-crash-disposition.test.ts | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/ci-workflows/ci-crash-disposition.test.ts b/tests/ci-workflows/ci-crash-disposition.test.ts index 5034e27f14..b87e4eeefb 100644 --- a/tests/ci-workflows/ci-crash-disposition.test.ts +++ b/tests/ci-workflows/ci-crash-disposition.test.ts @@ -220,9 +220,13 @@ describe.skipIf(process.platform !== "linux")("the Linux batch runner, executed" test("all scope preserves the dedicated families in the Windows suite", () => { const run = runBatches("green", "all"); expect(`status:${run.status}`, run.output).toBe("status:0"); - expect(batchCalls(run)).toHaveLength(3); + // Seven files at batch size three produce two full primary batches and one + // one-file primary batch. `singletonCalls` deliberately classifies by file + // count for the failure fixtures below, so it cannot distinguish that final + // primary batch from attribution. Assert the complete green call sequence. + expect(run.calls.map(call => Number(call.split("|", 1)[0]))).toEqual([3, 3, 1]); + expect(run.output).toContain("7 files in 3 primary Bun processes (scope all"); expect(run.calls.some(call => call.includes(DEDICATED_FILE))).toBe(true); - expect(singletonCalls(run)).toEqual([]); }, SPAWN_BUDGET_MS); test("a runtime crash fails the shard even though every file passes alone", () => {