Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -833,12 +833,20 @@ jobs:
# only 0.106-0.168s of wrapper overhead per process: at most ~2.1s against the margin.
# A timeout or crash still fixes the shard red before singleton attribution. scope=all
# preserves the full Windows suite; Linux keeps its correctly sized 12-file/120s defaults.
#
# The preload's Windows-only user lock serializes separate test runners on one machine.
# These batches are already one dedicated job's sequential pieces, so treating each Bun
# process as a competing runner can queue batch N+1 behind a straggler from batch N until
# this step's 480s process bound fires without running a test. Disable that outer queue for
# this step only. Every process still creates its own isolated home and arms the live-home
# and service-manager guard before the lock boundary.
shell: bash
env:
TEST_SHARD: ${{ matrix.shard }}/9
BUN_TEST_FILE_SCOPE: all
BUN_TEST_BATCH_SIZE: "6"
BUN_TEST_BATCH_TIMEOUT_SECONDS: "480"
OCX_TEST_NO_QUEUE: "1"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep the Windows lock integration test enabled

Setting this variable on the whole workflow step propagates it into every Bun test process, so tests/ci-workflows/test-runner.test.ts:1038 evaluates its Windows-only test.if(... && process.env[TEST_RUN_NO_QUEUE_ENV] !== "1") to false. That test is the only executed coverage proving nested Windows Bun processes inherit and validate the live lock capability; because it is already skipped on non-Windows platforms, this change silently disables it everywhere. Scope the bypass so the outer batch runner consumes it without exposing it to test code, or explicitly restore the queued environment for this integration case.

AGENTS.md reference: .github/AGENTS.md:L18-L18

Useful? React with 👍 / 👎.

run: bash scripts/ci/run-bun-test-batches.sh "$TEST_SHARD"

- name: CLI help smoke
Expand Down
14 changes: 9 additions & 5 deletions structure/ops/docs-and-release.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ Those controls still have no owner, so there is no image-publish workflow or off

| Workflow | Trigger | Purpose |
| --- | --- | --- |
| `.github/workflows/ci.yml` | Any `pull_request`; runtime/package `push` to `main`/`preview`/`dev`; manual dispatch | Linux runs four suite shards plus `gates`; macOS runs two shards. Windows runs nine shards only on manual dispatch with `lane=all` (or empty), not on push events. Linux runs at-most-12-file processes with a 120-second process bound; Windows uses measured six-file/480-second processes and all-file scope so its full-suite contract is unchanged. No lane retries: a test failure, a process timeout and a Bun runtime crash each fail their job on the first occurrence. Aggregate `ci` is event-aware — it derives which jobs this event requested and requires `success` from each of them and `skipped` from the rest, and on a `lane=all` dispatch it reads the run's own job list and requires nine concrete successful `windows N/9` results. `npm-global-smoke` remains GitHub-hosted because it mutates the global package prefix. |
| `.github/workflows/ci.yml` | Any `pull_request`; runtime/package `push` to `main`/`preview`/`dev`; manual dispatch | Linux runs four suite shards plus `gates`; macOS runs two shards. Windows runs nine shards only on manual dispatch with `lane=all` (or empty), not on push events. Linux runs at-most-12-file processes with a 120-second process bound; Windows uses measured six-file/480-second processes and all-file scope so its full-suite contract is unchanged. The dedicated Windows batch step sets `OCX_TEST_NO_QUEUE=1` because its sequential processes are one logical runner; each process still creates an isolated home and arms the test guards before the lock boundary. No lane retries: a test failure, a process timeout and a Bun runtime crash each fail their job on the first occurrence. Aggregate `ci` is event-aware — it derives which jobs this event requested and requires `success` from each of them and `skipped` from the rest, and on a `lane=all` dispatch it reads the run's own job list and requires nine concrete successful `windows N/9` results. `npm-global-smoke` remains GitHub-hosted because it mutates the global package prefix. |
| `.github/workflows/dev-version-bump.yml` | Manual dispatch with an intended version and `pre-move` or `repair` mode | Opens the reviewed pull request that moves `dev` past a release target. The default `pre-move` mode runs before promotion and publication; explicit `repair` mode retains the post-publish catch-up path. It is neither called by `release.yml` nor triggered by publication. |
| `.github/workflows/release.yml` | Manual dispatch only | npm publish/dry-run workflow. It requires successful Cross-platform CI for the exact `GITHUB_SHA`, requires `dev` to outrank the target, then checks the target against the freshly fetched global tag set before publish or dry-run. |
| `.github/workflows/deploy-docs.yml` | `push` to `main` touching `docs-site/**` or the workflow, or manual dispatch | Build and publish the Astro/Starlight docs site to GitHub Pages. |
Expand Down Expand Up @@ -299,10 +299,14 @@ aggregate green `ci` check on those events legitimately includes a deliberate Wi
Nothing in the workflow retries. Linux and Windows use `scripts/ci/run-bun-test-batches.sh`, but
each lane owns its measured process shape: Linux keeps the default twelve files and 120 seconds;
Windows uses six files and 480 seconds. Windows selects all test families, while Linux leaves the
storage-policy and api-usage families to its dedicated jobs. A test failure, a process timeout
and a Bun runtime crash each fail their job on the first occurrence; the batch runner still sweeps a
crashed or timed-out batch one file per process, but only to attribute a failure the shard has
already taken. The aggregate `ci` gate derives, from the event and the `changes` outputs, which
storage-policy and api-usage families to its dedicated jobs. The Windows step disables the
user-scoped test-run queue with `OCX_TEST_NO_QUEUE=1`: the batches already run sequentially in one
dedicated job, and queueing a new batch behind a surviving process from the preceding batch spends
the process timeout without executing tests. The per-process home isolation and live-home/service
manager guards remain active because the preload installs them before the lock boundary. A test
failure, a process timeout and a Bun runtime crash each fail their job on the first occurrence; the
batch runner still sweeps a crashed or timed-out batch one file per process, but only to attribute a
failure the shard has already taken. The aggregate `ci` gate derives, from the event and the `changes` outputs, which
jobs this run actually requested, then requires `success` from every one of them and `skipped`
from every job the event did not request — so a job that was requested and never started can no
longer report as a deliberate skip. On a `lane=all` dispatch the gate additionally reads the
Expand Down
4 changes: 3 additions & 1 deletion structure/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,9 @@ CI enumerates that domain layout through `scripts/ci/run-bun-test-batches.sh`. I
scope and 12-file/120-second process shape leave the dedicated Linux storage-policy and api-usage
jobs out of the general shards. The manual Windows matrix selects all-file scope and overrides the
process shape to six files and 480 seconds, so batching changes process size without changing the
platform suite's file set. The workflow contract and process bounds live in
platform suite's file set. Its dedicated batch step sets `OCX_TEST_NO_QUEUE=1`: those sequential
processes are one logical runner, while each process still installs its own isolated home and test
guards. The workflow contract and process bounds live in
[`ops/docs-and-release.md`](ops/docs-and-release.md#cross-platform-ci).

Two invariants are stated here without a binding, and `grace.unboundInvariants` in
Expand Down
9 changes: 8 additions & 1 deletion tests/ci-workflows/ci-crash-disposition.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ const FAKE_BUN = [
'for arg in "$@"; do',
' case "$arg" in *.test.ts) files=$((files + 1)) ;; esac',
"done",
"printf '%s|%s\\n' \"$files\" \"$*\" >> \"$FIXTURE_CALLS\"",
"printf '%s|%s|%s\\n' \"$files\" \"${OCX_TEST_NO_QUEUE:-}\" \"$*\" >> \"$FIXTURE_CALLS\"",
'if [ "$files" -le 1 ]; then',
" exit 0",
"fi",
Expand Down Expand Up @@ -185,6 +185,7 @@ function runBatches(
CI: "true",
BUN_TEST_BATCH_SIZE: "3",
BUN_TEST_FILE_SCOPE: fileScope,
OCX_TEST_NO_QUEUE: "1",
OPENCODEX_BUN_PATH: join(binDirectory, "bun"),
FIXTURE_MODE: mode,
FIXTURE_CALLS: calls,
Expand All @@ -207,13 +208,16 @@ const batchCalls = (result: RunnerResult): string[] =>
result.calls.filter(call => !call.startsWith("1|"));
const singletonCalls = (result: RunnerResult): string[] =>
result.calls.filter(call => call.startsWith("1|"));
const noQueueFlags = (result: RunnerResult): string[] =>
result.calls.map(call => call.split("|")[1] ?? "");

describe.skipIf(process.platform !== "linux")("the Linux batch runner, executed", () => {
test("a clean run is green and runs each batch exactly once", () => {
const run = runBatches("green");
expect(`status:${run.status}`, run.output).toBe("status:0");
expect(batchCalls(run)).toHaveLength(2);
expect(singletonCalls(run)).toEqual([]);
expect(noQueueFlags(run)).toEqual(["1", "1"]);
expect(run.calls.some(call => call.includes(DEDICATED_FILE))).toBe(false);
}, SPAWN_BUDGET_MS);

Expand Down Expand Up @@ -249,6 +253,9 @@ describe.skipIf(process.platform !== "linux")("the Linux batch runner, executed"
// This is the exact case the deleted "a timeout may still recover" contract pinned green.
expect(`status:${run.status}`, run.output).toBe("status:124");
expect(singletonCalls(run)).toHaveLength(FIRST_BATCH.length);
// The bypass reaches both the primary process and every attribution process;
// otherwise a survivor from the failed process can queue the diagnostic sweep too.
expect(new Set(noQueueFlags(run))).toEqual(new Set(["1"]));
expect(run.output).toContain("every file passed alone");
expect(run.output).not.toContain("continuing");
}, SPAWN_BUDGET_MS);
Expand Down
4 changes: 4 additions & 0 deletions tests/ci-workflows/ci-workflows.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,10 @@ describe("GitHub Actions hardening", () => {
expect(windowsTest?.env?.BUN_TEST_FILE_SCOPE).toBe("all");
expect(windowsTest?.env?.BUN_TEST_BATCH_SIZE).toBe("6");
expect(windowsTest?.env?.BUN_TEST_BATCH_TIMEOUT_SECONDS).toBe("480");
// The 25 sequential Bun processes are one logical runner. On Windows the preload's
// machine-local queue can otherwise hold batch N+1 behind a straggler from batch N
// until the process bound fires without executing a test.
expect(windowsTest?.env?.OCX_TEST_NO_QUEUE).toBe("1");
expect(windowsTest?.run).toBe('bash scripts/ci/run-bun-test-batches.sh "$TEST_SHARD"');
expect(ci.jobs?.["platform-windows"]?.name).toBe(`windows \${{ matrix.shard }}/${windowsShards.length}`);
expect(workflow).toContain(`shards=${windowsShards.length}`);
Expand Down
27 changes: 22 additions & 5 deletions tests/ci-workflows/test-runner.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -773,6 +773,11 @@ describe("bun test argv", () => {
});

describe("bun test user lock", () => {
// Lock behavior must not inherit a workflow-level opt-out. The Windows batch leg
// intentionally sets OCX_TEST_NO_QUEUE for its outer processes, while these unit
// cases exercise the queued implementation itself.
const queuedTestEnv: NodeJS.ProcessEnv = {};

test("distinct POSIX users receive distinct temp-runtime locks", () => {
const common = { env: {}, tempDir: "/tmp", hostName: "builder-1", platform: "linux" as const };
const alice = resolveDefaultTestRunLockPath({
Expand Down Expand Up @@ -1163,8 +1168,9 @@ describe("bun test user lock", () => {
const root = mkdtempSync(join(tmpdir(), "opencodex-test-lock-"));
const lockPath = join(root, "suite.lock");
try {
const owner = await acquireTestRunLock({ runId: "suite-a", lockPath, pollMs: 5, maxWaitMs: 50 });
const sibling = await acquireTestRunLock({ runId: "suite-a", lockPath, pollMs: 5, maxWaitMs: 50 });
const options = { runId: "suite-a", lockPath, pollMs: 5, maxWaitMs: 50, env: queuedTestEnv };
const owner = await acquireTestRunLock(options);
const sibling = await acquireTestRunLock(options);
expect(owner.acquired).toBe(true);
expect(sibling.acquired).toBe(false);
sibling.release();
Expand All @@ -1180,12 +1186,15 @@ describe("bun test user lock", () => {
const root = mkdtempSync(join(tmpdir(), "opencodex-test-lock-"));
const lockPath = join(root, "suite.lock");
try {
const owner = await acquireTestRunLock({ runId: "wrapped", lockPath, pollMs: 5, maxWaitMs: 50 });
const owner = await acquireTestRunLock({
runId: "wrapped", lockPath, pollMs: 5, maxWaitMs: 50, env: queuedTestEnv,
});
expect(owner.owner).not.toBeNull();
const sibling = await acquireTestRunLock({
runId: "wrapped",
lockPath,
joinExistingOwnerToken: owner.owner!.token,
env: queuedTestEnv,
});
expect(sibling.acquired).toBe(false);
const wrongToken = owner.owner!.token === "57f44b0e-b750-4bd2-b23d-4a035e75da18"
Expand All @@ -1196,6 +1205,7 @@ describe("bun test user lock", () => {
runId: "wrapped",
lockPath,
joinExistingOwnerToken: wrongToken,
env: queuedTestEnv,
})).rejects.toThrow("refusing to create or reclaim");

owner.release();
Expand All @@ -1204,6 +1214,7 @@ describe("bun test user lock", () => {
runId: "wrapped",
lockPath,
joinExistingOwnerToken: owner.owner!.token,
env: queuedTestEnv,
})).rejects.toThrow("refusing to create or reclaim");
expect(existsSync(lockPath)).toBe(false);
} finally {
Expand All @@ -1221,8 +1232,11 @@ describe("bun test user lock", () => {
lockPath,
pollMs: 5,
maxWaitMs: 50,
env: queuedTestEnv,
});
const replacement = await acquireTestRunLock({
runId: "stale", lockPath, pollMs: 5, maxWaitMs: 50, env: queuedTestEnv,
});
const replacement = await acquireTestRunLock({ runId: "stale", lockPath, pollMs: 5, maxWaitMs: 50 });
expect(replacement.acquired).toBe(true);
stale.release();
expect(existsSync(lockPath)).toBe(true);
Expand All @@ -1237,13 +1251,16 @@ describe("bun test user lock", () => {
const root = mkdtempSync(join(tmpdir(), "opencodex-test-lock-"));
const lockPath = join(root, "suite.lock");
try {
const owner = await acquireTestRunLock({ runId: "live", lockPath, pollMs: 5, maxWaitMs: 50 });
const owner = await acquireTestRunLock({
runId: "live", lockPath, pollMs: 5, maxWaitMs: 50, env: queuedTestEnv,
});
let waits = 0;
await expect(acquireTestRunLock({
runId: "blocked",
lockPath,
pollMs: 5,
maxWaitMs: 20,
env: queuedTestEnv,
onWait: () => { waits += 1; },
})).rejects.toThrow("timed out");
expect(waits).toBe(1);
Expand Down
Loading