Skip to content

chore: standardize on bun test, fix CI lockfile drift, refresh testing docs - #10

Merged
queso merged 4 commits into
mainfrom
chore/bun-test-migration
Aug 9, 2026
Merged

queso merged 4 commits into
mainfrom
chore/bun-test-migration

Conversation

@queso

@queso queso commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Why

Two runner problems and one CI problem, all tangled together.

The test runner was split across two tools. package.json ran vitest; CI ran bun test. Only vitest read vitest.config.ts, so the isolation settings meant to keep the browser tests stable never applied in CI — and bun run test hung locally on those same tests.

CI could not read the lockfile. bun-version was pinned to 1.1.42, which predates the text bun.lock format:

error: Unknown lockfile version
InvalidLockfileVersion: failed to parse lockfile: 'bun.lock'
warn: Ignoring lockfile

CI discarded the lockfile and resolved every range to latest. The pin was added for reproducibility but produced the opposite. @biomejs/biome: "^2.3.13" installed as 2.5.7, whose formatter disagrees with 2.3.13 about it(name, opts, fn) — failing lint on a file nobody had touched. This is what is currently breaking #9, and it is unrelated to that PR's contents.

What changed

Test runner — standardize on bun's built-in runner, the one that was already passing:

  • test → bun test --timeout 30000, test:coverage likewise
  • Drop vitest and @vitest/coverage-v8; delete vitest.config.ts
  • CI calls bun run test so the timeout lives in one place

Test files keep importing from "vitest" — bun maps that specifier to bun:test natively, so no install is needed and 19 files stay untouched.

CI — bun-version → 1.3.11 (matches local, reads bun.lock), and bun install --frozen-lockfile so drift fails loudly instead of silently floating.

docs/testing.md — had drifted from the code since PRD-0001. It documented result.passed (the field is success) and a string error (it is a structured FlowError), invented markup for all three HTML fixtures, and omitted wait_for and assertion retry entirely. It also described test/fixtures/flows/ as specs that run, when they are only ever fed through FlowSpecSchema. Rewritten against the actual code, with fixture contents described rather than duplicated so they stop drifting.

flowspec init --dir bug — with no value it silently scaffolded the current directory. Found it when a test run wrote flowspec.config.yaml and specs/ into this repo: init-dir-flag.test.ts spawned that case with cwd defaulting to the project root and asserted only that an exit code was defined, so it passed while scaffolding the repo. The CLI now errors, and also rejects --dir --verbose, which previously created a directory literally named --verbose. --help is checked first so it still works when --dir is malformed.

Verification

  • bun run test — 307 pass / 0 fail across 19 files
  • bun run typecheck, bun run lint — clean
  • Working tree stays clean across a full run (no more stray flowspec.config.yaml / specs/)
  • flowspec init --dir with no value: exit 1, nothing created; --help and valid --dir unaffected

Note

Once this merges, #9 should stop failing lint — it is hitting the floating-biome issue fixed here, not a problem with its own changes.

queso added 2 commits August 9, 2026 02:41
The test runner was split between two tools: package.json ran vitest while
CI ran `bun test`. Only vitest read vitest.config.ts, so the isolation
settings meant to keep the browser tests stable never applied in CI, and
`bun run test` hung locally on those same tests.

Standardize on bun's built-in runner, which was already the one passing:

- package.json: test -> `bun test --timeout 30000`, test:coverage likewise
- Drop vitest and @vitest/coverage-v8; delete vitest.config.ts
- CI calls `bun run test` so the timeout is defined in one place

Test files keep importing from "vitest" — bun maps that specifier to
bun:test natively, so no install is needed and 19 files stay untouched.

config.test.ts asserted the vitest dep and vitest.config.ts existed, which
went red once both were removed. Replaced with an assertion that both test
scripts use `bun test` with an explicit timeout — the invariant that
matters now, since bun's 5s default is too short for the browser tests.

docs/testing.md had drifted from the code since PRD-0001. It documented
result.passed (the field is `success`) and a string `error` (it is a
structured FlowError), invented markup for all three HTML fixtures, and
omitted wait_for and assertion retry entirely. It also described
test/fixtures/flows/ as specs that run, when they are only ever fed
through FlowSpecSchema. Rewritten against the actual code, with fixture
contents described rather than duplicated so they stop drifting.

Separately, `flowspec init --dir` with no value silently scaffolded the
current directory. Found it when a test run wrote flowspec.config.yaml and
specs/ into this repo: init-dir-flag.test.ts spawned that case with cwd
defaulting to the project root, and asserted only that an exit code was
defined, so it passed while scaffolding the repo. The CLI now errors, and
also rejects `--dir --verbose`, which previously created a directory named
"--verbose". --help is checked first so it still works when --dir is
malformed. The test runs from a temp dir and asserts the real behavior.
CI pinned bun 1.1.42, which predates the text bun.lock format. Every run
hit:

    error: Unknown lockfile version
    InvalidLockfileVersion: failed to parse lockfile: 'bun.lock'
    warn: Ignoring lockfile

so CI discarded the lockfile and resolved every range to latest. The pin
was added for reproducibility but produced the opposite: @biomejs/biome
"^2.3.13" installed as 2.5.7, whose formatter disagrees with 2.3.13 about
`it(name, opts, fn)`, failing lint on a file nobody had touched.

Bump to 1.3.11 to match local, and use --frozen-lockfile so a lockfile
that drifts from package.json fails loudly instead of silently floating.
@coderabbitai

coderabbitai Bot commented Aug 9, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

@queso, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 35 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f91f4c23-8d22-408e-aaa3-ac88e582ed1d

📥 Commits

Reviewing files that changed from the base of the PR and between 844c5b2 and 7d2798e.

⛔ Files ignored due to path filters (1)
  • bun.lock is excluded by !**/*.lock
📒 Files selected for processing (8)
  • .github/workflows/ci.yml
  • CLAUDE.md
  • docs/testing.md
  • package.json
  • src/index.ts
  • test/config.test.ts
  • test/init-dir-flag.test.ts
  • vitest.config.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

queso added 2 commits August 9, 2026 03:25
With the lockfile actually honored, every browser test failed:

    browserType.launch: Executable doesn't exist at
      .../ms-playwright/chromium_headless_shell-1208/...

while the install step had just downloaded build 1234.

The install step ran `bunx agent-browser install`, which resolves the
latest release from the registry — currently 0.33.2 — but package.json
pins agent-browser to 0.9.3, and the runner always launches the local
copy via node_modules/.bin (src/runner.ts:81). 0.9.3 carries
playwright-core 1.58.1, which looks for Chromium 1208; 0.33.2 installs
1234.

The two were never guaranteed to agree. It worked by accident because CI
could not read bun.lock, so playwright-core floated to latest alongside
whatever bunx fetched. Pinning the lockfile removed the coincidence.

Install browsers with ./node_modules/.bin/agent-browser so the build
downloaded is the one the runner launches.
Pinning agent-browser to 0.9.3 did not fix the build mismatch — the
pinned copy installs Chromium 1234 too, while launch still looks for
1208. The disagreement is inside agent-browser: `install` runs through a
native shim carrying its own newer Playwright, but launch goes through
the hoisted playwright-core (1.58.1), whose browsers.json pins
chromium-headless-shell to revision 1208.

Install through playwright-core's own CLI instead. It resolves the exact
revision the runner will launch:

    $ node node_modules/playwright-core/cli.js install --dry-run chromium
    Chrome Headless Shell (playwright chromium-headless-shell v1208)

so install and runtime cannot drift apart again regardless of what
agent-browser's shim bundles.
@queso
queso merged commit 8c47e56 into main Aug 9, 2026
2 checks passed
@queso
queso deleted the chore/bun-test-migration branch August 9, 2026 03:44
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 0.2.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant