Skip to content

feat: verify runs the selected journeys' Playwright tests (declared) - #173

Merged
inth3shadows merged 1 commit into
mainfrom
feat/122-playwright-verify
Nov 16, 2026
Merged

inth3shadows merged 1 commit into
mainfrom
feat/122-playwright-verify

Conversation

@inth3shadows

Copy link
Copy Markdown
Owner

Part of #122 (the Playwright half; the Vitest half is #170 and #171)

What

python3 -m testgraph.verify --repo . --base origin/main --runner playwright runs the Playwright tests tagged for the journeys a diff endangers, and gives one exit code.

The registry says where the suite lives:

"e2e": {"runner": "playwright", "cwd": "frontend", "config": "playwright.real.config.js"}

A test declares a journey with Playwright's own tag, @tg:<id>, either as an option or in its title:

test("a submitter adds a task", { tag: "@tg:J1" }, async ({ page }) => { /* ... */ });

The verdicts are declared, not trace-validated. Nothing traces the backend during a browser run yet, so a tagged test that mocks the app's own API would pass the same way. Rows carry edge_provenance=e2e-declared, and the summary prints that on every run. Backend tracing is #157.

How

  • Selection: npx --no-install playwright test -c <config> --grep '/(@tg:J1|@tg:J2)(\s|$)/' --reporter=json --pass-with-no-tests, run in cwd. The report goes to a temp file.
    • The trailing boundary keeps @tg:J1 from selecting @tg:J10.
    • No leading boundary is needed. Playwright reads any @… run in a title as a tag (/@[\S]+/), so x@tg:J1 really does carry @tg:J1.
    • The /…/ form makes the match case-sensitive. A bare --grep is compiled with the flags gi, so it would also run @tg:j1.
  • Verdicts per selected journey:
    • fail (exit 1): a tagged test failed, timed out or crashed. This also covers a run that -x or maxFailures stopped early after that failure.
    • incomplete (exit 3):
      • a tagged test passed only on a retry (flaky)
      • every tagged test was skipped
      • a test.fail() test failed as it declared
      • no test carries the tag (uncovered)
      • Playwright exited non-zero with nothing failing
    • refused (exit 2):
      • a test outside the selected journeys failed, tagged or not. This is the login or seed setup project, a broken environment rather than a verdict about any journey.
      • any other global error in the report
      • no report, or an unreadable one
      • a malformed e2e block, or a cwd or config that resolves outside the repo
      • your own --grep, --reporter or --config after --, since verify sets those itself
  • Sharing with Vitest: --runner playwright goes through the same entry point as --runner vitest, the same runner-exit rule and the same summary keys for quiet runs.
  • Tried and dropped:
    • A separate --e2e flag (the first draft, never released). One --runner choice is simpler.
    • A bare --grep. It ignores case and lets a failure tagged @tg:j1 disappear.
    • Treating "No tests found" as a refusal. It refused every diff whose selected journeys had no browser test yet.

Measured (base 10ea764 vs this branch, Linux, Python 3.14 + pytest 9.0.3 / Python 3.11 without pytest, Node 22, Playwright 1.60)

  • Tests: tests/test_e2e.py has 47 tests. They run against recorded report shapes, plus six that run real Playwright on tests that need no browser; those six skip where Node or a Playwright install is missing, as in CI.
    • The review fixes were test-first. Before the fixes, the new tests gave failures=26, errors=32; afterwards all 47 pass.
    • Mutation check: I reverted each fix one at a time, and each revert turned at least one test red (18 of 18).
  • Real repo: honeyslate (FastAPI + SvelteKit), using its real-backend J1 suite: a throwaway Postgres on a local port, the real API, and a login through the real magic-link endpoint. It runs on a branch in that repo that isn't pushed yet. The verify range was d6ed34a~1..d6ed34a.
    • Green: J1 passes (declared). The other 5 selected journeys have no browser test, so the run exits 3.
    • With create_task raising, the same run exits 1.
    • honeyslate's working tree was unchanged afterwards.
  • Review: two review rounds, both reviewers running real Playwright, about 50 flag spellings and 20 journey ids containing regex metacharacters.
    • Exit 0 only ever came from a first-attempt pass of every selected journey.
    • The tests the filter selects and the tags the report gives agreed on every spelling tried: option tags, title tags, describe tags, @tg:J1,, (@tg:J1), x@tg:J1.
  • Full suite: Ran 752 tests ... OK on 3.14, and Ran 752 tests ... OK (skipped=4) on 3.11 (705 on base).
  • Compatibility: pytest and Vitest verify are unchanged. The --json summary for a quiet Playwright run carries runner and runner_exit, as Vitest's does.

Known limits

  • A NUL byte or a lone surrogate in cwd or config crashes with exit 1 instead of being refused. This only happens with an escape written by hand in the registry JSON.
  • A mistyped spec path after -- now reads as "uncovered" (exit 3) instead of a refusal, because --pass-with-no-tests hides Playwright's location error. An empty testDir behaves the same way.
  • An in-repo symlink combined with .. in config can point Playwright at a config outside the repo. The containment check resolves symlinks, but Playwright resolves -c lexically.
  • -uchanged is refused, because its value contains a c. --update-snapshots=changed works.
  • Journeys cut off by an -x stop are shown as "(skipped)". Detecting the stop relies on Playwright's English error message. When the stop comes with a second global error, the refusal names the stop, not the other error.
  • A journey with one test.fail() failure and one skipped test reads as a skip.
  • The flaky line contradicts itself. It sits inside a line that ends "none passed".
  • Multi-project configs count one spec once per project.
  • Some malformed reports crash. A report with suites: null raises instead of refusing, and an empty {} reads as an empty run.
  • An interrupted run (SIGINT) exits 3 rather than being refused.
  • Duplicate code:
    • run_e2e repeats about 20 lines of run's preamble.
    • The tag pattern and the flag guard are written separately from Vitest's.
    • The block checks run twice per call.
  • The real-Playwright tests default to a local install path, and skip quietly without one.

…rowser tests (#122)

The registry's `e2e` block names where the Playwright suite lives
({"runner": "playwright", "cwd": ..., "config": ...}). verify selects with a
case-sensitive `--grep '/(@tg:J1|...)(\s|$)/'`, reads the JSON report, and
credits each selected journey. Verdicts are DECLARED, not trace-validated
(edge_provenance=e2e-declared; backend tracing is #157), and the summary says so.

- fail (1): a tagged test failed, timed out or crashed, also when -x or
  maxFailures stopped the run after it.
- incomplete (3): flaky (passed only on a retry), skip-only, a test.fail()
  test that failed as declared, no tagged test (uncovered; --pass-with-no-tests
  keeps Playwright's "No tests found" from reading as an error), or a
  non-zero runner exit with nothing failed.
- refused (2): a test outside the selected journeys failed (setup/environment),
  other global report errors, no or unreadable report, a malformed e2e block,
  a cwd or config that resolves outside the repo, or a user --grep, --reporter
  or --config after `--` (verify owns them).

`--e2e` from the checkpoint became `--runner playwright`, sharing the
runner_exit rule and quiet-run summary keys with the Vitest path.
@inth3shadows
inth3shadows merged commit a8b7df0 into main Nov 16, 2026
6 checks passed
@inth3shadows
inth3shadows deleted the feat/122-playwright-verify branch November 16, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant