Repository navigation
feat: verify runs the selected journeys' Playwright tests (declared) - #173
Merged
Merged
Conversation
…rowser tests (#122) The registry's `e2e` block names where the Playwright suite lives ({"runner": "playwright", "cwd": ..., "config": ...}). verify selects with a case-sensitive `--grep '/(@tg:J1|...)(\s|$)/'`, reads the JSON report, and credits each selected journey. Verdicts are DECLARED, not trace-validated (edge_provenance=e2e-declared; backend tracing is #157), and the summary says so. - fail (1): a tagged test failed, timed out or crashed, also when -x or maxFailures stopped the run after it. - incomplete (3): flaky (passed only on a retry), skip-only, a test.fail() test that failed as declared, no tagged test (uncovered; --pass-with-no-tests keeps Playwright's "No tests found" from reading as an error), or a non-zero runner exit with nothing failed. - refused (2): a test outside the selected journeys failed (setup/environment), other global report errors, no or unreadable report, a malformed e2e block, a cwd or config that resolves outside the repo, or a user --grep, --reporter or --config after `--` (verify owns them). `--e2e` from the checkpoint became `--runner playwright`, sharing the runner_exit rule and quiet-run summary keys with the Vitest path.
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #122 (the Playwright half; the Vitest half is #170 and #171)
What
python3 -m testgraph.verify --repo . --base origin/main --runner playwrightruns the Playwright tests tagged for the journeys a diff endangers, and gives one exit code.The registry says where the suite lives:
A test declares a journey with Playwright's own tag,
@tg:<id>, either as an option or in its title:The verdicts are declared, not trace-validated. Nothing traces the backend during a browser run yet, so a tagged test that mocks the app's own API would pass the same way. Rows carry
edge_provenance=e2e-declared, and the summary prints that on every run. Backend tracing is #157.How
npx --no-install playwright test -c <config> --grep '/(@tg:J1|@tg:J2)(\s|$)/' --reporter=json --pass-with-no-tests, run incwd. The report goes to a temp file.@tg:J1from selecting@tg:J10.@…run in a title as a tag (/@[\S]+/), sox@tg:J1really does carry@tg:J1./…/form makes the match case-sensitive. A bare--grepis compiled with the flagsgi, so it would also run@tg:j1.-xormaxFailuresstopped early after that failure.test.fail()test failed as it declarede2eblock, or acwdorconfigthat resolves outside the repo--grep,--reporteror--configafter--, since verify sets those itself--runner playwrightgoes through the same entry point as--runner vitest, the same runner-exit rule and the same summary keys for quiet runs.--e2eflag (the first draft, never released). One--runnerchoice is simpler.--grep. It ignores case and lets a failure tagged@tg:j1disappear.Measured (base
10ea764vs this branch, Linux, Python 3.14 + pytest 9.0.3 / Python 3.11 without pytest, Node 22, Playwright 1.60)tests/test_e2e.pyhas 47 tests. They run against recorded report shapes, plus six that run real Playwright on tests that need no browser; those six skip where Node or a Playwright install is missing, as in CI.failures=26, errors=32; afterwards all 47 pass.d6ed34a~1..d6ed34a.create_taskraising, the same run exits 1.@tg:J1,,(@tg:J1),x@tg:J1.Ran 752 tests ... OKon 3.14, andRan 752 tests ... OK (skipped=4)on 3.11 (705 on base).--jsonsummary for a quiet Playwright run carriesrunnerandrunner_exit, as Vitest's does.Known limits
cwdorconfigcrashes with exit 1 instead of being refused. This only happens with an escape written by hand in the registry JSON.--now reads as "uncovered" (exit 3) instead of a refusal, because--pass-with-no-testshides Playwright's location error. An emptytestDirbehaves the same way...inconfigcan point Playwright at a config outside the repo. The containment check resolves symlinks, but Playwright resolves-clexically.-uchangedis refused, because its value contains ac.--update-snapshots=changedworks.-xstop are shown as "(skipped)". Detecting the stop relies on Playwright's English error message. When the stop comes with a second global error, the refusal names the stop, not the other error.test.fail()failure and one skipped test reads as a skip.suites: nullraises instead of refusing, and an empty{}reads as an empty run.run_e2erepeats about 20 lines ofrun's preamble.