Repository navigation
Description
What happens
These are the low-severity findings left open after two review rounds of #173 (verify --runner playwright). None of them gives a false pass: exit 0 still comes only from a first-attempt pass of every selected journey. The worst ones give the wrong exit code or a misleading message.
Checked on main at a8b7df0, with Playwright 1.60. Each item is marked verified (reproduced against real Playwright or the code) or inferred (read in the source).
Checklist
- A NUL byte or a lone surrogate in
e2e.cwdore2e.configcrashes with exit 1 (verified).- Input:
"cwd": "frontend\u0000x"or"cwd": "frontend/\ud800". - Result: a
ValueErrororUnicodeEncodeErrortraceback, and exit 1, the "tests failed" code. - Expected: a refusal (exit 2). Catch the error in the containment check, or reject the string up front.
- Input:
- A mistyped spec path after
--reads as "uncovered" (exit 3), not a refusal (inferred).- Cause:
--pass-with-no-testshides Playwright's "No tests found… matching test files" location error. - An empty
testDirbehaves the same way. - Fix: pass the flag only when no location arguments are given, or treat only the bare "No tests found" as uncovered.
- Cause:
- A symlink plus
..inconfigcan load a config outside the repo (verified).- Setup: the block is
{"cwd": "frontend/fx1", "config": "lnk/../../../../../x.js"}, wherelnkis an in-repo symlink pointing deeper into the repo. - The containment check resolves the symlink and passes, but Playwright resolves
-clexically and loads a config outside the repo. - Fix: normalise the path before the check, as
cwdalready is.
- Setup: the block is
-
-uchangedis refused (verified). Its value contains ac, which the short-flag guard reads as-c. - A run stopped by
-xormaxFailures:- Journeys that never started are shown as "(skipped)".
- The stop is detected by matching Playwright's English, ANSI-coloured message. A rewording would quietly turn red journeys back into refusals. Reading
config.maxFailuresandstats.unexpectedwould not depend on the wording. - When the stop comes with a second global error (a teardown that throws), the refusal names the stop instead of the real cause.
- A journey with a
test.fail()failure and a skipped test reads as a skip. The condition's first branch is redundant;'expected-not-passed' in statusesalone is enough. - The flaky line contradicts itself: "flaky — passed only on a retry" sits inside a line that ends "none passed".
- Multi-project configs count one spec once per project, in the collected count,
per_journeyand the ledger note. - A report with the right JSON syntax but the wrong shape crashes.
suites: nullandtags: nullraise instead of refusing, and{}reads as an empty run. - An interrupted run (SIGINT) reports every test skipped with no errors, which comes out as exit 3 rather than a refusal.
- Cleanups:
run_e2erepeats about 20 lines ofrun's preamble, and the two copies have already drifted on check order.- The tag pattern and the owned-flag guard are written separately from Vitest's (Vitest verify: leftover review items (name-filter spellings, journey classification) #172).
check_configandowned_flagrun twice per call.parse_reporthas a filter that can never remove anything.
- Tests and docs:
- The real-Playwright tests default to a local install path and skip quietly without one.
- Untagged flaky tests refuse the run, which is stricter than needed.
- Refusal text carries raw ANSI codes.
- The
test_e2e.pydocstring points at TECHNICAL.md for the real-run record, but TECHNICAL.md doesn't have it.
Related: #173 (where these were found), #156 (the registry e2e runner config these path checks guard; that issue is about invoking a provided environment, not validating the block), #172 (the matching Vitest leftovers; the classification and flag-guard items overlap), #157 (backend tracing, which would turn these verdicts from declared into trace-validated)
Activity