Skip to content

Playwright verify: leftover review items (bad paths, stopped runs, report shapes) #174

Open

Description

@inth3shadows

What happens

These are the low-severity findings left open after two review rounds of #173 (verify --runner playwright). None of them gives a false pass: exit 0 still comes only from a first-attempt pass of every selected journey. The worst ones give the wrong exit code or a misleading message.

Checked on main at a8b7df0, with Playwright 1.60. Each item is marked verified (reproduced against real Playwright or the code) or inferred (read in the source).

Checklist

  • A NUL byte or a lone surrogate in e2e.cwd or e2e.config crashes with exit 1 (verified).
    • Input: "cwd": "frontend\u0000x" or "cwd": "frontend/\ud800".
    • Result: a ValueError or UnicodeEncodeError traceback, and exit 1, the "tests failed" code.
    • Expected: a refusal (exit 2). Catch the error in the containment check, or reject the string up front.
  • A mistyped spec path after -- reads as "uncovered" (exit 3), not a refusal (inferred).
    • Cause: --pass-with-no-tests hides Playwright's "No tests found… matching test files" location error.
    • An empty testDir behaves the same way.
    • Fix: pass the flag only when no location arguments are given, or treat only the bare "No tests found" as uncovered.
  • A symlink plus .. in config can load a config outside the repo (verified).
    • Setup: the block is {"cwd": "frontend/fx1", "config": "lnk/../../../../../x.js"}, where lnk is an in-repo symlink pointing deeper into the repo.
    • The containment check resolves the symlink and passes, but Playwright resolves -c lexically and loads a config outside the repo.
    • Fix: normalise the path before the check, as cwd already is.
  • -uchanged is refused (verified). Its value contains a c, which the short-flag guard reads as -c.
  • A run stopped by -x or maxFailures:
    • Journeys that never started are shown as "(skipped)".
    • The stop is detected by matching Playwright's English, ANSI-coloured message. A rewording would quietly turn red journeys back into refusals. Reading config.maxFailures and stats.unexpected would not depend on the wording.
    • When the stop comes with a second global error (a teardown that throws), the refusal names the stop instead of the real cause.
  • A journey with a test.fail() failure and a skipped test reads as a skip. The condition's first branch is redundant; 'expected-not-passed' in statuses alone is enough.
  • The flaky line contradicts itself: "flaky — passed only on a retry" sits inside a line that ends "none passed".
  • Multi-project configs count one spec once per project, in the collected count, per_journey and the ledger note.
  • A report with the right JSON syntax but the wrong shape crashes. suites: null and tags: null raise instead of refusing, and {} reads as an empty run.
  • An interrupted run (SIGINT) reports every test skipped with no errors, which comes out as exit 3 rather than a refusal.
  • Cleanups:
  • Tests and docs:
    • The real-Playwright tests default to a local install path and skip quietly without one.
    • Untagged flaky tests refuse the run, which is stricter than needed.
    • Refusal text carries raw ANSI codes.
    • The test_e2e.py docstring points at TECHNICAL.md for the real-run record, but TECHNICAL.md doesn't have it.

Related: #173 (where these were found), #156 (the registry e2e runner config these path checks guard; that issue is about invoking a provided environment, not validating the block), #172 (the matching Vitest leftovers; the classification and flag-guard items overlap), #157 (backend tracing, which would turn these verdicts from declared into trace-validated)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions