Repository navigation
fix(testing): say how check-visual-parity runs a suite and what a run with no verdict means #10105
Description
Parent: #7195 (split slice: a defect the first-consumer pass found, owner decision Q112 of 2026-10-25).
Problem
/testing:check-visual-parity route 1 says to run the repository's own visual suite and report its
verdict (plugins/testing/skills/check-visual-parity/SKILL.md:35-38). It does not say how to run
the suite, and it has no rule for a suite that ends without a verdict.
In the first-consumer pass against melodic-software/medley identity-server on native Windows, the
repository's runner (npm test, which moves non-Linux hosts into a Linux Playwright container)
failed twice before any screenshot was compared: once on a stale .NET SDK default, once on a
Playwright image older than the @playwright/test version in its lockfile. Each run exited
non-zero with no pixel result. Read literally, the skill reports that as the verdict for the five
states the suite covers. It does not say whether those states fall through to route 2 or stop with
a gap report.
The skill also does not say to use the repository's runner rather than the test tool directly. In
this repository a direct npx playwright test on Windows would compare against -win32 baselines
that do not exist (updateSnapshots: "none", Linux-only baselines). That is inferred from its
playwright.config.ts, not run. The runner avoids it by using Docker.
Fix
- Route 1 runs the suite through the repository's own entry point (its documented test command or
runner script) when one exists, and names that command in the report. - A suite run that ends before comparing (a build, install, browser-launch or missing-baseline
error) is reported as an environment failure with its last lines, not as a verdict. The states it
covers then take the gap report; they never fall through to another route, for the same reason
compare never switches routes (two renderers).
Done when
- Route 1 names the command to run and says what an environment failure is.
- An eval case covers a suite whose run fails before comparing, and the expected report holds
the environment failure and the gap report for its states. -
scripts/check-changelog-fragments.shpasses.
Activity
github-actions commentedon Oct 25, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: the check-visual-parity skill's route 1 should run the repository's own test entry point and name it in the report. A suite run that fails before comparing (build, install, browser-launch or missing-baseline error) should be reported as an environment failure with its last lines, not as a verdict. The states that suite covers then take the gap report and never fall through to another route. An eval case for that failure is also requested. The approach is clear and the done-when items can be checked in the lane. No likely duplicates found in the open issue list.
Automated triage, applied by the lanes App.