Skip to content

fix(testing): say how check-visual-parity runs a suite and what a run with no verdict means #10105

Closed

Description

@kyle-sexton

Parent: #7195 (split slice: a defect the first-consumer pass found, owner decision Q112 of 2026-10-25).

Problem

/testing:check-visual-parity route 1 says to run the repository's own visual suite and report its
verdict (plugins/testing/skills/check-visual-parity/SKILL.md:35-38). It does not say how to run
the suite, and it has no rule for a suite that ends without a verdict.

In the first-consumer pass against melodic-software/medley identity-server on native Windows, the
repository's runner (npm test, which moves non-Linux hosts into a Linux Playwright container)
failed twice before any screenshot was compared: once on a stale .NET SDK default, once on a
Playwright image older than the @playwright/test version in its lockfile. Each run exited
non-zero with no pixel result. Read literally, the skill reports that as the verdict for the five
states the suite covers. It does not say whether those states fall through to route 2 or stop with
a gap report.

The skill also does not say to use the repository's runner rather than the test tool directly. In
this repository a direct npx playwright test on Windows would compare against -win32 baselines
that do not exist (updateSnapshots: "none", Linux-only baselines). That is inferred from its
playwright.config.ts, not run. The runner avoids it by using Docker.

Fix

  • Route 1 runs the suite through the repository's own entry point (its documented test command or
    runner script) when one exists, and names that command in the report.
  • A suite run that ends before comparing (a build, install, browser-launch or missing-baseline
    error) is reported as an environment failure with its last lines, not as a verdict. The states it
    covers then take the gap report; they never fall through to another route, for the same reason
    compare never switches routes (two renderers).

Done when

  • Route 1 names the command to run and says what an environment failure is.
  • An eval case covers a suite whose run fails before comparing, and the expected report holds
    the environment failure and the gap report for its states.
  • scripts/check-changelog-fragments.sh passes.

Activity

  1. github-actions commented on Oct 25, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Understood: the check-visual-parity skill's route 1 should run the repository's own test entry point and name it in the report. A suite run that fails before comparing (build, install, browser-launch or missing-baseline error) should be reported as an environment failure with its last lines, not as a verdict. The states that suite covers then take the gap report and never fall through to another route. An eval case for that failure is also requested. The approach is clear and the done-when items can be checked in the lane. No likely duplicates found in the open issue list.

    Automated triage, applied by the lanes App.

  2. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    and removed on Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions