Skip to content

feat: verify runs only the selected journeys' Vitest tests - #171

Merged
inth3shadows merged 3 commits into
mainfrom
feat/122-vitest-verify
Nov 16, 2026
Merged

inth3shadows merged 3 commits into
mainfrom
feat/122-vitest-verify

Conversation

@inth3shadows

@inth3shadows inth3shadows commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Part of #122 (the Vitest half; Playwright is separate)

What

python3 -m testgraph.verify --repo . --base origin/main --runner vitest does for a Vitest suite what verify already does for pytest. It selects the journeys a diff endangers, runs only the tests that declare those journeys, and gives one exit code.

A Vitest test declares a journey with @tg:<id> in its own title or in a describe title. That is the same string a Playwright tag uses:

describe("checkout @tg:J4", () => {
  it("charges the card", async () => { /* ... */ });
});
it("submits a task @tg:J1 @tg:J7", () => { /* ... */ });

How

  • Selection: Vitest's -t, with the pattern (^|\s)(@tg:J1|@tg:J2)(\s|$). Vitest matches -t against the test's full name: the file, every describe title and the test title, joined by single spaces. Both boundaries are needed. Without the trailing one, @tg:J1 would also select @tg:J10. Without the leading one, x@tg:J1 would select J1.
  • Declarations: each tag must be a whole token of the describe chain or the title. Tags the registry does not define are ignored. Tests that -t filtered out come back from Vitest as skipped and declare nothing.
  • Validation: the same validate the pytest path uses. A tagged test's own V8 trace must reach one of the journey's entry symbols, or the declaration is reported unvalidated and the journey gets no credit.
  • Exit codes: the same as pytest's (0 / 1 / 2 / 3). One rule is added: a Vitest exit other than 0 with no failing journey counts as incomplete (3).
  • Code changes in vitest_adapter:
    • The traced run is split into locate and run_traced, so the adapter and verify share one path.
    • run now checks for the codegraph index before it looks for the plugin and node_modules. This only changes which message you see when both are missing.
  • The pytest path: unchanged, except that the --json summary gains "runner": "pytest".
  • Tried and dropped:
    • Vitest 4.1's native tags with --tags-filter. strictTags defaults to on, so every repo would have to list each tg:J* tag in its config, and it needs Vitest 4.1 or later. Titles work on any version with no config change.
    • Running the whole suite and keeping only the tagged tests. Simpler, but it defeats running only what the diff endangers.

Measured (base 510e94a vs this branch, Linux, Python 3.14 + pytest 9.0.3 / Python 3.11 without pytest, Node 22, Vitest 4.1.9)

  • New tests: 22 in tests/test_verify_vitest.py. Before the change, 21 of them error and the one that pins the unchanged pytest output passes. After the change, all 22 pass.
    • The 22 include 5 that run real Vitest on a small fixture project. One case has an untagged failing test next to a passing tagged one, and the journey still passes, which shows the untagged test did not run. Another case confirms a @tg:J10 test is not run for J1.
    • The real-Vitest tests skip where Node or a Vitest install is missing, as they do in CI.
  • Mutation check: I broke each guard one at a time and each break turned at least one test red (10 of 10). The guards are: the trailing boundary, the leading boundary, the selected-journey filter, whole-token matching, the exit rule, -t itself, setting declared, the collected count, the render branch and the CLI flag.
  • Real repo: signedintake (Next.js, 167 Vitest tests, no tags yet), with --dry-run --base HEAD~10. It selected 14 journeys and found 0 tagged tests, so it reports every journey as having no journey-level test, exit 3, in 6.8 s. The repo's working tree was unchanged afterwards.
  • Review: a reviewer ran about 80 real-Vitest scenarios against this commit.
    • A tag alone never credited a pass.
    • 20 regex metacharacters tried in journey ids all selected correctly under Vitest's new RegExp.
    • Tests and files that -t filtered out never caused a refusal.
    • A throwing hook or an import error in a tagged file still refuses (exit 2).
  • Full suite: Ran 686 tests ... OK on 3.14, and Ran 686 tests ... OK (skipped=4) on 3.11 (664 on base).
  • Compatibility: no format or index changes. The --json summary gains a runner key (additive).

Changes since review (2026-11-16)

88b7268 fixes five review findings, each test-first. Before the fix the new tests had 15 failures; after it, all of them pass.

Mutation check: I broke each fix one at a time and each break turned at least one test red. The one exception swaps $ for \Z in the tag regex, which can't change any result: the two differ only just before a final newline, and a newline is whitespace on both sides anyway.

Full suite: Ran 701 tests ... OK on 3.14 and OK (skipped=4) on 3.11. The signedintake dry run is unchanged (exit 3, working tree clean).

  • Duplicate titles: two skipped tests that share an id no longer refuse the run. A duplicate where either test ran is still refused.
  • Quiet runs: a Vitest run that selects nothing, or that the index guard refuses, now reports "runner": "vitest" and runner_exit.
  • Your own -t: a -t or --testNamePattern after -- is refused up front with the reason. Narrow the run with a file path instead.
  • Skipped tagged tests: a journey whose tagged tests were all skipped is now reported as incomplete (skip), not as having no tagged test. The "no journey-level test" message now says "no test in this run carries the tag".
  • Whitespace: declarations now match tags over the same full name Vitest's -t sees, file path included, using JavaScript's whitespace set. Selection and declaration agree on every separator character.

Changes since review round 2 (2026-11-16)

a0f1e6c fixes what the second review round found. The fixes were test-first: before them the new tests had 9 failures and 9 errors, and afterwards all 41 pass. The full suite passes on both Pythons: Ran 705 tests ... OK.

  • A tag in a file path no longer counts. Vitest matches -t against the describe chain and the test title only, never the file path; -t '^src' selects nothing. The round-1 whitespace fix had put the path into the name that tags are read from. So a file named src/x @tg:J1 y.test.ts declared J1, and could credit J1 from a test that was tagged only for J2.
    • Round 3 compared, on real Vitest, which tests -t selects with which tests are declared. There were 0 mismatches across nested and .each describes, empty names, a root config and a file outside src, plus 20,000 fuzzed names.
  • A journey whose tagged tests ran but missed its entry is no longer described as having no tagged test.
  • More spellings of the name filter are refused up front: --t, --t=x, and short flag clusters that contain t (-ut x, -tx).
  • Every declaration now reads the test's full name, and the list of journeys that ran but did not pass is in selection order.

Known limits

  • Two titles that differ only in / versus :: versus . collide on one join key and refuse the run. This predates this PR.
  • Some spellings of the name filter are still not refused up front: -t5, -t.x, ---t x and --no-t. Vitest then crashes, so the run is refused (exit 2) without the reason. --no-t may instead drop the journey filter, which ends in a refusal or an incomplete result (exit 3), never a pass.
  • --json still lists journeys whose tagged tests never reached the journey's entry functions under uncovered. Those journeys are also listed under unvalidated, and the text output tells the two cases apart.
  • The line saying unvalidated tests are "listed below" can name a journey that isn't shown, because the list below is capped at 8 entries.
  • Cleanups left for later:
    • The plugin directory is looked up twice on the Vitest path.
    • The runner's exit is kept under two keys (pytest_exit and runner_exit) instead of one.
    • Tag patterns are built once per call, not cached.
  • Every check from the Vitest adapter PR (feat: Vitest adapter — per-test V8 trace, same journey attribution #170) still applies: the .concurrent coverage window, anonymous functions not recorded, test.projects and workspace configs refused, no timeout on the Vitest subprocess.

…tests (#122)

A Vitest test declares a journey with `@tg:<id>` in its title or a describe
title. verify selects with `-t` (whole-tag boundaries, so J1 never selects
J10), and each tagged test's own V8 trace must reach the journey's entry
symbols, through the same `validate` the pytest path uses. A non-zero Vitest
exit with no failing journey is incomplete (exit 3).
… -t, skip wording, JS whitespace

- W1: two skipped tests sharing an id no longer refuse the run; a duplicate
  that ran still does. `-t` turns every deselected test.each repeat into one.
- W2: the refused and nothing-selected returns of a vitest run carry
  runner=vitest and runner_exit.
- W3: a user -t / --testNamePattern after `--` is refused up front with the
  reason, instead of crashing Vitest or replacing the journey filter.
- W4: a journey whose tagged tests were all skipped is incomplete (skip), not
  "no test carries the tag"; the uncovered message speaks of this run.
- W5: declarations match tags over the same full name Vitest's -t sees, with
  JavaScript's whitespace set, so selection and declaration cannot disagree.
… (F1), R2-1/2/4/7

- F1: Vitest matches -t against the describe chain and the title only. The
  W5 fix had put the file path into the name tags are read from, so a path
  token like `src/x @tg:J1 y.test.ts` declared J1 and could credit it.
- R2-1: a journey whose tagged tests ran but never reached its entry is no
  longer described as having no tagged test.
- R2-2: --t, --t=x and short clusters holding t (-ut x, -tx) are refused
  up front like -t.
- R2-4: declarations read only full_name; the test-id fallback missed a
  leading tag.
- R2-7: `incomplete` is built in one pass, in selection order.
@inth3shadows
inth3shadows merged commit 10ea764 into main Nov 16, 2026
6 checks passed
@inth3shadows
inth3shadows deleted the feat/122-vitest-verify branch November 16, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant