Repository navigation
feat: verify runs only the selected journeys' Vitest tests - #171
Merged
Merged
Conversation
…tests (#122) A Vitest test declares a journey with `@tg:<id>` in its title or a describe title. verify selects with `-t` (whole-tag boundaries, so J1 never selects J10), and each tagged test's own V8 trace must reach the journey's entry symbols, through the same `validate` the pytest path uses. A non-zero Vitest exit with no failing journey is incomplete (exit 3).
… -t, skip wording, JS whitespace - W1: two skipped tests sharing an id no longer refuse the run; a duplicate that ran still does. `-t` turns every deselected test.each repeat into one. - W2: the refused and nothing-selected returns of a vitest run carry runner=vitest and runner_exit. - W3: a user -t / --testNamePattern after `--` is refused up front with the reason, instead of crashing Vitest or replacing the journey filter. - W4: a journey whose tagged tests were all skipped is incomplete (skip), not "no test carries the tag"; the uncovered message speaks of this run. - W5: declarations match tags over the same full name Vitest's -t sees, with JavaScript's whitespace set, so selection and declaration cannot disagree.
… (F1), R2-1/2/4/7 - F1: Vitest matches -t against the describe chain and the title only. The W5 fix had put the file path into the name tags are read from, so a path token like `src/x @tg:J1 y.test.ts` declared J1 and could credit it. - R2-1: a journey whose tagged tests ran but never reached its entry is no longer described as having no tagged test. - R2-2: --t, --t=x and short clusters holding t (-ut x, -tx) are refused up front like -t. - R2-4: declarations read only full_name; the test-id fallback missed a leading tag. - R2-7: `incomplete` is built in one pass, in selection order.
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #122 (the Vitest half; Playwright is separate)
What
python3 -m testgraph.verify --repo . --base origin/main --runner vitestdoes for a Vitest suite whatverifyalready does for pytest. It selects the journeys a diff endangers, runs only the tests that declare those journeys, and gives one exit code.A Vitest test declares a journey with
@tg:<id>in its own title or in a describe title. That is the same string a Playwright tag uses:How
-t, with the pattern(^|\s)(@tg:J1|@tg:J2)(\s|$). Vitest matches-tagainst the test's full name: the file, every describe title and the test title, joined by single spaces. Both boundaries are needed. Without the trailing one,@tg:J1would also select@tg:J10. Without the leading one,x@tg:J1would select J1.-tfiltered out come back from Vitest as skipped and declare nothing.validatethe pytest path uses. A tagged test's own V8 trace must reach one of the journey's entry symbols, or the declaration is reportedunvalidatedand the journey gets no credit.vitest_adapter:locateandrun_traced, so the adapter andverifyshare one path.runnow checks for the codegraph index before it looks for the plugin andnode_modules. This only changes which message you see when both are missing.--jsonsummary gains"runner": "pytest".tagswith--tags-filter.strictTagsdefaults to on, so every repo would have to list eachtg:J*tag in its config, and it needs Vitest 4.1 or later. Titles work on any version with no config change.Measured (base
510e94avs this branch, Linux, Python 3.14 + pytest 9.0.3 / Python 3.11 without pytest, Node 22, Vitest 4.1.9)tests/test_verify_vitest.py. Before the change, 21 of them error and the one that pins the unchanged pytest output passes. After the change, all 22 pass.@tg:J10test is not run for J1.-titself, settingdeclared, thecollectedcount, the render branch and the CLI flag.--dry-run --base HEAD~10. It selected 14 journeys and found 0 tagged tests, so it reports every journey as having no journey-level test, exit 3, in 6.8 s. The repo's working tree was unchanged afterwards.new RegExp.-tfiltered out never caused a refusal.Ran 686 tests ... OKon 3.14, andRan 686 tests ... OK (skipped=4)on 3.11 (664 on base).--jsonsummary gains arunnerkey (additive).Changes since review (2026-11-16)
88b7268fixes five review findings, each test-first. Before the fix the new tests had 15 failures; after it, all of them pass.Mutation check: I broke each fix one at a time and each break turned at least one test red. The one exception swaps
$for\Zin the tag regex, which can't change any result: the two differ only just before a final newline, and a newline is whitespace on both sides anyway.Full suite:
Ran 701 tests ... OKon 3.14 andOK (skipped=4)on 3.11. The signedintake dry run is unchanged (exit 3, working tree clean)."runner": "vitest"andrunner_exit.-t: a-tor--testNamePatternafter--is refused up front with the reason. Narrow the run with a file path instead.-tsees, file path included, using JavaScript's whitespace set. Selection and declaration agree on every separator character.Changes since review round 2 (2026-11-16)
a0f1e6cfixes what the second review round found. The fixes were test-first: before them the new tests had 9 failures and 9 errors, and afterwards all 41 pass. The full suite passes on both Pythons:Ran 705 tests ... OK.-tagainst the describe chain and the test title only, never the file path;-t '^src'selects nothing. The round-1 whitespace fix had put the path into the name that tags are read from. So a file namedsrc/x @tg:J1 y.test.tsdeclared J1, and could credit J1 from a test that was tagged only for J2.-tselects with which tests are declared. There were 0 mismatches across nested and.eachdescribes, empty names, arootconfig and a file outsidesrc, plus 20,000 fuzzed names.--t,--t=x, and short flag clusters that containt(-ut x,-tx).Known limits
/versus::versus.collide on one join key and refuse the run. This predates this PR.-t5,-t.x,---t xand--no-t. Vitest then crashes, so the run is refused (exit 2) without the reason.--no-tmay instead drop the journey filter, which ends in a refusal or an incomplete result (exit 3), never a pass.--jsonstill lists journeys whose tagged tests never reached the journey's entry functions underuncovered. Those journeys are also listed underunvalidated, and the text output tells the two cases apart.pytest_exitandrunner_exit) instead of one..concurrentcoverage window, anonymous functions not recorded,test.projectsand workspace configs refused, no timeout on the Vitest subprocess.