Skip to content

feat: Vitest adapter — per-test V8 trace, same journey attribution - #170

Merged
inth3shadows merged 2 commits into
mainfrom
feat/122-vitest-adapter
Nov 16, 2026
Merged

inth3shadows merged 2 commits into
mainfrom
feat/122-vitest-adapter

Conversation

@inth3shadows

@inth3shadows inth3shadows commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Part of #122 (the Vitest half; Playwright is separate)

What

python3 -m testgraph.vitest_adapter --repo <path> runs a Vitest suite once and records which journeys its tests actually ran. It does what pytest_adapter does for pytest suites: the same attribution against journey entry symbols, the same ledger rows tagged edge_provenance=trace, and the same --dry-run, --json and --trace-out options.

How

  • Recording what ran: harness/plugin/tgtrace-vitest.ts is a Vitest setup file written in TypeScript. It ships next to tgtrace.py and imports only vitest and Node built-ins. For each test it collects V8 precise coverage through node:inspector, resetting the counters before the test and reading them in onTestFinished. Then it writes one part file per test file, atomically.

  • Verdicts: these come from Vitest's JSON reporter. A test's id is <relfile>::<describe> > ... > <title>, built the same way on both sides.

  • Reused code: attribute, journey_entries, write_outcomes and render are shared with the pytest adapter unchanged. render now shows which runner it reports for.

  • Injection, with nothing written into the target repo: Vitest has no --setupFiles flag. So a generated config in a temp dir imports the repo's own vitest.config.* (or vite.config.*), whether it exports an object, a function or a Promise, and merges the setup file in with mergeConfig. The temp dir's node_modules is a real directory of symlinks to the repo's own entries, so Vite's config bundle lands in the temp dir. Vitest's normal node_modules/.vite cache is still written, as with any vitest run.

  • What it refuses (exit 2): any broken or partial run.

    • no Vitest install, or a missing, truncated or malformed report
    • a pass or fail verdict with no trace (a worker was killed, or the repo's afterAll threw)
    • a test that never finished
    • a duplicate test id: one id for two tests can't say which verdict goes with which trace
    • a file that failed outside any test (an import error, a throwing beforeAll)
    • a test.projects config or workspace file: projects don't inherit setupFiles

    A run where every test is skipped, or a -t filter matches nothing, is an empty successful run, not a refusal.

  • Other settings: coverage is turned off on the command line, because the repo's own V8 coverage would fight over the same counters. The repo's root is left alone.

  • Tried and dropped:

    • afterEach as the collector: Vitest runs afterEach hooks in reverse order and stops at the first one that throws, so a failing test lost its trace.
    • --configLoader runner: it doesn't define __dirname, which real configs use.
    • Forcing root in the generated config: it changed which tests ran.
    • Vitest's coverage-v8 provider: it aggregates over the whole run, not per test.
    • A custom reporter: reporters run in the main process and can't reach the workers.

Measured (base fc3d000 vs this branch, Linux, Python 3.14 + pytest 9.0.3 / Python 3.11 without pytest, Node 22.23.5, Vitest 4.1.9)

  • Real repo: signedintake (Next.js, 167 Vitest tests), run with --dry-run. Journeys J1 (18 tests), J2 (5), J6 (9), J10 (12) and J13 (17) all pass, with Vitest exiting 0. Every traced test joined to its verdict, both ways. The other 9 journeys are pages and components with no unit test reaching them. The repo's working tree was unchanged afterwards (git status).
  • Full suite: Ran 664 tests ... OK on 3.14, and Ran 664 tests ... OK (skipped=4) on 3.11 (571 on base). A real-Vitest test class builds small fixture projects and runs actual Vitest. It runs when Node and a Vitest install are available (TG_VITEST_NODE_MODULES) and skips cleanly otherwise. CI has no Node, so CI runs only the hermetic tests.
  • Review fixes: a dual review of the first commit found 12 defects (V1–V12; the reviewer agent reproduced them on real Vitest fixtures). The new tests fail against the first commit's code (failures=15, errors=25). Mutation check: I reverted each fix one at a time, and each revert turned at least one test red (17 of 17 checks, including all three sides of the duplicate-id fix and the extra pending refusal).
  • Type-check: tsc --noEmit --strict on the setup file exits 0.
  • Compatibility: pytest adapter output is unchanged apart from the renderer's runner label. No format changes.

Known limits

  • verify doesn't run Vitest suites yet: no running only the selected journeys' tests, and no @tg: declarations. That's the next step.
  • Anonymous functions aren't recorded. Attribution matches named entry symbols, so it loses nothing there, but reconcile would need source maps for a full footprint.
  • .concurrent tests in one file share a coverage window, so their traces can bleed into each other or lose calls. No warning is shown.
  • A failing test next to tests skipped by a throwing beforeAll in the same file can't be told apart from it.skip through the reporter. USAGE.md documents this.
  • Projects and workspace configs are refused rather than supported.
  • The design duplicates code: run and main repeat about 120 lines of pytest_adapter's. Sharing one runner-neutral run is a follow-up (review V13).
  • No timeout on the Vitest subprocess, the same as the pytest adapter.

)

testgraph.vitest_adapter runs a Vitest suite once and attributes journeys
the way pytest_adapter does. harness/plugin/tgtrace-vitest.ts (TypeScript,
shipped beside tgtrace.py) takes V8 precise coverage per test and writes
one atomic part file per test file; Vitest's JSON reporter gives verdicts;
attribute / journey_entries / write_outcomes are reused unchanged.

The setup file is injected through a generated config in a temp dir that
imports the repo's own config and mergeConfigs it in, with a node_modules
symlink so vitest resolves as the repo's tests see it. Nothing is written
into the target repo. Refusals (no vitest install, no or truncated report,
a broken part, tests run but no trace) exit 2.

pytest_adapter.render labels the runner, so both adapters share it.
A broken or partial run is a refusal (exit 2), never a verdict.

V1  a passed/failed test with no trace refuses (count + example id); a
    reporter "pending" test (killed worker) refuses instead of reading as skip
V2  trace recorded in onTestFinished (from beforeEach), which still fires when
    the repo's afterEach throws; V1 backstops it
V3  duplicate test ids refuse (report side and setup-file side). Chosen over
    worst-status/union: with one id for two tests the join cannot say which
    verdict belongs to which trace, so any merge is a guess
V4  a file that failed outside any test (import error, throwing beforeAll)
    refuses by name; the reporter marks beforeAll-skipped tests "skipped"
V5  realpath for cwd, TGTRACE_ROOT and the id root (Python and TS)
V6  the generated config no longer forces `root`
V7  all-skipped and no-match runs are empty successes; test.projects and
    vitest.workspace.* are refused up front with their real cause
V8  --coverage.enabled=false
V9  empty describe names kept in the TS id, as ancestorTitles keeps them
V10 temp node_modules is a dir of per-entry symlinks so Vite's .vite-temp stays
    in the temp dir (--configLoader runner rejected: no __dirname)
V11 report shape validated, refusal instead of a crash
V12 default-export config awaited whether function result or Promise

Adds a real-Vitest integration class (TG_VITEST_NODE_MODULES) covering the
above; `execute` factored out of run() so it can be driven directly.
@inth3shadows
inth3shadows merged commit 510e94a into main Nov 16, 2026
6 checks passed
@inth3shadows
inth3shadows deleted the feat/122-vitest-adapter branch November 16, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant