Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,21 @@ below is on `main` or in the branch that introduced this file.

### Added

- **`testgraph.vitest_adapter`** (#122): the pytest adapter's journey attribution for
Vitest suites. A TypeScript setup file (`tgtrace-vitest.ts`, shipped beside
`tgtrace.py`) records each test's named functions through V8 precise coverage;
Vitest's JSON reporter supplies verdicts; the join, the attribution and the ledger
rows are the pytest adapter's own. Injected through a generated config in a temp
dir, so nothing is written into the target repo (Vitest's own `node_modules/.vite`
cache aside). A broken or partial run is a refusal (exit 2), never a verdict: a
passed or failed test with no trace (a dead worker, a hook that threw), a
never-finished test, a duplicate test id, a file that failed outside any test
(import error, throwing `beforeAll`/`afterAll`), a malformed report, and a
`test.projects`/workspace config (projects do not inherit `setupFiles`). The trace
is recorded in `onTestFinished`, so a repo `afterEach` that throws no longer loses
it; the repo is resolved through `realpath`; the repo's own `root` is kept; coverage
is forced off for the run. On a real Next.js repo (167 Vitest tests) it attributes
5 journeys with no unjoined test.
- **`testgraph.reconcile`** — two-sided reconciliation of the static graph
against a real test run, classifying every disagreement as a **STATIC MISS**
(a symbol the journey's tests executed, outside its static footprint) or a
Expand Down
32 changes: 32 additions & 0 deletions USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,38 @@ tests, and nothing but that line would say so.
The trace plugin it loads ships in the wheel (`testgraph/_plugin/tgtrace.py`), so this
works from `pip install testgraph` as well as from a checkout.

### Vitest suites

The same attribution for a TypeScript repo whose tests run on Vitest (4.x):

```bash
python3 -m testgraph.vitest_adapter --repo . --dry-run
python3 -m testgraph.vitest_adapter --repo . --json -- src/lib # vitest's own args after --
```

It needs the repo's dependencies installed (`node_modules` with `vitest`, at the repo or
a workspace root). Nothing is written into the repo: a generated config in a temp dir
imports the repo's own `vitest.config.*` (or `vite.config.*`) and adds the
`tgtrace-vitest.ts` setup file, which records each test's named functions through V8
precise coverage. Rows carry `edge_provenance=trace`, like the pytest adapter's. (Vitest
still writes its own cache to `node_modules/.vite`, as any `vitest run` does.)

A run that cannot be read is refused (exit 2), never recorded: any passed or failed test
without a trace, a test that never finished (killed worker), two tests with the same id
(same title in one describe, e.g. a repeated `test.each` row), a file that failed outside
any test (import error, throwing `beforeAll`/`afterAll`; Vitest reports the tests under
a throwing `beforeAll` as skipped, so only the file's status shows it), and a report of
the wrong shape. An all-skipped run, or a `-t` filter that matches nothing, is an empty
success.

Limits: `test.projects` and `vitest.workspace.*` configs are not supported yet (projects
do not inherit `setupFiles`, so nothing would be traced) and are refused up front;
anonymous functions are not recorded (attribution matches named entry symbols, so this
costs nothing there); tests marked `.concurrent` in one file share a coverage window,
so their traces can bleed into each other; a failing test next to tests skipped by a
throwing `beforeAll` in the same file is not distinguishable from `it.skip`; `verify`
does not run Vitest suites yet.

## Checking the Graph Against Reality

`reconcile` puts CodeGraph's edges on trial, using the same run:
Expand Down
139 changes: 139 additions & 0 deletions harness/plugin/tgtrace-vitest.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
/**
* Vitest setup file: record every NAMED function that actually ran, per test.
*
* The Vitest counterpart of `tgtrace.py` (#122). It is loaded into the TARGET
* repo's test workers through a generated config (see
* `testgraph/vitest_adapter.py`), so it imports nothing from testgraph and only
* `vitest` plus Node built-ins.
*
* How: V8 precise coverage through `node:inspector`. Each `takePreciseCoverage`
* returns call counts since the previous take and resets them, so a take in
* `beforeEach` clears the slate and a take in `afterEach` is exactly what the
* test body (plus its hooks) ran. Function granularity (`detailed: false`) is
* enough for attribution, which matches journey ENTRY symbols by name.
*
* Configured by environment, like tgtrace.py:
* TGTRACE_OUT_DIR directory for one part file per test file (required, else inert)
* TGTRACE_ROOT only record functions defined under this directory
* TGTRACE_SKIP colon-separated path fragments to ignore
* (default: /node_modules/ .test. .spec. /__tests__/)
*
* Part file: {"root", "backend", "file", "duplicates", "tests": {"<id>": [["<relpath>", "<name>"], ...]}}
* where <id> is `<relfile>::<describe> > ... > <title>`, the same id the
* adapter builds from Vitest's JSON reporter.
*
* Known limits: anonymous functions are dropped (nothing to resolve them by
* without source maps); tests marked `.concurrent` in one file share one
* coverage window, so their traces can bleed into each other.
*/
import { Session } from "node:inspector/promises";
import { closeSync, fsyncSync, mkdirSync, openSync, realpathSync, renameSync, rmSync, writeSync } from "node:fs";
import { createHash } from "node:crypto";
import { join, relative, sep } from "node:path";
import { fileURLToPath } from "node:url";
import { afterAll, beforeEach, onTestFinished } from "vitest";

const OUT_DIR = process.env.TGTRACE_OUT_DIR;
// Real path: V8 script URLs and Vitest's test file paths are symlink-resolved, so
// a root given through a symlink would never prefix-match them.
const ROOT = (() => {
const root = process.env.TGTRACE_ROOT ?? process.cwd();
try {
return realpathSync(root);
} catch {
return root;
}
})();
const SKIP = (process.env.TGTRACE_SKIP ?? "/node_modules/:.test.:.spec.:/__tests__/")
.split(":")
.filter(Boolean);

type Hit = [string, string];
type Task = { id: string; name: string; type?: string; suite?: Task; file?: { filepath?: string; name: string } };

function testId(task: Task): string {
const filepath = task.file?.filepath ?? task.file?.name ?? "";
const names: string[] = [task.name];
for (let s = task.suite; s && s.type === "suite" && s.name !== task.file?.name; s = s.suite) {
names.unshift(s.name); // empty names stay: the JSON reporter's ancestorTitles keeps them
}
return `${relative(ROOT, filepath).split(sep).join("/")}::${names.join(" > ")}`;
}

function scriptPath(url: string): string | null {
if (!url.startsWith("file://")) return null;
try {
return fileURLToPath(url.split("?")[0]);
} catch {
return null;
}
}

if (OUT_DIR) {
const session = new Session();
session.connect();
await session.post("Profiler.enable");
await session.post("Profiler.startPreciseCoverage", { callCount: true, detailed: false });

const tests: Record<string, Hit[]> = {};
let file = "";
const owner = new Map<string, string>(); // test id -> task id, to spot two tests sharing an id
const duplicates = new Set<string>();

beforeEach(async (ctx) => {
await session.post("Profiler.takePreciseCoverage"); // reset: drop setup-time calls
// Not afterEach: Vitest runs afterEach hooks in reverse order and a throwing
// one stops the chain, so ours (registered first) would be skipped and a
// failing test would lose its trace. onTestFinished runs after every
// afterEach whether or not they threw.
onTestFinished(async () => {
await record(ctx.task as unknown as Task);
});
});

async function record(task: Task): Promise<void> {
const { result } = await session.post("Profiler.takePreciseCoverage");
file = task.file?.filepath ?? file;
const hits = new Map<string, Hit>();
for (const script of result) {
const abs = scriptPath(script.url);
if (!abs || !abs.startsWith(ROOT + sep)) continue;
const rel = relative(ROOT, abs).split(sep).join("/");
if (SKIP.some((frag) => ("/" + rel).includes(frag))) continue;
for (const fn of script.functions) {
const count = fn.ranges[0]?.count ?? 0;
if (count > 0 && fn.functionName) hits.set(`${rel}\0${fn.functionName}`, [rel, fn.functionName]);
}
}
const id = testId(task);
const prior = owner.get(id);
if (prior !== undefined && prior !== task.id) duplicates.add(id);
owner.set(id, task.id);
tests[id] = [...hits.values()].sort();
}

afterAll(async () => {
await session.post("Profiler.stopPreciseCoverage");
session.disconnect();
mkdirSync(OUT_DIR, { recursive: true });
const name = createHash("sha1").update(file).digest("hex").slice(0, 16);
const target = join(OUT_DIR, `${name}.json`);
const tmp = `${target}.tmp.${process.pid}`;
const payload = { root: ROOT, backend: "v8-precise-coverage", file: relative(ROOT, file), tests, duplicates: [...duplicates].sort() };
// Whole or absent, as tgtrace.py writes: a worker killed mid-write must
// not leave a truncated part for the adapter to merge.
try {
const fd = openSync(tmp, "w");
try {
writeSync(fd, JSON.stringify(payload, null, 1));
fsyncSync(fd);
} finally {
closeSync(fd);
}
renameSync(tmp, target);
} catch (err) {
rmSync(tmp, { force: true });
throw err;
}
});
}
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ packages = ["testgraph"]
# must list it too: `uv build` builds the wheel FROM the sdist.
[tool.hatch.build.targets.wheel.force-include]
"harness/plugin/tgtrace.py" = "testgraph/_plugin/tgtrace.py"
"harness/plugin/tgtrace-vitest.ts" = "testgraph/_plugin/tgtrace-vitest.ts"

[tool.hatch.build.targets.sdist]
include = ["testgraph", "journeys", "skills", "hooks", "harness/plugin/tgtrace.py", "README.md", "USAGE.md", "TECHNICAL.md", "LICENSE"]
include = ["testgraph", "journeys", "skills", "hooks", "harness/plugin/tgtrace.py", "harness/plugin/tgtrace-vitest.ts", "README.md", "USAGE.md", "TECHNICAL.md", "LICENSE"]
8 changes: 6 additions & 2 deletions testgraph/pytest_adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -542,9 +542,13 @@ def render(summary, registry=None):
gap is the actionable half — measured on testgraph itself, J6 (the harness
CLIs) has zero covering tests in a 359-test suite, and nothing but this
line would say so."""
# One renderer for every runner's adapter (vitest_adapter sets `runner`
# and `runner_exit`); pytest summaries keep their `pytest_exit` key.
runner = summary.get("runner", "pytest")
rc = summary.get("runner_exit", summary.get("pytest_exit"))
lines = [
f"testgraph pytest-adapter[{summary['repo']}] @ {summary['commit'][:9]} "
f"(pytest exit {summary['pytest_exit']}, {summary['backend']})"
f"testgraph {runner}-adapter[{summary['repo']}] @ {summary['commit'][:9]} "
f"({runner} exit {rc}, {summary['backend']})"
]
covered = summary["journeys"]
names = {}
Expand Down
Loading
Loading