Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,19 @@ jobs:
- run: python3 -m unittest discover -s tests
env:
TG_REQUIRE_PYTEST: "1"

# The release builds with `uv build` (sdist first, then the wheel FROM the sdist), so
# anything the wheel needs must survive the sdist. A wheel-only build hid exactly that
# (#165 review): the trace plugin was force-included from a path the sdist dropped.
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
fetch-depth: 0 # hatch-vcs derives the version from tags
- uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
- run: uv build
- name: The wheel built from the sdist carries the trace plugin
run: |
python3 -m zipfile -l dist/*.whl | grep -q 'testgraph/_plugin/tgtrace.py' \
|| { echo "::error::wheel has no testgraph/_plugin/tgtrace.py"; exit 1; }
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,8 @@ traces/

# generated by hatch-vcs at build time; never committed
testgraph/_version.py

# The wheel-only copy of harness/plugin/tgtrace.py. A stray one in a checkout would be
# swept into the sdist and collide with the force-include (`uv build`: "A second file
# is being added"). Runtime lookup prefers harness/plugin either way.
testgraph/_plugin/
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,16 @@ below is on `main` or in the branch that introduced this file.

### Fixed

- **`verify` works from an installed wheel.** The wheel shipped `testgraph/` only, so
the README's headline `verify` command failed after `pip install testgraph` with
"cannot find harness/plugin/tgtrace.py". The wheel now carries the trace plugin as
`testgraph/_plugin/tgtrace.py` (a build-time copy; `harness/plugin/` stays the source).
- **`harness/accuracy.py` no longer reports PASS on a run that scored nothing** (#97).
A failed worktree, failed index or BLOCKED commit skipped scoring without lowering
`min_recall` (which starts at 1.0), so a machine without the bare repo printed
`S1 VERDICT : PASS` and exited 0. It now fails when no commit was scored or any errored,
and prints the scored count.

- **`verify --json` changed shape on its quiet exits** (#89). The refused and
nothing-selected paths built their own summary without `repo` and `commit`,
so a consumer aggregating runs hit a `KeyError` on exactly the runs most
Expand Down
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ pip install testgraph
```

No runtime dependencies — the stdlib `sqlite3` to read the CodeGraph index, and
`git` for the diff. The wheel ships `testgraph/` alone: the measurement harness
and the three dogfood registries live in the repo, not the package.
`git` for the diff. The wheel ships the `testgraph/` package and its trace plugin: the
measurement harness and the three dogfood registries live in the repo, not the package.

From source, if you want those too:

Expand Down Expand Up @@ -86,7 +86,8 @@ than none: every downstream check then reports its own mismatch as a stale index
`python3 -m testgraph.propose --repo <path>` drafts into (2) by default.

Note for anyone reading this on PyPI: (3) does not exist in an installed wheel,
which ships `testgraph/` alone. Put your registry in (2).
which ships the `testgraph/` package and its trace plugin, not `journeys/`. Put your
registry in (2).

## Prerequisites

Expand Down
3 changes: 2 additions & 1 deletion USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,8 @@ A journey with **no covering test** is named as loudly as a covered one. That is
the actionable half: on testgraph's own 389-test suite, J6 has zero covering
tests, and nothing but that line would say so.

This adapter needs a source checkout — `harness/` is not in the wheel.
The trace plugin it loads ships in the wheel (`testgraph/_plugin/tgtrace.py`), so this
works from `pip install testgraph` as well as from a checkout.

## Checking the Graph Against Reality

Expand Down
15 changes: 12 additions & 3 deletions harness/accuracy.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,19 @@ def main():
print(f" {mstr}")
mean_p = sum(precisions) / len(precisions) if precisions else float("nan")
print("\n--- summary ---")
print(f" MIN RECALL : {min_recall:.2f} (S1 requires 1.00)")
scored = sum(1 for r in rows if len(r) == 5)
# With nothing scored, `min_recall` is still its 1.0 seed: print n/a, not a perfect score.
print(f" MIN RECALL : {f'{min_recall:.2f}' if scored else 'n/a'} (S1 requires 1.00)")
print(f" mean precision (non-empty oracles): {mean_p:.2f}")
print(f" S1 VERDICT : {'PASS' if min_recall >= 1.0 else 'FAIL'}")
return 0 if min_recall >= 1.0 else 1
# #97: `min_recall` starts at 1.0 and error rows `continue` past it, so a run that
# scored NOTHING (every worktree failed) used to print PASS. A commit that errored or
# was BLOCKED produced no answer to score, which is not a pass either.
errored = [r for r in rows if len(r) == 3]
ok = scored > 0 and not errored and min_recall >= 1.0
print(f" commits scored: {scored} / {len(labels['commits'])}"
f" (errored or blocked: {len(errored)})")
print(f" S1 VERDICT : {'PASS' if ok else 'FAIL'}")
return 0 if ok else 1


if __name__ == "__main__":
Expand Down
11 changes: 9 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -53,12 +53,19 @@ source = "vcs"
[tool.hatch.build.hooks.vcs]
version-file = "testgraph/_version.py"

# The installable package is `testgraph/` alone. `harness/` (the measurement rig),
# The installable package is `testgraph/` (plus the trace plugin, below). `harness/` (the measurement rig),
# `hooks/`, `skills/`, `journeys/` and `maps/` are repo artifacts a user reads or
# installs from a checkout — shipping them in a wheel would imply an import contract
# none of them have.
[tool.hatch.build.targets.wheel]
packages = ["testgraph"]

# The one exception: `verify` needs the trace plugin at runtime, so the wheel carries
# that single file (see `pytest_adapter.plugin_dir`). `harness/plugin/tgtrace.py` stays
# the source of truth; nothing else from `harness/` ships. The sdist `include` below
# must list it too: `uv build` builds the wheel FROM the sdist.
[tool.hatch.build.targets.wheel.force-include]
"harness/plugin/tgtrace.py" = "testgraph/_plugin/tgtrace.py"

[tool.hatch.build.targets.sdist]
include = ["testgraph", "journeys", "skills", "hooks", "README.md", "USAGE.md", "TECHNICAL.md", "LICENSE"]
include = ["testgraph", "journeys", "skills", "hooks", "harness/plugin/tgtrace.py", "README.md", "USAGE.md", "TECHNICAL.md", "LICENSE"]
34 changes: 22 additions & 12 deletions testgraph/pytest_adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -236,18 +236,32 @@ def write_outcomes(repo_name, commit, attribution, append=None):
return written


PLUGIN_MISSING = ("cannot find the tgtrace plugin: an installed testgraph carries it at "
"testgraph/_plugin/tgtrace.py, a checkout at harness/plugin/tgtrace.py — "
"reinstall the package, or restore the file in the checkout")


def plugin_dir(start=None):
"""Where `tgtrace.py` lives, or None.

The installed wheel ships `testgraph/` ONLY — `harness/` is a repo
artifact with no import contract (the packaging decision recorded in
`testgraph-public-release.md`). So this resolves relative to the source
checkout and returns None rather than guessing when it is not there; the
caller turns that into one clear error instead of an opaque pytest
`-p tgtrace` import failure twenty lines into a subprocess log."""
`harness/` is a repo artifact with no import contract, so the wheel does not
ship it — but `verify` cannot run without the trace plugin, so the wheel
carries that ONE file as `testgraph/_plugin/tgtrace.py` (a build-time
`force-include` of `harness/plugin/tgtrace.py`; the source of truth stays in
`harness/`). The checkout's `harness/plugin` is checked FIRST, so a stray
`testgraph/_plugin/` copy in a checkout can never shadow the file being edited;
an installed package has no `harness/` beside it and finds `_plugin`. None when
neither exists, which the caller turns into one clear error instead of an
opaque pytest `-p tgtrace` import failure."""
root = start or os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
candidate = os.path.join(root, "harness", "plugin")
return candidate if os.path.isfile(os.path.join(candidate, "tgtrace.py")) else None
if os.path.isfile(os.path.join(candidate, "tgtrace.py")):
return candidate
if start is None:
packaged = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_plugin")
if os.path.isfile(os.path.join(packaged, "tgtrace.py")):
return packaged
return None


def run(repo, registry_path=None, db_path=None, pytest_args=(), commit=None,
Expand Down Expand Up @@ -275,11 +289,7 @@ def run(repo, registry_path=None, db_path=None, pytest_args=(), commit=None,

plugins = plugin_dir()
if plugins is None:
return None, (
"cannot find harness/plugin/tgtrace.py — it ships with the source "
"checkout, not the installed wheel, so this adapter needs to run "
"from a checkout"
)
return None, PLUGIN_MISSING

db_path = db_path or os.path.join(repo, ".codegraph", "codegraph.db")
if not os.path.isfile(db_path):
Expand Down
4 changes: 2 additions & 2 deletions testgraph/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ def load(path):
# Where a registry lives, in search order. See `search_dirs`.
#
# This used to be the package-relative directory ALONE, which works only in a
# source checkout and is silently useless anywhere else: the wheel ships
# `testgraph/` and nothing beside it, so on a `pip install testgraph` the path
# source checkout and is silently useless anywhere else: the wheel does not ship
# `journeys/`, so on a `pip install testgraph` the path
# resolves to `site-packages/journeys`, which does not exist. Every repo then
# answered "no journey registry found" forever — including through the MCP
# server, which would have returned that to every agent that ever called it.
Expand Down
3 changes: 1 addition & 2 deletions testgraph/verify.py
Original file line number Diff line number Diff line change
Expand Up @@ -171,8 +171,7 @@ def run(repo, base, head="HEAD", registry_path=None, db_path=None,
return None, f"no codegraph index at {db_path} — run `codegraph init` in {repo}"
plugins = pa.plugin_dir()
if plugins is None:
return None, ("cannot find harness/plugin/tgtrace.py — it ships with "
"the source checkout, not the installed wheel")
return None, pa.PLUGIN_MISSING

# Strict (the default): an unresolvable journey BLOCKs and the run is
# refused. `strict_registry=False` is reserved for history analysis
Expand Down
61 changes: 61 additions & 0 deletions tests/test_accuracy_harness.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
"""harness/accuracy.py verdict (#97): a run that scored nothing must not PASS."""
import os
import sys
import unittest
from unittest import mock

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)

from harness import _worktree, accuracy # noqa: E402


def _fail_add(*_a, **_k):
raise _worktree.WorktreeError("WORKTREE-FAIL", "no such bare repo")


class AccuracyVerdict(unittest.TestCase):
def _run(self, add_and_index, select=None):
with mock.patch.object(accuracy._worktree, "remove"), \
mock.patch.object(accuracy._worktree, "add_and_index", add_and_index), \
mock.patch.object(accuracy.sel, "select", select or mock.DEFAULT), \
mock.patch("builtins.print") as out:
rc = accuracy.main()
text = "\n".join(" ".join(map(str, c.args)) for c in out.call_args_list)
return rc, text

def test_every_worktree_failing_is_a_fail_not_a_pass(self):
rc, text = self._run(_fail_add)
self.assertEqual(rc, 1)
self.assertIn("S1 VERDICT : FAIL", text)
self.assertIn("commits scored: 0", text)
self.assertIn("MIN RECALL : n/a", text)

def test_one_blocked_commit_fails_the_run(self):
labels = accuracy.json.load(open(os.path.join(accuracy.HERE, "labels_honeyslate.json")))
first = labels["commits"][0]["sha"]

def select(wtdir, base, sha, db, reg, strict_registry):
if sha == first:
return {"status": "BLOCKED", "blocking": ["registry rot"]}
oracle = next(c["oracle"] for c in labels["commits"] if c["sha"] == sha)
return {"status": "OK", "journeys": [{"id": j} for j in oracle]}

rc, text = self._run(lambda *a, **k: "db", select)
self.assertEqual(rc, 1)
self.assertIn("errored or blocked: 1", text)

def test_perfect_recall_on_every_commit_passes(self):
labels = accuracy.json.load(open(os.path.join(accuracy.HERE, "labels_honeyslate.json")))

def select(wtdir, base, sha, db, reg, strict_registry):
oracle = next(c["oracle"] for c in labels["commits"] if c["sha"] == sha)
return {"status": "OK", "journeys": [{"id": j} for j in oracle]}

rc, text = self._run(lambda *a, **k: "db", select)
self.assertEqual(rc, 0, text)
self.assertIn("S1 VERDICT : PASS", text)


if __name__ == "__main__":
unittest.main()
61 changes: 61 additions & 0 deletions tests/test_plugin_packaging.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
"""The trace plugin ships in the wheel: `verify` failed after `pip install` (audit 2026-11-12)."""
import os
import sys
import tempfile
import tomllib
import unittest
from unittest import mock

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)

from testgraph import pytest_adapter as pa # noqa: E402


class PluginPackaging(unittest.TestCase):
def test_wheel_force_includes_the_plugin_from_its_source_of_truth(self):
with open(os.path.join(ROOT, "pyproject.toml"), "rb") as f:
conf = tomllib.load(f)
inc = conf["tool"]["hatch"]["build"]["targets"]["wheel"]["force-include"]
self.assertEqual(inc, {"harness/plugin/tgtrace.py": "testgraph/_plugin/tgtrace.py"})
self.assertTrue(os.path.isfile(os.path.join(ROOT, "harness/plugin/tgtrace.py")))

def test_an_installed_package_finds_the_packaged_plugin(self):
with tempfile.TemporaryDirectory() as site:
pkg = os.path.join(site, "testgraph")
os.makedirs(os.path.join(pkg, "_plugin"))
open(os.path.join(pkg, "_plugin", "tgtrace.py"), "w").close()
with mock.patch.object(pa, "__file__", os.path.join(pkg, "pytest_adapter.py")):
self.assertEqual(pa.plugin_dir(), os.path.join(pkg, "_plugin"))

def test_a_checkout_uses_harness_plugin(self):
self.assertEqual(pa.plugin_dir(), os.path.join(ROOT, "harness", "plugin"))

def test_a_stray_packaged_copy_in_a_checkout_does_not_shadow_the_source(self):
with tempfile.TemporaryDirectory() as repo:
for d in ("harness/plugin", "testgraph/_plugin"):
os.makedirs(os.path.join(repo, d))
open(os.path.join(repo, d, "tgtrace.py"), "w").close()
fake = os.path.join(repo, "testgraph", "pytest_adapter.py")
with mock.patch.object(pa, "__file__", fake):
self.assertEqual(pa.plugin_dir(), os.path.join(repo, "harness", "plugin"))

def test_the_sdist_carries_every_file_the_wheel_force_includes(self):
# `uv build` builds the wheel FROM the sdist (#165 review, B1).
with open(os.path.join(ROOT, "pyproject.toml"), "rb") as f:
targets = tomllib.load(f)["tool"]["hatch"]["build"]["targets"]
include = targets["sdist"]["include"]
for src in targets["wheel"]["force-include"]:
self.assertTrue(any(src == i or src.startswith(i.rstrip("/") + "/")
for i in include), src)

def test_neither_present_is_none_not_a_guess(self):
with tempfile.TemporaryDirectory() as site:
pkg = os.path.join(site, "testgraph")
os.makedirs(pkg)
with mock.patch.object(pa, "__file__", os.path.join(pkg, "pytest_adapter.py")):
self.assertIsNone(pa.plugin_dir())


if __name__ == "__main__":
unittest.main()
4 changes: 2 additions & 2 deletions tests/test_registries.py
Original file line number Diff line number Diff line change
Expand Up @@ -213,8 +213,8 @@ class RegistryDiscoveryTests(unittest.TestCase):
"""Where `resolve_for_repo` looks, and why the old answer was unusable.

The search directory used to be package-relative and NOTHING else. That works
in a source checkout and is silently useless anywhere else: the wheel ships
`testgraph/` alone, so on `pip install testgraph` the path resolves to
in a source checkout and is silently useless anywhere else: the wheel does not
ship `journeys/`, so on `pip install testgraph` the path resolves to
`site-packages/journeys`, which does not exist. Every repo answered "no
journey registry found" forever — including through the MCP server, which
would have returned that to every agent that ever called it. Measured on a
Expand Down
Loading