Accuracy testing for the code graphs AI agents query. graph-assay finds wrong and missing edges in a code-graph tool by checking its graph against the compiler, helps turn each confirmed finding into a minimal repro for an upstream issue (minimizing is a manual step today), and scores tools on a published spec (SPEC.md).
A code-graph tool (an MCP server or CLI that answers "who calls this?", "what breaks if I change this?") is only as good as its edges. A call linked to the wrong definition makes every answer built on it confidently wrong, and these defects rarely show up in the tool's own test suite.
Status. SPEC v5.0 is in force. Two tools are measured: codegraph and code-graph-rag. The compiler oracle covers TypeScript/JavaScript, Python, Go and the JVM (Java and Kotlin). The fixture corpus holds 45 codegraph repros (44 name the upstream issue or PR they reproduce, 42 distinct), plus 4 for code-graph-rag. Adapters for codebase-memory-mcp and code-review-graph replay the fixtures (codebase-memory-mcp also has 2 repros of its own; code-review-graph none yet) but have not had an oracle run yet. Benchmark tables are not published yet: each tool's maintainers see their results first.
Intent. graph-assay is built for tool maintainers first. Run it on a pull request, replay the fixtures before a release, and file what it finds. The tables are the reporting layer, not the product. It stays neutral: no tool is scored against itself, every subject goes through the same adapter contract, and the scoring rules are versioned.
Every check compares a tool's graph with something independent and reports where they disagree:
- Compiler oracle (
oracle/): the tool's call edges against a SCIP index of the same checkout, produced by the language's real type checker (scip-typescript, scip-python, scip-go, scip-java). Each disagreement is joined at the exact call site and sorted by cause, for example "linked a project method where the compiler resolves a library method". Missing-edge mode runs it the other way: calls the compiler resolves that the tool never links. - Fixtures (
fixtures/): minimal repros of confirmed bugs, each with anexpect.jsonnaming the wrong edge and the right one. Replaying them against a build says which bugs it still has. - Metamorphic checks (
metamorphic/): change a repo in a way that must not change its graph (add decoy files nothing imports, shift every line down, index twice) and diff the graphs. - Differentials (
differential/): the same source under each execution mode a tool documents as equivalent (native vs WASM, worker counts, incremental sync), two builds of a tool on a pinned corpus, and the agent-facing answers (callers,explore) of two builds. - Benchmarks and tables (
benchmarks/,bench/): the PyCG micro-benchmark, plus tables that score call sites and (caller, callee) pairs with confidence intervals and an oracle-noise estimate from a blind-labelled sample (labels/). - Triage (
bench/triage.ts): ranks a tool's bug shapes by how many headline points they cost, and matches them to fixtures already filed.
Tools plug in through one adapter per subject (lib/subjects/). The adapter indexes a directory in
a sandbox and exports a graph snapshot (schema/snapshot.v1.json). The scorers read only
snapshots, so a new tool needs a new adapter, not new scorers.
- Node 22.18 to 24, not 23.0–23.5 (
lib/is TypeScript run by Node's type stripping, unflagged from 22.18 and 23.6; the codegraph adapter usesnode:sqlite; codegraph does not build on Node 25+) andgit. - Python 3 for code-graph-rag (3.12 on x86_64 gets the hash-locked install), Go on PATH for Go repos,
Docker for
scip-java.node bin/graph-assay setupinstalls the rest. - Docker, for the pinned end-to-end reproduction (
Dockerfile,tools/repro.ts).
Everything runs through bin/graph-assay (bin/graph-assay help lists the commands; bin/cgqa is a
deprecated alias).
- Check graph-assay itself (what CI runs; no tool build needed):
npm ci --ignore-scripts(the type checker, a dev dependency), thennode bin/graph-assay check. - Install the SCIP indexers and build both tools at their pinned versions, where graph-assay looks
for them:
node bin/graph-assay setup(--dry-runshows the steps;--codegraph <commit>builds another commit;--no-cgrand friends skip parts). - Replay the fixtures:
node bin/graph-assay replay default. Each fixture prints BROKEN, FIXED, GUESSED or CHANGED next to its upstream issue number. - Run the oracle on any repo you have checked out:
node bin/graph-assay oracle <name> <path to repo> <ts|python|go|jvm>. AddCGQA_SUBJECT=code-graph-ragto measure the other tool (environment variables keep the prefix from the project's old name, codegraph-qa). - Rank what it found:
node bin/graph-assay triage. To explain one call site, usenode bin/graph-assay site <name> <file:line>.
node bin/graph-assay config prints every resolved path. Each can be overridden by an environment
variable or ~/.config/graph-assay/config.json.
| Path | Purpose |
|---|---|
bin/graph-assay |
The command-line entry point |
SPEC.md |
Scoring semantics: units, tracks, equivalences, verdicts, failure policy |
lib/ |
Shared code: paths (config.ts), the snapshot contract, one adapter per tool (subjects/), SCIP reading, fixture expectations, run manifests, adapter conformance |
schema/ |
JSON Schemas for snapshots and fixture expectations |
oracle/ |
The compiler oracle, per-language SCIP wrappers, and self-tests with a mutation pass (oracle/test/) |
fixtures/ |
Bug repros: oracle-candidates/ (codegraph), cgr-candidates/ (code-graph-rag), cbm-candidates/ (codebase-memory-mcp) |
metamorphic/, differential/ |
Graph-must-not-change checks, execution-mode and build-vs-build diffs |
corpus/ |
The preregistered public corpus: pinned commits, licenses, dev / held-out split |
benchmarks/, bench/ |
Third-party benchmark scorers, tables, robustness and triage |
labels/ |
Blind labelling of call sites for the oracle-noise estimate |
conformance/ |
Awkward-case sources for the adapter conformance check |
docker/, Dockerfile, tools/ |
Pinned reproduction, and build/setup scripts |
- Usage Guide: what each workflow is for and how to read the results
- Technical Reference: architecture, every file, configuration, reproduction, limits
- SPEC.md: what is scored and how
- CHANGELOG.md, SECURITY.md (graph-assay indexes untrusted code; read before running it on a repo), AGENTS.md (for coding agents)