Skip to content

[Handoff] Compiler-owned graph snapshot campaign, after PR #164 #167

Open
Open
[Handoff] Compiler-owned graph snapshot campaign, after PR #164#167

Description

@samchon
samchon
opened on Sep 1, 2026

Purpose

Hand the compiler-owned graph snapshot campaign to a fresh issue campaign after
PR #164 merged as 1bf9486. This body is the operative handoff: it states what
exists at master, what was measured on it, what is verified broken, and what
each remaining route needs, so a fresh agent can start without reading the
campaign conversation.

Supersedes #166, whose stated purpose ? adjudicate PR #164 ? is discharged.

Exact stop state

What exists at master

Graph Snapshot Protocol v1

packages/graph/src/provider/GraphSnapshotProtocol.ts is the shared contract
every strict provider commits through. Frames are hello, begin,
upsertShard, deleteShard, commit. A generation is content-addressed and
carries a manifest digest over the inputs it was computed from and a fact digest
over what it publishes. Store.commit refuses a generation whose assembly does
not reproduce the commit frame's digest, and a delta that cannot be served
raises a reload rather than publishing a graph nobody has.

Files a new route will touch:

File Role
provider/GraphSnapshotProtocol.ts frames, Store, fold, factDigest, proven
provider/canonicalFactText.ts the one canonical serializer both digest modules use
provider/graphSnapshotDigests.ts manifestOf and contentOf
provider/assertGraphSnapshotContract.ts the boundary check a provider's validate runs
utils/freezeDeep.ts seals a published generation in place
indexer/buildLspGraph.ts strict-provider selection, then the generic per-language lane
indexer/LANGUAGE_SPECS.ts extension-to-language map used for enumeration and the generic/static lanes

Three existing adapters are the worked examples, in ascending size:
provider/rust/RustGraphSnapshotAdapter.ts (1,018 lines),
provider/java/JavaGraphSnapshotAdapter.ts (1,081) and
provider/cpp/CppGraphSnapshotAdapter.ts (2,198).

Registered providers

packages/graph/src/provider/GRAPH_PROVIDERS.ts registers twelve. The generated
manifest docs/provider-support.json is the machine-checked source of truth and
pnpm provider-support fails CI on drift between it, the registry and the
README.

Provider Languages Authority Defensible edge families
samchon-rust-analyzer-hir rust analyzer 14
javac-graph java compiler 14
clangd-snapshot c, cpp compiler 12
samchon-graph-go go compiler 11
ttscgraph typescript compiler 9
scip-kotlinc kotlin semantic-index 2
samchon-graph-lua lua analyzer 1
scip-python python semantic-index 1
scip-dotnet csharp semantic-index 0
scip-ruby ruby semantic-index 0
scip-dart dart semantic-index 0
scip-php php semantic-index 0

The manifest states those limits in its own words: scip-dotnet "proves
declarations but no graph edge family", scip-ruby "proves no graph edge
family", scip-python "proves references only".

scip-clang and scip-java are deliberately not in that list. They are
filtered out of GRAPH_PROVIDERS because each is owned by a strict route as its
fallback tier, and one language cannot have two owners; cppGraphProvider looks
the scip-clang descriptor up by name and fails closed if it is missing. So a
servedBy naming scip-clang or scip-java always means the strict route
above it failed.

C/C++ route

clangd-snapshot is served by packages/graph/src/provider/cpp/
(CppGraphSnapshotAdapter.ts, CppGraphClient.ts, cppGraphProvider.ts). It
answers one bulk samchon/graphSnapshot method from a resident clangd
background index built from a fork.

  • Producer fork: samchon/llvm-project, branch graph-bound-resident-views.
  • Pin: CPP_CLANG_PRODUCER_COMMIT = e33d8f51552a523b5696691738f1ef95f8e3a730.
  • Moving the pin forces a ~110-minute LLVM rebuild in CI. The producer cache key
    is hashFiles over CPP_CLANG_PRODUCER_COMMIT.ts, catalog.mjs and
    setup-language.mjs.
  • cppGraphProvider declares both languages, so a mixed C/C++ project runs one
    clangd on the strict path. buildLspGraph records every language a strict
    provider serves and the generic per-language loop skips those.

Design decisions worth copying into a new route:

  • Node identity is (language, target triple, declaring file, producer symbol
    id). The reading translation unit is deliberately excluded, so one entity seen
    from five units is one entity.
  • A shard carries only what it first derived, not everything it saw.
    Building and digesting shard-sized lists rather than read-sized ones is what
    made a large project tractable.
  • A published body is a view, streamed and hashed on the way to disk rather
    than copied, and a unit remembers the paths and digests of what it published
    so a later full walk does not rebuild it.

Measured state

Experiment matrix

.github/workflows/experiment.yml, fixtures declared in
tests/experiment/src/catalog.mjs. C/C++ jobs carry timeout-minutes: 150; the
fixtures are libuv for c and fmtlib/fmt for cpp.

before the campaign at 1bf9486
c full index, libuv, 469 shards 2,574,787 ms, then out of heap at the commit 66,152 ms
c incremental refresh 118 s 2.5 s for 470 shards
c LSP job red 18 min 34 s
cpp walk to committed, fmt, 1,405,417 relationships cancelled at the 150-minute cap 189,830 ms
cpp closing one generation 6,574,459 ms 74,827 ms
cpp LSP job cancelled at 150 min 40 min 45 s

Counts are stable across lifecycle walks: c reports 38,441 entities and
263,978 relationships, cpp 90,321 and 1,405,417.

Index-time matrix

.github/workflows/index-time.yml, thirteen projects, one per language.
timeout-minutes is 210 for c/cpp and 120 otherwise. Run 33377586947 on
1bf9486, Ubuntu 24.04, AMD EPYC 7763, 4 cores, 16 GB RAM:

Project Language Intended owner Actually served by Cold index Scale
slim php scip-php scip-php 3.3 s 125 files / 16,393 lines
excalidraw typescript ttscgraph static, fell back 6.1 s 599 / 172,012
flask python scip-python scip-python 12.8 s 83 / 18,337
serilog csharp scip-dotnet scip-dotnet 17.3 s 216 / 24,764
lualine lua samchon-graph-lua samchon-graph-lua 25.1 s 119 / 11,649
gin go samchon-graph-go samchon-graph-go 36.8 s 99 / 24,099
gson java javac-graph scip-java, fell back 105.9 s 262 / 55,620
leveldb cpp clangd-snapshot clangd-snapshot 176.0 s 132 / 28,516
koin kotlin scip-kotlinc scip-kotlinc 214.8 s 446 / 35,649
redis c clangd-snapshot scip-clang, fell back 325.7 s 783 / 366,929
tokio rust samchon-rust-analyzer-hir samchon-rust-analyzer-hir 445.9 s 789 / 177,251
sinatra ruby scip-ruby none, timed out at 300 s ? 147 / 23,891
darthttp dart scip-dart none, timed out at 300 s ? 324 / 115,359

Two readings matter here.

First, the fastest cells are fast because they publish no relationships. Pairing
this table with the edge-family column of the registry table above: scip-php
at 3.3 s and scip-dotnet at 17.3 s each publish zero edge families, while the
three cells that publish a usable graph ? go at 11 families, cpp at 12, rust at
14 ? take 37 s, 176 s and 446 s.

Second, and more consequential, three of the four compiler-authority routes
did not serve their own project in this matrix. ttscgraph, javac-graph and
clangd-snapshot each failed and fell through to a weaker tier while the run
still reported success. Only clangd-snapshot on leveldb and
samchon-graph-go on gin served as intended. The next section records each
failure with its message.

Per-project artifacts of that run carry report.json with cells[].buildMs,
servedBy, scale, host and toolchain provenance, plus the raw index logs and,
where a lane was slow, lsp-request-diagnosis-<project>.json.

Verified defects on master

1. Three strict routes fall back silently in the index-time matrix

A strict provider that fails is caught, recorded as a warning, and replaced by
the next tier; the index then completes and the run reports success. The
degradation is visible only in the servedBy field of report.json and in the
warning line of the index log, so a matrix can be entirely green while most of
the compiler-owned work is not running. All three cases below are from run
33377586947 on 1bf9486.

TypeScript, excalidraw ? the installed ttsc predates protocol v1:

@samchon/graph: indexing with ttscgraph(typescript)
@samchon/graph: typescript: the ttscgraph compiler provider failed, so these
languages fall through to the generic language-server lane: ttscgraph: binary
returned a legacy full dump instead of graph snapshot protocol v1; install a
matching ttsc
@samchon/graph: indexer=static no strict provider served

docs/provider-support.json records ttsc@>=0.24.0 as the first release whose
ttscgraph serve answers protocol v1. The fixture's toolchain provisioning is
what needs to change, not the provider.

Java, gson ? javac-graph fails on a Maven project:

@samchon/graph: indexing with javac-graph(java)
@samchon/graph: java: the javac-graph compiler provider failed, so these
languages fall through to the scip-java semantic-index provider: javac-graph:
.../tools/bin/scip-java exited with code 1: Picked up JAVA_TOOL_OPTIONS:
-Dmaven.repo.local=".../gson/cell-caches/gson/samchon-graph/maven"

gson is Maven-built. This is the same gap #74 records as a missing acceptance
item ? the acceptance requires multi-module Gradle and Maven fixtures, and
no pom.xml exists anywhere under tests/ ? except that it is not only a
missing fixture: the route demonstrably fails on Maven today, and no test covers
it. Whoever takes #74 should treat this cell as the reproduction.

C, redis ? the producer does not become ready in time:

@samchon/graph: indexing with clangd-snapshot(c)
@samchon/graph: c: the clangd-snapshot compiler provider failed, so these
languages fall through to the scip-clang semantic-index provider: C/C++ clang
graph: producer did not become ready within 300000 ms: graph snapshot is not
ready: 22 translation units are still indexing; the largest body built so far
holds 137190 occurrences, 37692 symbols and 126906 relations, 292 MiB in all,
from .../redis@6bf6224c3dad/deps/jemalloc/src/ctl.c
@samchon/graph: indexer=lsp scip-clang(c)

The readiness ceiling is lspReadyTimeoutMs in
indexer/IBuildGraphOptions.ts, threaded through cppGraphProvider to
CppGraphClient.readyTimeoutMs. redis is 783 files and 366,929 lines including
a vendored deps/ tree; libuv, which passes the same route in the experiment
matrix, is smaller.

scip-clang is the tier that catches this, and it publishes no defensible edge
family for C or C++, so a redis-sized C project receives declarations without
relationships. The C route passes the experiment matrix on libuv and does not
serve redis; both facts are true and the second is the one a claim of completion
has to be read against.

Questions a fresh campaign should settle before choosing a fix: whether the
ceiling is simply too low for this input class, whether vendored third-party
translation units belong in the universe at all, and whether a producer that is
still indexing should be waited on rather than abandoned.

A cross-cutting question sits above all three cases: whether a silent fall to a
weaker tier is the right behaviour for a matrix whose purpose is to prove the
strict routes. report.json already records servedBy, so the information
exists; nothing currently fails on it.

2. Dart and Ruby time out in per-symbol fan-out

Both lanes hit the 300-second cutoff issuing textDocument/references one
symbol at a time, which is the pattern #63 exists to remove. From
lsp-request-diagnosis-darthttp.json and lsp-request-diagnosis-sinatra.json:

Project Language Requests Completed Outcome In flight at cutoff
darthttp dart 3,266 3,250 timed out at 300,170 ms textDocument/references
sinatra ruby 214 198 timed out at 300,090 ms textDocument/references

The registered scip-dart and scip-ruby providers were attempted and not
selected, so the measured cell is 0.0 s and the project publishes nothing. These
are the fallback lanes behind #84 and #81.

3. isInside reports true across Windows drives, and the obvious fix is gated

Three identical copies, at repository/cargoRepositoryContextProvider.ts:451,
repository/cmakeRepositoryContextProvider.ts:628 and
repository/gradleRepositoryContextProvider.ts:397:

function isInside(root: string, file: string): boolean {
  const relative = path.relative(root, file);
  return relative !== ".." && !relative.startsWith(`..${path.sep}`);
}

On Windows path.relative("D:\\a", "C:\\b") returns the absolute C:\b, which
does not start with .., so a file on another drive is reported as inside the
root. Adding an absolute-path guard introduces a branch that can never be true
on Linux, and the 100% branch gate then fails on CI. A fix has to be shaped so
every branch is reachable on all platforms ? comparing resolved prefixes rather
than adding a platform-specific test is one such shape ? and must preserve the
case-insensitive comparison path.relative performs on Windows today.

Remaining routes

Each route needs a producer that answers in bulk, a consumer adapter committing
through the protocol, fixtures, an experiment job and 100% coverage. The
dominant variable is whether the producer already exists upstream.

Issue Language Producer situation Edge families today
#75 C# wrap scip-dotnet and Roslyn Workspaces 0
#76 Kotlin wrap the compiler Analysis API and scip-java per source set 2
#77 Scala consume SemanticDB and TASTy through BSP ?
#78 Swift consume SourceKit index generations through IndexStoreDB ?
#80 Python wrap Pyright and scip-python with truthful dynamic limits 1
#81 Ruby separate Sorbet bulk truth from the RubyIndexer fallback 0
#79 Zig add a compiler Sema exporter upstream ?
#82 PHP export PHPStan semantic snapshots 0
#83 Lua expose LuaLS semantic VM snapshots through a new bulk API 1
#84 Dart export analyzer AnalysisSession snapshots atomically 0

The last four require building or forking a producer in an upstream repository,
which is the shape the C/C++ route took with samchon/llvm-project. The first
six wrap or consume a bulk API that already exists.

Finishing items already recorded on their own issues:

Working conditions

  • CI is the slow loop. A cpp LSP job is about 40 minutes and a full PR
    rollup about 70. Pushing again while a run is in flight leaves several
    experiment runs competing for runners; cancel superseded runs rather than
    letting them queue.
  • A local harness is faster than CI for performance work. Every
    closing-cost defect in this cycle was found by driving
    CppGraphSnapshotAdapter and CppGraphClient against a fake producer
    in-process, at a scale chosen to match the fixture (52 units, about 1.27M
    edges), which turned a 50-minute cycle into an 8-second one.
    tests/test-graph/src/internal/fake-cpp-graph-server.cjs is the in-repo fake
    such a harness can be built from; the campaign's own harness lived in a
    scratch directory and is not in the repository.
  • Profile before changing, and re-measure after. Each accepted fix came from
    a --cpu-prof profile, and one plausible hypothesis ? that holding two
    generations at once was the cost ? was rejected because the measurement did
    not move.
  • Digest changes must be proved equivalent. canonicalFactText feeds both
    digest modules. The rewrite in this cycle was checked against the previous
    implementation over 400,000 randomly shaped values and on a 260,000-fact
    generation before it was accepted.
  • Sealing and memoization are load-bearing. Four separate costs in the
    closing path came from keeping records that outgrew the work they saved: a
    per-fact canonical-text memo, a seal that recorded every object it touched,
    the same generation digested twice, and canonical text built through
    intermediate arrays. Their commit messages in 1bf9486's range carry the
    measurements.

Repository procedure that was missed this cycle

.agents/skills/issue-campaign/development.md requires each issue to be closed
from the commit that earns it, with one Close #n: <issue title> line per
resolved issue, so the squash merge closes exactly what landed. No commit in
PR #164 carried such a line, so nothing closed automatically and #143, #163 and
#165 were reconciled by hand after the merge. A fresh campaign should carry the
closing lines from the start and reconcile them against HEAD before merging.

Acceptance for this handoff

Primary references

Current campaign revalidation ? 2026-09-02

A fresh solo campaign resumed from this handoff on clean
master@1bf9486. Four complete full-scope discovery rounds were run; rounds
one through three produced surviving candidates, and round four added none
after fact-checking, closing discovery for the current cycle.

Reconciliation and published issue units:

Local baseline on the unchanged target passed pnpm build, package-local type
checks, four complete 491-test runs, benchmark deterministic tests, provider
support generation, and the Go sidecar suite. Local pnpm coverage could not
satisfy the gate under Node 22.21.0 (81.69/76.47/93.31); the exact PR #164
Windows CI used Node 22.23.2 and reported 100%, with Ubuntu and macOS also green.
The cycle CI must prove 100% again on its immutable head.

No campaign implementation pull request has been claimed yet. Merge, release,
paid agent runs, and global toolchain installation remain unauthorized under
#162.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions