Repository navigation
perf(build): build very large repos end to end; opt-in parallel parsing with graft build --workers - #548
Draft
csm-kb wants to merge 8 commits into
Draft
perf(build): build very large repos end to end; opt-in parallel parsing with graft build --workers#548csm-kb wants to merge 8 commits into
graft build --workers#548csm-kb wants to merge 8 commits into
Conversation
…ed runtime extractGeneric and parseWasm created a Parser per file and never deleted it or the Tree. Both live in the WASM heap (2 GB cap, no finalizer), so a 65k-file C repo aborted after ~12k files and every later parse failed with Aborted(). Reuse one parser per grammar and delete every tree. After a real abort, fail every later parse with one message instead of one per file. parseWasm now returns the tree for the container tier to free, and rethrows instead of caching a symbol-less file as a success.
A cached error was replayed until the file's bytes changed, so one abort pinned 52k files as failed. Skip the cache hit for an errored entry. The fingerprint keeps its hash, so a refresh does not rebuild just to retry it.
A C++ header misparsed by the C grammar can nest one level per initializer element; the recursive walks overflowed the call stack.
Each artifact was one JSON.stringify and one JSON.parse; V8 caps a string at 512 MB, and a 65k-file C repo produces a 1.9 GB extract cache, a 700 MB wiring.json and a 590 MB ask index. The extract cache is now NDJSON read one line at a time from a Buffer. wiring.json and the ask index are written one element per line (still plain JSON for every reader) and read back line by line when they are over the cap.
buildGraph kept every file's source text for the whole build, about 1 GB of strings on a 65k-file repo, for a pass a Tier-1 build never runs. The meaning pass now reads a file when it summarizes it, and only if its bytes still hash to what was parsed. The per-file read/hash/extract step moves into extractOne so the in-thread loop and a parser child run the same code.
Large cold builds can parse in child processes; results are folded in file order so the graph is byte-identical to a single-process build. Off by default and read only by the build command, so the pre-query refresh, the MCP server and the App never fork. A dead child's job is retried once; a job that kills two children is parked and becomes an error unless the pool never succeeded at all, in which case the parent parses the rest itself.
…ckground sync in-process; changelog
Contributor
🌱 graft blast radius5 areas changed → 13 areas can be affected. 46 dependent symbols, depth 2. flowchart TB
A0(("Build And Review Pipeline<br/>13 symbols"))
A1(("Graph Loading And Traversal<br/>10 symbols"))
A2(("Build Progress UI<br/>7 symbols"))
A3(("Build Scripts<br/>2 symbols"))
A4(("MCP Tool Invocation<br/>2 symbols"))
AX(("8 smaller areas<br/>12 symbols"))
classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
class A0,A1,A2,A3,A4 reached;
classDef tail fill:#EEF2F3,stroke:#9AA4A9,stroke-width:1px,color:#3A4247;
class AX tail;
Who knows this code — 4 people across 18 areas
Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no All 46 dependent symbols, grouped by areaBuild And Review Pipeline — 13 symbols in 8 files
Graph Loading And Traversal — 10 symbols in 6 files
Build Progress UI — 7 symbols in 3 files
Build Scripts — 2 symbols in 2 files
MCP Tool Invocation — 2 symbols in 1 file
Ask Corpus Loading — 2 symbols in 1 file
Context Building — 2 symbols in 1 file
Brain Rules Refresh — 2 symbols in 1 file
Background Push And Auth — 2 symbols in 2 files
LSP Client Connection — 1 symbol in 1 file
Telemetry Flushing — 1 symbol in 1 file
Blast Command Execution — 1 symbol in 1 file
Grep Search CLI — 1 symbol in 1 file
Test signal per changed area — 4 ✓ · 1 ✗Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.
41 test suites also reference this code60 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.
Open the interactive graph → — click an area to see its dependent symbols at file:line. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
this is the follow-up to #340 (branch is based on that one's commits): the last 4 commits here are this PR -> can rebase once #340 lands
with #340, the 65k-file C repo no longer aborts in the WASM heap -- but the build still can't finish correctly. this PR gets it through end-to-end, and adds an opt-in'able way to parse using multiple parallel workers
Problem
After #340, a cold build of the ~65k-file C repo hit two more limits:
buildGraphheld every file's source text for the whole build (about 1 GB of strings here) for a meaning pass that a Tier-1 build never runs. With Node's default heap the build died withJavaScript heap out of memoryat file 38,988.JSON.stringifyand oneJSON.parse. On this repo the extract cache is 1.87 GB, wiring.json 704 MB and the ask index 586 MB.writeExtractCacheswallowed theInvalid string length(so every later build ran cold),writeGraphthrew after the full parse, and a graph that did get written couldn't be loaded byask/check.Parsing is also single-threaded, at about 3.3 ms per C file.
Solution
b5469edStream the artifacts. The extract cache is now NDJSON (extract.<stamp>.ndjson, cache v2). wiring.json and the ask index are written one element per line; they are still plain JSON, so everyJSON.parsereader works unchanged, and above the cap they're read back line by line. wiring.json is now compact rather than indented. Old.jsoncaches age out through the existing sidecar pruner.516059fRead sources lazily. The meaning pass re-reads a file when it summarizes it, and only if the bytes still hash to what was parsed; otherwise the node stays pending for the next build. The per-file read/hash/extract step moves intoextractOne, so the in-thread loop and a worker run the same code.69525d2graft build --workers <n|auto>(orGRAFT_PARSE_WORKERS). Parses inchild_process.forkchildren and folds results in file order, so wiring.json is byte-identical to an in-thread build. Off by default and read only by thebuildcommand: the pre-query refresh, the MCP server and the App never fork.auto=min(cores - 1, free memory / 300 MB, files / 200), and stays in-thread below 200 files or 2 workers.b1d2ff8Harden it.--workersis validated and capped at the core and file counts. The background sync passes--workers 0, so aGRAFT_PARSE_WORKERSin.envcan't make it fork.readFileSynclimit..env.example, plus the changelog.Measurements
Windows 10, Node v22.23.2, 65,159-file C repo (1,226,623 nodes, 1,261,404 edges), cold build (extract and fingerprint caches deleted),
NODE_OPTIONS=--max-old-space-size=16384. Peak RSS is summed across the parent and its children.--workers auto(15 children)Both cold builds parse all 65,159 files with no errors. wiring.json from the in-thread and pooled builds is byte-identical, and
graft askandgraft checkboth load the graph.Known limits / follow-ups
body_text(up to 5,000 chars, needed by the ask index) stays resident until the index is written. Tokenizing it during the fold would remove that requirement.buildre-hashes every file and rewrites the artifacts even when nothing changed. A no-drift fast path would help.graft vizstill reads wiring.json as one string, so it's limited to graphs under 512 MB.--workersisn't wired into workspace builds yet; a workspace build says it's ignoring it.