Repository navigation
Conversation
🌱 graft blast radius2 areas changed → 7 areas can be affected. 27 dependent symbols, depth 2. flowchart TB
A0(("Build And Review Pipeline<br/>13 symbols"))
A1(("Workspace Graph Operations<br/>7 symbols"))
A2(("Engine Core<br/>2 symbols"))
A3(("Build Context<br/>2 symbols"))
A4(("Viewer Build Script<br/>1 symbol"))
AX(("2 smaller areas<br/>2 symbols"))
classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
class A0,A1,A2,A3,A4 reached;
classDef tail fill:#EEF2F3,stroke:#9AA4A9,stroke-width:1px,color:#3A4247;
class AX tail;
Who knows this code — 4 people across 9 areas
Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no All 27 dependent symbols, grouped by areaBuild And Review Pipeline — 13 symbols in 8 files
Workspace Graph Operations — 7 symbols in 5 files
Engine Core — 2 symbols in 1 file
Build Context — 2 symbols in 1 file
Viewer Build Script — 1 symbol in 1 file
MCP Tools — 1 symbol in 1 file
Claude Sync Execution — 1 symbol in 1 file
Test signal per changed area — 1 ✓ · 1 –Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.
35 test suites also reference this code47 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.
Open the interactive graph → — click an area to see its dependent symbols at file:line. |
|
Confirming this on a second language, and offering a measurement. Env: graft 0.18.0, Node v24.9.0, Windows 11. Private C# monorepo, 5361 parseable files. Your diagnosis reproduces exactly. I arrived at The +const parserPool = new Map();
+function pooledParser(langName, language) {
+ let p = parserPool.get(langName);
+ if (!p) { p = new tsMod.Parser(); p.setLanguage(language); parserPool.set(langName, p); }
+ return p;
+}
export function extractGeneric(rel, source, langName) {
@@
let tree;
try {
- const parser = new tsMod.Parser();
- parser.setLanguage(entry.language);
- tree = parser.parse((i) => source.slice(i, i + PARSE_CHUNK));
+ tree = pooledParser(langName, entry.language).parse((i) => source.slice(i, i + PARSE_CHUNK));
}
@@
else if (langName === "php")
extractPhpUses(tree.rootNode, rel, rawEdges);
+ tree.delete();
return { nodes, rawEdges };
}Results on the repo above, same commit,
Edges recover +56% against +18% for nodes — the dropped files took every edge crossing into them with them. This path is shared by every WASM-parsed language, so it is not C#-specific. Two things it deliberately does not do, both of which your PR does: it leaves Caveat on pooling: it assumes single-threaded parsing, so it would need revisiting alongside the streaming worker pool you mention. |
cheers for this! apologies for the delay, let me get this updated and tuned as I found a C++ repo case where I run into a going to use the Linux kernel as the test case, as I found current |
…ed runtime extractGeneric and parseWasm created a Parser per file and never deleted it or the Tree. Both live in the WASM heap (2 GB cap, no finalizer), so a 65k-file C repo aborted after ~12k files and every later parse failed with Aborted(). Reuse one parser per grammar and delete every tree. After a real abort, fail every later parse with one message instead of one per file. parseWasm now returns the tree for the container tier to free, and rethrows instead of caching a symbol-less file as a success.
A cached error was replayed until the file's bytes changed, so one abort pinned 52k files as failed. Skip the cache hit for an errored entry. The fingerprint keeps its hash, so a refresh does not rebuild just to retry it.
A C++ header misparsed by the C grammar can nest one level per initializer element; the recursive walks overflowed the call stack.
63ab0c7 to
487513a
Compare
|
rebased on main (0.21.1); the I also added the
@pavelsem on your caveat about pooling: the pool in the follow-up PR ( #548 ) forks child processes, so each child gets its own module state and its own parser per grammar + a pooled parser is never shared across workers. agreed that -> overlap with open PRs, to help with merge order:
|
csm-kb
changed the title
This branch has not been deployed
2 participants
Was working with
grafton a large codebase (~65k files) to stress-test, noticed some failure doing so -> attempted a fix pass to move it towards functional. Feel free to leave inputs or modify/reject!I have a subsequent PR ( #548 ) that improves the
graft buildmemory tolerance + performance using a streaming worker pool (measured 2.2x speedup on that codebase) that I can open afterwards.Problem
extractGenericandparseWasmcreated aParserper file and never deleted it or theTree. Both live in the WASM heap (2 GB cap, no finalizer), so a 65k-file C repo aborted after ~12k files and every subsequent parse failed withAborted()after a few minutes.Solution
Reuse one parser per grammar and delete every tree. After a real abort, fail every later parse with one message instead of one per file.
parseWasmnow returns the tree for the container tier to free, and rethrows instead of caching a symbol-less file as a success.