feat(ingest): configurable size cap, graftignore and minified-bundle skip

[webmaniabr]

Two problems with the current file selection, both hit on a real PHP monorepo:

• Files over 1 MB are skipped silently, and there is no way to raise the cap. A hand-written 1.5 MB product file (hundreds of methods) never made it into the graph, and nothing told the user why or how to change it.
• Tracked bundles and minified files become noise, and .gitignore cannot help. Git's ignore rules only gate untracked files, so committed *.min.js files and build bundles are always indexed and crowd query results.

This PR makes both configurable, with the defaults left exactly as they are.

Related: #370 (the silent 1 MB drop; this adds the override that issue suggests, and is complementary to #399, which adds the notice but keeps the cap fixed) and #575 (also adds .graftignore; the two overlap there, and I'm happy to rebase onto whichever approach you prefer).

What changes

• GRAFT_MAX_FILE_BYTES=<n> overrides the per-file size cap. Default is still 1000000. A value that is not a positive integer is ignored with a single warning and the default is kept.
• gitignore-style ignore files that also apply to tracked files. They are read from three sources, lowest precedence first, and later rules override earlier ones as in Git:
  1. ~/.graft/ignore: user-wide.
  2. $GIT_COMMON_DIR/info/graftignore (usually .git/info/graftignore): this clone only, never committed, and shared by all worktrees of the clone.
  3. .graftignore at the repo root: committed with the code.
• GRAFT_SKIP_MINIFIED=1 (opt-in, off by default) skips .js/.mjs/.cjs files of at least 100 KB whose longest line is over 20,000 characters. The check is memoized by size+mtime.

Supported syntax: # comments, *, ?, [a-z], **, / anchoring (a leading or inner slash), trailing / for directories only, ! negation, last match wins. As in Git, a file can't be re-included once a parent directory is excluded.

Limitations: matching is case-sensitive, and a submodule's own .graftignore is not read. Submodule paths go in the superproject's file.

How

• New src/ingest/filters.ts. walkDir builds one FileFilter per walk and passes it through all three enumeration paths: shallow git, recursive git (submodules/nested repos) and the filesystem fallback.
• Env vars and ignore files are read at walk time, so graft build, the freshness probe, the pre-query refresh, the MCP server and the hooks always see the same file set. Changing a rule shows up as drift on the next query.
• The glob matcher is hand-written and segment-based. It never compiles patterns to RegExp, so a pathological pattern in a committed .graftignore can't cause catastrophic backtracking (ReDoS).

Compatibility

• Defaults are unchanged. The cap is 1,000,000 bytes, minified skipping is off, and with no ignore files present the walk keeps exactly the same files as before.
• MAX_FILE_BYTES is still exported from src/ingest/fs.ts with the same value.
• No new dependencies.

Tests

• npm test: 1357 pass on main, 1373 pass on this branch (+16, in test/ingest-filters.test.ts), 0 fail. tsc --noEmit is clean and npm run build passes.
• Coverage: the size cap (default, override, invalid value), precedence across the three ignore sources including ! overrides, tracked files being excluded, directory pruning, minified detection, and an end-to-end build followed by the freshness probe, so a filter seen differently by the refresh path would show up as drift.
• ReDoS: six pathological patterns (**/**/**/**/a*a*a*a*b, a*a*a*a*…*b, 500 *s, *?*?…, a 20,000-char pattern) are matched against a 10,000-char segment and a 200-level-deep path. Locally a full round takes about 0.3 ms (median of 20) and about 4 ms on a cold first run. The test asserts under 50 ms so it stays stable on a loaded CI runner.

Real-world check (PHP repo)

• With GRAFT_MAX_FILE_BYTES=2000000, a 1.5 MB tracked PHP file went from 0 to 383 indexed methods.
• With .graftignore + GRAFT_SKIP_MINIFIED=1, about 300 tracked .min.js files and bundles dropped out of the graph while staying tracked in Git.

🤖 Generated with Claude Code

[trailhq-graft]

🌱 graft blast radius

1 area changed → 4 areas can be affected. 12 dependent symbols, depth 2.
Tests: 1 area updated its tests.
Tag: @anirudhkumar-nanonets — 5 of 5 areas · @shhdwi — Dependency Graph, Context Analysis · @yescine — Filesystem Ingestion

flowchart TB
  A0(("Dependency Graph<br/>6 symbols"))
  A1(("Context Analysis<br/>3 symbols"))
  A2(("Pull Request Review<br/>2 symbols"))
  A3(("Graph Engine<br/>1 symbol"))
  classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
  class A0,A1,A2,A3 reached;

Loading

Can be affected	Symbols	Nearest hop	Reached from
Dependency Graph	6	src/graph/build.ts:L151-L410 buildGraph — calls, depth 1	Filesystem Ingestion
Context Analysis	3	src/context/build.ts:L109-L122 listContextFiles — calls, depth 1	Filesystem Ingestion
Pull Request Review	2	src/app/brain-build.ts:L251-L358 readRepository — calls, depth 2	Filesystem Ingestion
Graph Engine	1	src/engine.ts:L91-L101 graph — calls, depth 2	Filesystem Ingestion

Who knows this code — 3 people across 5 areas

Area	Who knows it
Filesystem Ingestion · changed	@anirudhkumar-nanonets — 3 commits, last 2mo ago · @yescine — 2 commits, last 2mo ago
Dependency Graph · affected	@anirudhkumar-nanonets — 19 commits, last 2mo ago · @shhdwi — 11 commits, last 2mo ago
Context Analysis · affected	@anirudhkumar-nanonets — 5 commits, last 2mo ago · @shhdwi — 3 commits, last 2mo ago
Pull Request Review · affected	@anirudhkumar-nanonets — 9 commits, last 1mo ago
Graph Engine · affected	@anirudhkumar-nanonets — 16 commits, last 3mo ago

Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no @ has no GitHub handle in its commit email — tag them by hand, or add a .mailmap entry. A suggestion from history, not a CODEOWNERS rule.

All 12 dependent symbols, grouped by area

Dependency Graph — 6 symbols in 5 files

• src/graph/build.ts:L151-L410 — buildGraph (calls, depth 1)
  160: const walked = walkDir(root, readIncludeDirs(root), {
• src/graph/scopes.ts:L133-L272 — discoverScopes (calls, depth 1)
  135: repoFiles: string[] = walkDir(root, readIncludeDirs(resolve(root)), {
• src/graph/source-files.ts:L64-L85 — listSourceFiles (calls, depth 1)
  67: repoFiles: string[] = walkDir(root, readIncludeDirs(resolve(root)), {
• src/graph/check.ts:L58-L164 — checkGraph (calls, depth 2)
• src/graph/refresh.ts:L150-L227 — ensureFreshGraph (calls, depth 2)
  157: // One case has a graph to work with even though this checkout has none: a git
• src/graph/source-files.ts:L103-L120 — listSourceStats (calls, depth 2)

Context Analysis — 3 symbols in 2 files

• src/context/build.ts:L109-L122 — listContextFiles (calls, depth 1)
  115: const walked = walkDir(root, readIncludeDirs(root), {
• src/context/build.ts:L146-L374 — buildContext (calls, depth 2)
  252: // An empty array is a miss, not a hit: caching [] made a silent empty
• src/context/check.ts:L38-L111 — checkContext (calls, depth 2)

Pull Request Review — 2 symbols in 2 files

• src/app/brain-build.ts:L251-L358 — readRepository (calls, depth 2)
• src/app/review.ts:L45-L99 — reviewPullRequest (calls, depth 2)
  93: // The token is in git's error text on a permissions failure, and this string

Graph Engine — 1 symbol in 1 file

• src/engine.ts:L91-L101 — graph (calls, depth 2)

Test signal per changed area — 1 ✓

Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.

• ✓ Filesystem Ingestion — 6 of 30 reached · 1 test file changed here: test/ingest-filters.test.ts
  • not reached: charMatches, compileRule, compileSegments, createFileFilter, envTruthy, extensionOf, globalIgnorePath, ignoreFilesFor, …16 more

34 test suites also reference this code

38 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.

• test/ask-index.test.ts
• test/ask.test.ts
• test/container-extract.test.ts
• test/context-only-dir.test.ts
• test/context.test.ts
• test/covers.test.ts
• test/generic-extract.test.ts
• test/graph-go.test.ts
• test/graph-incremental.test.ts
• test/graph-invariants.test.ts
• test/graph-java.test.ts
• test/graph-languages.test.ts
• test/graph-php.test.ts
• test/graph-posix-paths.test.ts
• test/graph-python.test.ts
• test/graph-r-classes.test.ts
• test/graph-r-phase3.test.ts
• test/graph-r-phase4.test.ts
• test/graph-r-phase5.test.ts
• test/graph-r.test.ts
• …14 more

⚠️ 1 changed file not in the graph (README.md) — no parser claims the extension, or the index predates the file.

_{graft blast · origin/main...HEAD · depth 2 · 4 changed files}

Open the interactive graph → — click an area to see its dependent symbols at file:line.

[webmaniabr]

Withdrawing this PR — we're consolidating our changes in an internal build and won't be able to maintain this branch. #575 covers the .graftignore part; the size-cap env (GRAFT_MAX_FILE_BYTES), the minified-JS skip and the regex-free glob matcher here are free to be picked up if useful. Thanks!