Skip to content
Merged
28 changes: 28 additions & 0 deletions benchmarks/conformance/baseline.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,18 @@
{
"apps_script": {
"edges_all": {
"precision": 1.0,
"recall": 1.0
},
"edges_call": {
"precision": 1.0,
"recall": 1.0
},
"symbols": {
"precision": 1.0,
"recall": 1.0
}
},
"cpp": {
"edges_all": {
"precision": 1.0,
Expand Down Expand Up @@ -27,6 +41,20 @@
"recall": 1.0
}
},
"l5x": {
"edges_all": {
"precision": 1.0,
"recall": 1.0
},
"edges_call": {
"precision": 1.0,
"recall": 1.0
},
"symbols": {
"precision": 1.0,
"recall": 1.0
}
},
"python": {
"edges_all": {
"precision": 1.0,
Expand Down
242 changes: 242 additions & 0 deletions docs/adr/ADR-052-scan-defaults-and-languages.md

Large diffs are not rendered by default.

11 changes: 9 additions & 2 deletions src/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,18 @@ Model IDs live in `indexer.toml` (`[embeddings]`, `[reranker]`, `[summarization]

**`src/scan_policy.py` is the single answer to "does the indexer look at this file?"** (ADR-026 §2). Both consumers call it — `incremental_indexer.scan_disk()`, which walks the tree, and `MCPServer._CodeChangeHandler._is_relevant`, which sees only a path string on each filesystem event. They used to be two hand-mirrored implementations over three module constants, which is how a JavaScript-seeded exclusion list survived on a Python tool long enough to index 510 files of third-party eval corpus from this repo's own `benchmarks/`.

It is a **leaf module** — `os`, `dataclasses`, `functools`, `config`. Do not import a sibling `src` module into it: `config` importing `incremental_indexer` for a constant is the cycle ADR-026 §2 was written around, and it would surface only through the server's function-local import path.
It is a **leaf module** — `os`, `dataclasses`, `functools`, `fnmatch`, `config`. Do not import a sibling `src` module into it: `config` importing `incremental_indexer` for a constant is the cycle ADR-026 §2 was written around, and it would surface only through the server's function-local import path.

Two predicates, and the difference matters: `is_indexable()` is source that gets chunked and embedded; `is_scannable()` is that plus the `.csproj`/`.sln`/`compile_commands.json` descriptors, which are parsed for edges only. The watchdog uses the second — using the first meant editing a `.csproj` never triggered a reindex.

Configuration is the `[ignore]` block. `extra_dirs` / `extra_root_dirs` / `extra_extensions` **add** to the defaults; the bare `dirs` / `root_dirs` / `extensions` keys **replace** them; setting both spellings of one knob raises. Defaults live in `scan_policy.py` and are deliberately **not** duplicated into the shipped `indexer.toml` — a bare key there would freeze this repo's gate and silently drop every future default.
Configuration is the `[ignore]` block. `extra_dirs` / `extra_root_dirs` / `extra_extensions` / `extra_files` **add** to the defaults; the bare `dirs` / `root_dirs` / `extensions` / `files` keys **replace** them; setting both spellings of one knob raises. A `!name` entry in `extra_dirs` / `extra_root_dirs` / `extra_extensions` drops that one default and keeps the rest (`!` in `extra_files` re-includes matching names instead). Defaults live in `scan_policy.py` and are deliberately **not** duplicated into the shipped `indexer.toml` — a bare key there would freeze this repo's gate and silently drop every future default.

What the defaults decide, and why (ADR-052):

- **`.gs` is JavaScript** (Apps Script), through `AppsScriptAdapter` in `adapters/__init__.py` — a subclass only because the JS adapter picks its grammar by extension. Its cross-file calls resolve by the ADR-021 unique-name rule, which matches Apps Script's one global scope; a function defined in two `.gs` files is a known wrong binding.
- **`.cshtml` / `.razor` are text**: no adapter, so `fallback_token_chunker` for every tier, no symbols or edges. `.xaml` is opt-in (`extra_extensions = [".xaml"]`): taking a default extension back out costs a full `extensions` replacement.
- **`obj` is ignored at any depth; bare `bin` is not.** A `dirs` entry may be a multi-segment path, matched as consecutive directory names at any depth, and `bin/Debug`, `bin/Release`, `bin/x64`, `bin/x86` catch MSBuild output while `bin/*.py` and npm `bin/*.js` stay. A custom build configuration is `extra_dirs = ["bin/Staging"]`. These entries cannot be pruned during the walk (prune sees one name), so the path check rejects the files instead — don't use one for a `node_modules`-sized tree. Generated `*.Designer.cs` / `*.g.cs` outside `obj/` stay: checked-in generated code is often the only definition the rest of the code calls. EF Core `Migrations/` is real source and stays; `extra_dirs = ["Migrations"]` if it is noise.
- **`files` is file-name patterns** (`fnmatch`, case-folded), never scanned whatever `extensions` says: credential files — `.env*`, `*credentials*.json`, service-account and Firebase key downloads, `secrets.json`, `*.pem`/`*.key`/`*.p12`/`*.pfx`, `id_rsa`-style keys. A `!` pattern re-includes (`.env.example` ships that way), so one exception never needs the defaults replaced. A pattern with a `/` raises. The gate sees only paths; key *content* is the MCP output-redaction pass's job, not this module's.

The scan is anchored to the directory holding `indexer.toml`, and a scan launched from a subdirectory is refused rather than resolving root-only exclusions against the wrong tree. A run that would delete more than `max(50, 20%)` of the index asks first (`--prune` to skip); unattended callers log and skip instead.

Expand Down
13 changes: 13 additions & 0 deletions src/adapters/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,21 @@
from adapters.cpp_adapter import CppAdapter
from adapters.l5x_adapter import L5xAdapter


class AppsScriptAdapter(JavaScriptAdapter):
"""Apps Script `.gs` (ADR-052 §1): plain JavaScript under the JS grammar.

Not a bare REGISTRY line because `_WebAdapter.parse` looks the grammar up by
extension, so `.gs` -> `JavaScriptAdapter` raises KeyError on the first parse.
"""
extensions = frozenset({".gs"})
_grammars = {".gs": JavaScriptAdapter._grammars[".js"]}


_python = PythonAdapter()
_ts = TypeScriptAdapter()
_js = JavaScriptAdapter()
_gs = AppsScriptAdapter()
_cs = CSharpAdapter()
_cpp = CppAdapter()
_l5x = L5xAdapter()
Expand All @@ -36,6 +48,7 @@
".tsx": _ts,
".js": _js,
".jsx": _js,
".gs": _gs,
".cs": _cs,
".csproj": _cs,
".sln": _cs,
Expand Down
8 changes: 8 additions & 0 deletions src/call_resolver.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@
ADR-044 (``member_call`` NULL, and every edge of an adapter that does not compute it) resolve
as before, except that a ``.py`` source skips import-scoping, which it never reached before.

**Apps Script** (ADR-052) — every ``.gs`` file shares one global scope and has no imports, and
when two files define the same top-level function, whichever loads last wins everywhere. So a
bare call from a ``.gs`` file with several candidates skips the positional steps (same-file and
import-scoped) and stays unresolved: binding the caller to its own file's copy would be a guess.
A member call (``this.x()``) still gets the same-file step.

The pass recomputes **all** CALLS edges each run, so a resolution is demoted back to NULL if
a later-added symbol makes its name ambiguous — a stale resolution never outlives the
uniqueness that justified it (ADR-021 §3).
Expand Down Expand Up @@ -151,6 +157,8 @@ def resolve_call_edges(db) -> dict:
elif len(cands) == 1:
resolved = cands[0][0]
stats["resolved"] += 1
elif len(cands) > 1 and src_file.endswith(".gs") and member != 1:
stats["ambiguous"] += 1 # one global scope: no file is nearer (ADR-052)
elif len(cands) > 1:
src_id = path_to_id.get(src_file)
same = [fqn for fqn, fid in cands if fid == src_id]
Expand Down
135 changes: 127 additions & 8 deletions src/scan_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
behaviour. Both consumers now call the same predicate.

**Leaf placement is load-bearing, not cosmetic.** This module imports only ``os``,
``dataclasses`` and ``config``. A policy helper living in ``incremental_indexer`` and
``dataclasses``, ``fnmatch`` and ``config``. A policy helper living in ``incremental_indexer`` and
imported by ``config`` deadlocks at import time, and would surface only through the
server's function-local import path — the hardest possible place to see it.

Expand All @@ -26,6 +26,7 @@

import os
from dataclasses import dataclass
from fnmatch import fnmatchcase
from functools import cached_property

from config import find_config_path, index_dir_setting, load_indexer_config
Expand Down Expand Up @@ -55,6 +56,13 @@
# Editor, tooling and infrastructure state
".continue", ".claude", ".vs", ".vscode",
"Modelfiles", ".github", ".firebase", ".idx", "genkit",

# .NET build output (ADR-052 §2, B-063). `obj` is MSBuild's intermediate tree —
# 38% of AcadeDesignTools' `.cs` files were generated `obj/` output. Bare `bin` is
# NOT here: `bin/*.py` and `bin/*.js` are real scripts in Python and npm projects.
# MSBuild writes `bin/<Configuration>/` or `bin/<Platform>/<Configuration>/`, so the
# multi-segment entries below match only that (consecutive directory names, any depth).
"obj", "bin/Debug", "bin/Release", "bin/x64", "bin/x86",
})

DEFAULT_IGNORE_ROOT_DIRS: frozenset[str] = frozenset({"functions"})
Expand All @@ -65,8 +73,23 @@
# statement about code that never runs.
DEFAULT_INDEXABLE_EXTS: frozenset[str] = frozenset({
".py", ".ts", ".tsx", ".js", ".jsx", # Tier-A: JS/TS/Python
".gs", # Apps Script: the JS adapter (ADR-052 §1)
".cs", # Tier-A: C#
".cpp", ".cc", ".cxx", ".h", ".hpp", # Tier-A: C++ (.h routed to the C++ adapter)
".cshtml", ".razor", # Razor/Blazor: text only, no adapter (ADR-052 §3)
})

# File-name patterns never scanned, whatever the extension knob says (ADR-052 §4, B-018):
# files that typically hold credentials. `fnmatch` over the case-folded file name; a `!`
# pattern re-includes. Narrow on purpose — `*key*.json` would take `keyboard.json`, and
# `*credentials*` without `.json` would take `credentials.py`, which is ordinary source.
DEFAULT_IGNORE_FILES: frozenset[str] = frozenset({
".env", ".env.*", "!.env.example", "!.env.sample", "!.env.template",
"*credentials*.json", "client_secret*.json",
"*service[-_]account*.json", "serviceaccount*.json", "*firebase-adminsdk*.json",
"*-" + "[0-9a-f]" * 12 + ".json", # gcloud key download: <project>-<12 hex>.json
"secrets.json",
"*.pem", "*.key", "*.p12", "*.pfx", "id_rsa", "id_dsa", "id_ecdsa", "id_ed25519",
})

# Project descriptor files: edges only, no chunking or embedding. Not operator knobs —
Expand Down Expand Up @@ -113,7 +136,31 @@ def _as_str_list(value: object, key: str) -> list[str]:
return list(value)


def _resolve(block: dict, key: str, defaults: frozenset[str]) -> frozenset[str]:
def _extend(defaults: frozenset[str], values: list[str], key: str) -> frozenset[str]:
"""``extra_*``: add each entry to the defaults, and remove each ``!name`` from them.

``!`` drops one built-in default without a bare-key replacement, which would freeze
every later default (ADR-052). A ``!name`` that matches no default raises: it removes
nothing, and a typo there would otherwise keep indexing what the user meant to drop.
Matching is folded like every directory comparison.
"""
removed: set[str] = set()
for value in values:
if not value.startswith("!"):
continue
hits = {d for d in defaults if _fold(d) == _fold(value[1:])}
if not hits:
raise ValueError(
f"[ignore].{key}: `{value}` names no built-in default, so it removes "
f"nothing. `!` drops a default; to add an entry, leave the `!` off."
)
removed |= hits
added = {v for v in values if not v.startswith("!")}
return (frozenset(defaults) - removed) | frozenset(added)


def _resolve(block: dict, key: str, defaults: frozenset[str], *,
removable: bool = True) -> frozenset[str]:
"""Apply the ADR-026 §1 precedence table to one knob.

============ ============= =========================================
Expand All @@ -129,6 +176,10 @@ def _resolve(block: dict, key: str, defaults: frozenset[str]) -> frozenset[str]:
absent key: ``extra_dirs = []`` is a no-op and ``dirs = []`` means "exclude
nothing" — legal, destructive, and exactly what someone asking for it asked for.

With ``removable`` (every knob but ``files``, whose ``!`` is a pattern re-include), a
``!name`` entry in ``extra_*`` removes that one default (:func:`_extend`). In the bare
key it raises: the bare key already replaces the defaults, so there is nothing to remove.

Both keys set is an error rather than a silent precedence because there is no
reading of "extend *and* replace" a user could have meant, and guessing is how
the defect this ADR removes was born in the first place.
Expand All @@ -145,9 +196,19 @@ def _resolve(block: dict, key: str, defaults: frozenset[str]) -> frozenset[str]:
)

if bare is not None:
return frozenset(_as_str_list(bare, key))
values = _as_str_list(bare, key)
negated = sorted(v for v in values if v.startswith("!"))
if removable and negated:
raise ValueError(
f"[ignore].{key} replaces the defaults, so {negated} has nothing to "
f"remove. Use `{extra_key}` to drop one default and keep the rest."
)
return frozenset(values)
if extra is not None:
return frozenset(defaults) | frozenset(_as_str_list(extra, extra_key))
values = _as_str_list(extra, extra_key)
if removable:
return _extend(defaults, values, extra_key)
return frozenset(defaults) | frozenset(values)
return frozenset(defaults)


Expand All @@ -169,6 +230,22 @@ def _resolve_exts(block: dict, defaults: frozenset[str]) -> frozenset[str]:
return frozenset(e.lower() for e in resolved)


def _resolve_files(block: dict, defaults: frozenset[str]) -> frozenset[str]:
"""Same precedence as :func:`_resolve`, for file-name patterns.

A pattern with a ``/`` raises: these match the file's *name*, so
``config/secret.json`` would match nothing and exclude nothing, silently.
"""
resolved = _resolve(block, "files", defaults, removable=False)
bad = sorted(p for p in resolved if "/" in p or "\\" in p)
if bad:
raise ValueError(
f"[ignore].files patterns match a file name, not a path: {bad}. "
f"Exclude a directory with `extra_dirs` instead."
)
return resolved


# ---------------------------------------------------------------------------
# The policy object
# ---------------------------------------------------------------------------
Expand All @@ -182,6 +259,7 @@ class ScanPolicy:
ignore_dirs: frozenset[str] # excluded at any depth
ignore_root_dirs: frozenset[str] # excluded at the repository root only
extensions: frozenset[str] # chunked + embedded
ignore_files: frozenset[str] = DEFAULT_IGNORE_FILES # file-name patterns; `!` re-includes
project_exts: frozenset[str] = PROJECT_EXTS
project_files: frozenset[str] = PROJECT_FILES
index_dir_name: str = ".code-index" # [indexer].index_dir's last component (ADR-042)
Expand All @@ -197,6 +275,21 @@ def _folded_dirs(self) -> frozenset[str]:
def _folded_root_dirs(self) -> frozenset[str]:
return frozenset(_fold(d) for d in self.ignore_root_dirs)

@cached_property
def _folded_dir_paths(self) -> tuple[tuple[str, ...], ...]:
"""Multi-segment `ignore_dirs` entries (`bin/Debug`), as folded name tuples."""
return tuple(
tuple(_fold(p) for p in d.strip("/").split("/"))
for d in self.ignore_dirs if "/" in d.strip("/")
)

@cached_property
def _folded_file_patterns(self) -> tuple[tuple[str, ...], tuple[str, ...]]:
"""(exclude, re-include) file-name patterns, folded once."""
pats = [_fold(p) for p in self.ignore_files]
return (tuple(p for p in pats if not p.startswith("!")),
tuple(p[1:] for p in pats if p.startswith("!")))

def is_ignored_dir(self, name: str, *, at_root: bool = False) -> bool:
"""Whether ``os.walk`` should prune a directory entry named ``name``.

Expand Down Expand Up @@ -225,11 +318,35 @@ def _under_ignored_dir(self, rel_path: str) -> bool:
return False
if parts[0] and self.is_ignored_dir(parts[0], at_root=True) and len(parts) > 1:
return True
return any(self.is_ignored_dir(part) for part in parts[:-1])
if any(self.is_ignored_dir(part) for part in parts[:-1]):
return True
return self._under_ignored_dir_path(parts[:-1])

def _under_ignored_dir_path(self, dir_parts: list[str]) -> bool:
"""A multi-segment entry matches consecutive directory names at any depth.

Path-level only: :meth:`prune` sees one name at a time, so the walk descends into
`bin/Debug/` and this rejects every file under it.
"""
if not self._folded_dir_paths:
return False
folded = [_fold(p) for p in dir_parts]
for seq in self._folded_dir_paths:
n = len(seq)
if any(tuple(folded[i:i + n]) == seq for i in range(len(folded) - n + 1)):
return True
return False

def is_ignored_file(self, rel_path: str) -> bool:
"""Does the file's name match an `ignore_files` pattern (ADR-052 §4)?"""
name = _fold(rel_path.rsplit("/", 1)[-1])
exclude, keep = self._folded_file_patterns
return (any(fnmatchcase(name, p) for p in exclude)
and not any(fnmatchcase(name, p) for p in keep))

def is_indexable(self, rel_path: str) -> bool:
"""Is this repo-relative path a source file the scan chunks and embeds?"""
if self._under_ignored_dir(rel_path):
if self._under_ignored_dir(rel_path) or self.is_ignored_file(rel_path):
return False
return _suffix(rel_path) in self.extensions

Expand All @@ -239,7 +356,7 @@ def is_scannable(self, rel_path: str) -> bool:
Broader than :meth:`is_indexable` by exactly the descriptor files, which are
parsed for edges but never chunked or embedded.
"""
if self._under_ignored_dir(rel_path):
if self._under_ignored_dir(rel_path) or self.is_ignored_file(rel_path):
return False
name = rel_path.rsplit("/", 1)[-1]
return (
Expand All @@ -262,7 +379,8 @@ def describe(self) -> str:
f"[scan] config={origin} root={self.root} "
f"ignore_dirs={len(self.ignore_dirs)} "
f"ignore_root_dirs={sorted(self.ignore_root_dirs)} "
f"extensions={len(self.extensions)}"
f"extensions={len(self.extensions)} "
f"ignore_files={len(self.ignore_files)}"
)


Expand Down Expand Up @@ -307,6 +425,7 @@ def scan_policy(start_dir: str | None = None) -> ScanPolicy:
ignore_dirs=_resolve(block, "dirs", DEFAULT_IGNORE_DIRS),
ignore_root_dirs=_resolve(block, "root_dirs", DEFAULT_IGNORE_ROOT_DIRS),
extensions=_resolve_exts(block, DEFAULT_INDEXABLE_EXTS),
ignore_files=_resolve_files(block, DEFAULT_IGNORE_FILES),
index_dir_name=os.path.basename(os.path.normpath(index_dir_setting(key))),
)
_cache[key] = policy
Expand Down
19 changes: 19 additions & 0 deletions tests/fixtures/conformance/apps_script/helpers.expected.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"feature": "apps_script/helpers",
"language": "apps_script",
"tier": "A",
"symbols": [
{"fqn": "getEstimateSheet", "kind": "function"},
{"fqn": "sumColumn", "kind": "function"},
{"fqn": "formatCurrency", "kind": "function"}
],
"edges": [
{"source": "getEstimateSheet", "target": "getActive", "kind": "call"},
{"source": "getEstimateSheet", "target": "getSheetByName", "kind": "call"},
{"source": "sumColumn", "target": "getRange", "kind": "call"},
{"source": "sumColumn", "target": "getLastRow", "kind": "call"},
{"source": "sumColumn", "target": "getValues", "kind": "call"},
{"source": "formatCurrency", "target": "currencySymbol", "kind": "call"},
{"source": "formatCurrency", "target": "toFixed", "kind": "call"}
]
}
Loading
Loading