From cbcbb70cc53a9c76b70564493b22d5b9da88dddc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:03:34 +0000 Subject: [PATCH 1/8] Add ADR-052: scan defaults for Apps Script, .NET output, Razor and credential files Co-Authored-By: Claude --- .../ADR-052-scan-defaults-and-languages.md | 166 ++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 docs/adr/ADR-052-scan-defaults-and-languages.md diff --git a/docs/adr/ADR-052-scan-defaults-and-languages.md b/docs/adr/ADR-052-scan-defaults-and-languages.md new file mode 100644 index 0000000..e39bcd0 --- /dev/null +++ b/docs/adr/ADR-052-scan-defaults-and-languages.md @@ -0,0 +1,166 @@ +# ADR-052: Scan Defaults — Apps Script, .NET Output, Razor as Text, Credential Files + +**Status:** proposed +**Date:** 2026-10-20 +**Branch:** `feature/adr-052-scan-defaults-and-languages` +**Reviewer:** @edb +**Backlog:** [B-015](../backlog.md#b-015) (Apps Script `.gs`), [B-063](../backlog.md#b-063) item 1 only +(`obj`/`bin` defaults), [B-064](../backlog.md#b-064) level 1 only (`.cshtml`/`.razor` as text), +[B-018](../backlog.md#b-018) the scan-policy half only (the output-redaction half is ADR-049) +**Depends on:** ADR-026 (the scan gate and its `extra_*`/bare-key precedence, which this extends) +**Depended on by:** none yet + +## Context + +The scan gate (`src/scan_policy.py`, ADR-026) decides what the indexer looks at. Four gaps in its +defaults were filed separately: + +- **B-015.** Apps Script projects whose clasp `fileExtension` is `gs` are invisible: `.gs` is not an + indexable extension and has no REGISTRY entry. Apps Script is plain JavaScript run in **one shared + global scope**: files have no imports and call each other's top-level functions by name. +- **B-063.** `DEFAULT_IGNORE_DIRS` has no `obj` or `bin`. On AcadeDesignTools, 56 of 147 `.cs` files + (38%) are generated `obj/` output (`*.g.cs`, `*.g.i.cs`, `*AssemblyAttributes.cs`); EganEstimator + has 8. `bin` is a common name outside .NET: Python projects keep real scripts in `bin/*.py`, npm + packages keep their CLI in `bin/*.js`, and Rust keeps binaries in `src/bin/`. +- **B-064.** `.cshtml` (MVC / Razor Pages) and `.razor` (Blazor) are not indexed at all, so "which + view shows the estimate total" has no answer. EganEstimator has 17 views. With no adapter they + would go through `fallback_token_chunker`, a path no test had run end to end. +- **B-018.** Nothing stops a credential file from being chunked, embedded and pasted into an agent's + context. Today's default extensions happen to exclude `.json`, `.env` and `.pem`, so the defaults + are safe by accident; the moment a repo adds `extra_extensions = [".json"]` to index its config, + a committed `service-account.json` goes into the index. + +## Decision + +### 1. Apps Script `.gs` is JavaScript (B-015) + +- `.gs` joins `DEFAULT_INDEXABLE_EXTS` and the REGISTRY. +- **One REGISTRY line is not enough.** `_WebAdapter.parse` looks its grammar up by extension + (`self._grammars[ext]`), so `.gs → JavaScriptAdapter` raises `KeyError` on the first parse. + `adapters/__init__.py` defines a three-line `AppsScriptAdapter(JavaScriptAdapter)` that maps `.gs` + to the JavaScript grammar. The adapter class in `ts_adapter.py` is unchanged. +- **Call resolution under one global scope is left as the ADR-021 order.** Measured on a two-file + fixture (Implementation Log): a call to a top-level function defined once anywhere in the project + resolves through the *unique repo-wide* rule, which is exactly Apps Script's semantics; a call to + a Google service (`SpreadsheetApp.getActive()`) stays unresolved; a name defined in no file stays + unresolved. The one wrong binding is a function name defined in **two** `.gs` files: Apps Script + runs whichever file loads last, everywhere, while the resolver's *same-file* step binds each + caller to its own file's copy. That is a duplicate definition — almost always a bug in the script + — and fixing it needs `call_resolver.py` to know a file's language. It is recorded as a known gap + (an `xfail(strict=True)` test asserting the correct answer), not fixed here. + +### 2. `obj` everywhere; `bin` only under MSBuild's configuration folders (B-063 item 1) + +- `obj` joins `DEFAULT_IGNORE_DIRS` at any depth. It is MSBuild's intermediate directory and the + C/C++ object-file directory of countless Makefiles; it is source in nothing we could find. +- **`bin` does not.** Instead, `ignore_dirs` entries may now be a **multi-segment path**, matched as + consecutive directory names at any depth, and the defaults gain `bin/Debug`, `bin/Release`, + `bin/x64` and `bin/x86`. MSBuild writes `bin//` or `bin///`, + so this catches its output (including `dotnet publish`'s `bin/Release/net8.0/publish/wwwroot/js/`, + which carries copies of every vendored script) and leaves `bin/cli.py`, `bin/run.js` and + `src/bin/main.rs` alone. A custom configuration is one line: `extra_dirs = ["bin/Staging"]`. +- **No generated-file name patterns** (`*.g.cs`, `*.g.i.cs`, `*.Designer.cs`, `*AssemblyAttributes.cs`). + Every generated file in the B-063 sample lives under `obj/`, which now covers it. Outside `obj/` a + generated file is checked in on purpose: `Form1.Designer.cs` carries the WinForms event wiring + (`button1.Click += button1_Click`), and a checked-in `*.g.cs` is often the only definition of types + the rest of the code calls, so dropping it costs the resolver its targets. +- EF Core `Migrations/` stays indexed: it is real source. A repo that finds it noisy adds + `extra_dirs = ["Migrations"]`. + +### 3. `.cshtml` and `.razor` are indexed as text; `.xaml` is not a default (B-064 level 1) + +- `.cshtml` and `.razor` join `DEFAULT_INDEXABLE_EXTS`. They have no adapter, so `chunk_file_ast` + falls back to `fallback_token_chunker` for tier 1 and tiers 2–3 use it as for every file: three + tiers of text chunks, no symbols, no edges. A test drives this end to end (scan → chunk → ingest + with a fake embedder). +- **`.xaml` stays opt-in** (`extra_extensions = [".xaml"]`, covered by the same end-to-end test). + In Razor the behavior lives in the view (`@code` blocks, tag helpers, `@Model.Total`); in WPF it + lives in the code-behind `.cs`, which is already indexed, while the `.xaml` is layout, styles and + resource dictionaries that can run to thousands of lines of noise. And taking a default extension + back out costs a user a full `extensions = [...]` replacement, which freezes every later default, + so the bar for adding one is evidence, not parity. The 11 AcadeDesignTools files are not that yet. +- **Level 2 (a Razor adapter) — grammar status.** The original `tree-sitter-razor` is archived and + does not build with current tree-sitter. A maintained fork by `tris203` is the grammar + nvim-treesitter and nixpkgs use (nixpkgs pinned a 2025-02-17 snapshot). There is **no PyPI wheel** + (`pip download tree-sitter-razor` finds nothing, 2026-10-20), so using it here means vendoring and + building a C grammar, which `CONTRIBUTING`'s no-new-runtime-dependency rule would have to allow. + +### 4. Credential files are never scanned (B-018, scan half) + +A fourth `[ignore]` knob, **`files` / `extra_files`**: `fnmatch` patterns matched against the file +**name** (case-folded, like directory names). A path matching one is neither indexable nor +scannable, whatever its extension. A pattern starting with `!` re-includes names it matches, so an +exception does not require replacing the defaults. A pattern containing `/` raises: these match +names, and `config/secret.json` would silently match nothing. + +Defaults (`DEFAULT_IGNORE_FILES`): + +| Pattern | Catches | +|---|---| +| `.env`, `.env.*` but `!.env.example`, `!.env.sample`, `!.env.template` | dotenv files; the documented templates stay | +| `*credentials*.json` | gcloud `application_default_credentials.json`, the Google API quickstart's `credentials.json` | +| `client_secret*.json` | Google OAuth client downloads (`client_secret_.apps.googleusercontent.com.json`) | +| `*service[-_]account*.json`, `serviceaccount*.json` | `service-account.json`, `my_service_account.json`, Firebase's `serviceAccountKey.json` | +| `*firebase-adminsdk*.json` | Firebase Admin SDK key downloads | +| `*-` + 12 × `[0-9a-f]` + `.json` | the gcloud console's key download, `-<12 hex>.json` | +| `secrets.json` | ASP.NET user-secrets copied into a repo | +| `*.pem`, `*.key`, `*.p12`, `*.pfx`, `id_rsa`, `id_dsa`, `id_ecdsa`, `id_ed25519` | private keys and keystores | + +Rules kept narrow on purpose: + +- **`*key*.json` is rejected** (the backlog's suggestion): it matches `keyboard.json`, `monkey.json`, + `keymap.json`. The key-download names above are the ones Google and Firebase actually write. +- **`*credentials*` is restricted to `.json`**: `credentials.py` is ordinary source in every auth + library (`google/auth/credentials.py`). +- No content check: the scan gate sees only paths (the watchdog never opens the file). Content is + the output-redaction pass's job (ADR-049). + +## Consequences + +**Better:** +- Apps Script projects with `.gs` files are indexed, with symbols and cross-file call edges. +- .NET repos stop indexing their own build output; on AcadeDesignTools that is 38% of the `.cs`. +- Razor views and Blazor components are searchable as text. +- A credential file stays out of the index even after a repo widens `extensions`. +- `ignore_dirs` can name a directory by its parent, which is how `bin` is handled without touching + `bin/*.py`. + +**Worse:** +- A repo that already has an index will **drop** its `obj/` and MSBuild `bin/` files on the next + incremental run, and gain `.gs`/`.cshtml`/`.razor` files. Both are the intent, but a large `obj/` + can trip the bulk-deletion prompt (`max(50, 20%)`) once; `--prune` accepts it. +- WPF code-behind loses the `InitializeComponent` and named-element fields generated into + `obj/*.g.cs`, so a reference to `OkButton` no longer has an in-index declaration. +- The Apps Script duplicate-definition case binds to the wrong copy (§1). +- A multi-segment `ignore_dirs` entry cannot be pruned during the walk (prune sees one name at a + time); the walk descends into `bin/Debug/` and the path check rejects every file under it. Cheap + for MSBuild output; it would not be for a `node_modules`-sized tree. +- One more knob to document. + +**Neutral:** +- `dirs = []` still means "exclude nothing": the multi-segment defaults live in the same set. +- `.env` files have no indexable suffix (`_suffix(".env") == ""`), so the `.env` patterns guard + `.env.local`-style names under a widened `extensions`, and document intent. + +## Alternatives Considered + +| Option | Why rejected | +|--------|-------------| +| Point `.gs` at the existing `_js` instance | `KeyError` on `_grammars[".gs"]`; mutating the shared class dict from `__init__` would hide the dependency | +| Make the resolver treat all `.gs` files as one module | Needs `call_resolver.py` to know languages, for a case (duplicate top-level names) that is a bug in the script | +| `bin` at any depth | Drops `bin/*.py` scripts and npm CLIs | +| `bin` only when a `*.csproj` sits beside it | The watchdog sees a path and never the disk; the two consumers would disagree, which is what ADR-026 §2 removed | +| Generated-file name patterns | Redundant with `obj`, and wrong for the checked-in generated files outside it (§2) | +| `.xaml` as a default | §3 | +| `*key*.json`, `*credentials*`, `secrets.py` | Too broad; each matches real source or data | +| Content sniffing for keys in the scan gate | The gate is path-only by contract; redaction is ADR-049 | + +## Implementation Log + +- [ ] `scan_policy.py`: `.gs`, `.cshtml`, `.razor`; `obj`, `bin/Debug`, `bin/Release`, `bin/x64`, `bin/x86`; multi-segment `ignore_dirs`; `files`/`extra_files` with `!` exceptions +- [ ] `adapters/__init__.py`: `AppsScriptAdapter` and the `.gs` REGISTRY entry +- [ ] Conformance fixture for `.gs`, scored 1.0/1.0, baseline row +- [ ] Tests: `.gs` scannable + cross-file resolution; `obj`/`bin` incl. Python `bin/*.py`; Razor end to end; credential files and their must-still-index cases; `extra_*`/bare-key behavior for the new knob +- [ ] `src/CLAUDE.md` "The Scan Gate" + +**Notes:** From ddc34d4cd8f6f2f0bc9ab744b9b57ce13c2e90a5 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:18:38 +0000 Subject: [PATCH 2/8] Keep .NET build output and credential files out of the scan; index .gs and Razor scan_policy gains .gs, .cshtml and .razor as default extensions; obj at any depth and MSBuild's bin/Debug, bin/Release, bin/x64, bin/x86 as ignored directories, via multi-segment ignore_dirs entries matched as consecutive directory names; and a files / extra_files knob of file-name patterns that are never scanned, defaulting to common credential files, with ! patterns re-including (.env.example). Co-Authored-By: Claude [ADR-052 src/CLAUDE.md src/scan_policy.py] --- src/CLAUDE.md | 11 +++++- src/scan_policy.py | 91 +++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 95 insertions(+), 7 deletions(-) diff --git a/src/CLAUDE.md b/src/CLAUDE.md index f0862c0..39fb2ad 100644 --- a/src/CLAUDE.md +++ b/src/CLAUDE.md @@ -93,11 +93,18 @@ Model IDs live in `indexer.toml` (`[embeddings]`, `[reranker]`, `[summarization] **`src/scan_policy.py` is the single answer to "does the indexer look at this file?"** (ADR-026 §2). Both consumers call it — `incremental_indexer.scan_disk()`, which walks the tree, and `MCPServer._CodeChangeHandler._is_relevant`, which sees only a path string on each filesystem event. They used to be two hand-mirrored implementations over three module constants, which is how a JavaScript-seeded exclusion list survived on a Python tool long enough to index 510 files of third-party eval corpus from this repo's own `benchmarks/`. -It is a **leaf module** — `os`, `dataclasses`, `functools`, `config`. Do not import a sibling `src` module into it: `config` importing `incremental_indexer` for a constant is the cycle ADR-026 §2 was written around, and it would surface only through the server's function-local import path. +It is a **leaf module** — `os`, `dataclasses`, `functools`, `fnmatch`, `config`. Do not import a sibling `src` module into it: `config` importing `incremental_indexer` for a constant is the cycle ADR-026 §2 was written around, and it would surface only through the server's function-local import path. Two predicates, and the difference matters: `is_indexable()` is source that gets chunked and embedded; `is_scannable()` is that plus the `.csproj`/`.sln`/`compile_commands.json` descriptors, which are parsed for edges only. The watchdog uses the second — using the first meant editing a `.csproj` never triggered a reindex. -Configuration is the `[ignore]` block. `extra_dirs` / `extra_root_dirs` / `extra_extensions` **add** to the defaults; the bare `dirs` / `root_dirs` / `extensions` keys **replace** them; setting both spellings of one knob raises. Defaults live in `scan_policy.py` and are deliberately **not** duplicated into the shipped `indexer.toml` — a bare key there would freeze this repo's gate and silently drop every future default. +Configuration is the `[ignore]` block. `extra_dirs` / `extra_root_dirs` / `extra_extensions` / `extra_files` **add** to the defaults; the bare `dirs` / `root_dirs` / `extensions` / `files` keys **replace** them; setting both spellings of one knob raises. Defaults live in `scan_policy.py` and are deliberately **not** duplicated into the shipped `indexer.toml` — a bare key there would freeze this repo's gate and silently drop every future default. + +What the defaults decide, and why (ADR-052): + +- **`.gs` is JavaScript** (Apps Script), through `AppsScriptAdapter` in `adapters/__init__.py` — a subclass only because the JS adapter picks its grammar by extension. Its cross-file calls resolve by the ADR-021 unique-name rule, which matches Apps Script's one global scope; a function defined in two `.gs` files is a known wrong binding. +- **`.cshtml` / `.razor` are text**: no adapter, so `fallback_token_chunker` for every tier, no symbols or edges. `.xaml` is opt-in (`extra_extensions = [".xaml"]`): taking a default extension back out costs a full `extensions` replacement. +- **`obj` is ignored at any depth; bare `bin` is not.** A `dirs` entry may be a multi-segment path, matched as consecutive directory names at any depth, and `bin/Debug`, `bin/Release`, `bin/x64`, `bin/x86` catch MSBuild output while `bin/*.py` and npm `bin/*.js` stay. A custom build configuration is `extra_dirs = ["bin/Staging"]`. These entries cannot be pruned during the walk (prune sees one name), so the path check rejects the files instead — don't use one for a `node_modules`-sized tree. Generated `*.Designer.cs` / `*.g.cs` outside `obj/` stay: checked-in generated code is often the only definition the rest of the code calls. EF Core `Migrations/` is real source and stays; `extra_dirs = ["Migrations"]` if it is noise. +- **`files` is file-name patterns** (`fnmatch`, case-folded), never scanned whatever `extensions` says: credential files — `.env*`, `*credentials*.json`, service-account and Firebase key downloads, `secrets.json`, `*.pem`/`*.key`/`*.p12`/`*.pfx`, `id_rsa`-style keys. A `!` pattern re-includes (`.env.example` ships that way), so one exception never needs the defaults replaced. A pattern with a `/` raises. The gate sees only paths; key *content* is the MCP output-redaction pass's job, not this module's. The scan is anchored to the directory holding `indexer.toml`, and a scan launched from a subdirectory is refused rather than resolving root-only exclusions against the wrong tree. A run that would delete more than `max(50, 20%)` of the index asks first (`--prune` to skip); unattended callers log and skip instead. diff --git a/src/scan_policy.py b/src/scan_policy.py index 24ba5ec..ca583a9 100644 --- a/src/scan_policy.py +++ b/src/scan_policy.py @@ -13,7 +13,7 @@ behaviour. Both consumers now call the same predicate. **Leaf placement is load-bearing, not cosmetic.** This module imports only ``os``, -``dataclasses`` and ``config``. A policy helper living in ``incremental_indexer`` and +``dataclasses``, ``fnmatch`` and ``config``. A policy helper living in ``incremental_indexer`` and imported by ``config`` deadlocks at import time, and would surface only through the server's function-local import path — the hardest possible place to see it. @@ -26,6 +26,7 @@ import os from dataclasses import dataclass +from fnmatch import fnmatchcase from functools import cached_property from config import find_config_path, index_dir_setting, load_indexer_config @@ -55,6 +56,13 @@ # Editor, tooling and infrastructure state ".continue", ".claude", ".vs", ".vscode", "Modelfiles", ".github", ".firebase", ".idx", "genkit", + + # .NET build output (ADR-052 §2, B-063). `obj` is MSBuild's intermediate tree — + # 38% of AcadeDesignTools' `.cs` files were generated `obj/` output. Bare `bin` is + # NOT here: `bin/*.py` and `bin/*.js` are real scripts in Python and npm projects. + # MSBuild writes `bin//` or `bin///`, so the + # multi-segment entries below match only that (consecutive directory names, any depth). + "obj", "bin/Debug", "bin/Release", "bin/x64", "bin/x86", }) DEFAULT_IGNORE_ROOT_DIRS: frozenset[str] = frozenset({"functions"}) @@ -65,8 +73,23 @@ # statement about code that never runs. DEFAULT_INDEXABLE_EXTS: frozenset[str] = frozenset({ ".py", ".ts", ".tsx", ".js", ".jsx", # Tier-A: JS/TS/Python + ".gs", # Apps Script: the JS adapter (ADR-052 §1) ".cs", # Tier-A: C# ".cpp", ".cc", ".cxx", ".h", ".hpp", # Tier-A: C++ (.h routed to the C++ adapter) + ".cshtml", ".razor", # Razor/Blazor: text only, no adapter (ADR-052 §3) +}) + +# File-name patterns never scanned, whatever the extension knob says (ADR-052 §4, B-018): +# files that typically hold credentials. `fnmatch` over the case-folded file name; a `!` +# pattern re-includes. Narrow on purpose — `*key*.json` would take `keyboard.json`, and +# `*credentials*` without `.json` would take `credentials.py`, which is ordinary source. +DEFAULT_IGNORE_FILES: frozenset[str] = frozenset({ + ".env", ".env.*", "!.env.example", "!.env.sample", "!.env.template", + "*credentials*.json", "client_secret*.json", + "*service[-_]account*.json", "serviceaccount*.json", "*firebase-adminsdk*.json", + "*-" + "[0-9a-f]" * 12 + ".json", # gcloud key download: -<12 hex>.json + "secrets.json", + "*.pem", "*.key", "*.p12", "*.pfx", "id_rsa", "id_dsa", "id_ecdsa", "id_ed25519", }) # Project descriptor files: edges only, no chunking or embedding. Not operator knobs — @@ -169,6 +192,22 @@ def _resolve_exts(block: dict, defaults: frozenset[str]) -> frozenset[str]: return frozenset(e.lower() for e in resolved) +def _resolve_files(block: dict, defaults: frozenset[str]) -> frozenset[str]: + """Same precedence as :func:`_resolve`, for file-name patterns. + + A pattern with a ``/`` raises: these match the file's *name*, so + ``config/secret.json`` would match nothing and exclude nothing, silently. + """ + resolved = _resolve(block, "files", defaults) + bad = sorted(p for p in resolved if "/" in p or "\\" in p) + if bad: + raise ValueError( + f"[ignore].files patterns match a file name, not a path: {bad}. " + f"Exclude a directory with `extra_dirs` instead." + ) + return resolved + + # --------------------------------------------------------------------------- # The policy object # --------------------------------------------------------------------------- @@ -182,6 +221,7 @@ class ScanPolicy: ignore_dirs: frozenset[str] # excluded at any depth ignore_root_dirs: frozenset[str] # excluded at the repository root only extensions: frozenset[str] # chunked + embedded + ignore_files: frozenset[str] = DEFAULT_IGNORE_FILES # file-name patterns; `!` re-includes project_exts: frozenset[str] = PROJECT_EXTS project_files: frozenset[str] = PROJECT_FILES index_dir_name: str = ".code-index" # [indexer].index_dir's last component (ADR-042) @@ -197,6 +237,21 @@ def _folded_dirs(self) -> frozenset[str]: def _folded_root_dirs(self) -> frozenset[str]: return frozenset(_fold(d) for d in self.ignore_root_dirs) + @cached_property + def _folded_dir_paths(self) -> tuple[tuple[str, ...], ...]: + """Multi-segment `ignore_dirs` entries (`bin/Debug`), as folded name tuples.""" + return tuple( + tuple(_fold(p) for p in d.strip("/").split("/")) + for d in self.ignore_dirs if "/" in d.strip("/") + ) + + @cached_property + def _folded_file_patterns(self) -> tuple[tuple[str, ...], tuple[str, ...]]: + """(exclude, re-include) file-name patterns, folded once.""" + pats = [_fold(p) for p in self.ignore_files] + return (tuple(p for p in pats if not p.startswith("!")), + tuple(p[1:] for p in pats if p.startswith("!"))) + def is_ignored_dir(self, name: str, *, at_root: bool = False) -> bool: """Whether ``os.walk`` should prune a directory entry named ``name``. @@ -225,11 +280,35 @@ def _under_ignored_dir(self, rel_path: str) -> bool: return False if parts[0] and self.is_ignored_dir(parts[0], at_root=True) and len(parts) > 1: return True - return any(self.is_ignored_dir(part) for part in parts[:-1]) + if any(self.is_ignored_dir(part) for part in parts[:-1]): + return True + return self._under_ignored_dir_path(parts[:-1]) + + def _under_ignored_dir_path(self, dir_parts: list[str]) -> bool: + """A multi-segment entry matches consecutive directory names at any depth. + + Path-level only: :meth:`prune` sees one name at a time, so the walk descends into + `bin/Debug/` and this rejects every file under it. + """ + if not self._folded_dir_paths: + return False + folded = [_fold(p) for p in dir_parts] + for seq in self._folded_dir_paths: + n = len(seq) + if any(tuple(folded[i:i + n]) == seq for i in range(len(folded) - n + 1)): + return True + return False + + def is_ignored_file(self, rel_path: str) -> bool: + """Does the file's name match an `ignore_files` pattern (ADR-052 §4)?""" + name = _fold(rel_path.rsplit("/", 1)[-1]) + exclude, keep = self._folded_file_patterns + return (any(fnmatchcase(name, p) for p in exclude) + and not any(fnmatchcase(name, p) for p in keep)) def is_indexable(self, rel_path: str) -> bool: """Is this repo-relative path a source file the scan chunks and embeds?""" - if self._under_ignored_dir(rel_path): + if self._under_ignored_dir(rel_path) or self.is_ignored_file(rel_path): return False return _suffix(rel_path) in self.extensions @@ -239,7 +318,7 @@ def is_scannable(self, rel_path: str) -> bool: Broader than :meth:`is_indexable` by exactly the descriptor files, which are parsed for edges but never chunked or embedded. """ - if self._under_ignored_dir(rel_path): + if self._under_ignored_dir(rel_path) or self.is_ignored_file(rel_path): return False name = rel_path.rsplit("/", 1)[-1] return ( @@ -262,7 +341,8 @@ def describe(self) -> str: f"[scan] config={origin} root={self.root} " f"ignore_dirs={len(self.ignore_dirs)} " f"ignore_root_dirs={sorted(self.ignore_root_dirs)} " - f"extensions={len(self.extensions)}" + f"extensions={len(self.extensions)} " + f"ignore_files={len(self.ignore_files)}" ) @@ -307,6 +387,7 @@ def scan_policy(start_dir: str | None = None) -> ScanPolicy: ignore_dirs=_resolve(block, "dirs", DEFAULT_IGNORE_DIRS), ignore_root_dirs=_resolve(block, "root_dirs", DEFAULT_IGNORE_ROOT_DIRS), extensions=_resolve_exts(block, DEFAULT_INDEXABLE_EXTS), + ignore_files=_resolve_files(block, DEFAULT_IGNORE_FILES), index_dir_name=os.path.basename(os.path.normpath(index_dir_setting(key))), ) _cache[key] = policy From 9b7d011c172be22a0a6eadd5980f619ccc6f536c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:18:38 +0000 Subject: [PATCH 3/8] Parse Apps Script .gs with the JavaScript grammar and test the ADR-052 defaults AppsScriptAdapter maps .gs to the JS grammar: a bare REGISTRY line raised KeyError because the web adapter picks its grammar by extension. Adds a two-file conformance fixture (1.0/1.0, new apps_script baseline row), the .gs extension in conformance_eval, and tests for cross-file resolution, the obj/bin rules, Razor through the fallback chunker end to end, and the credential-file patterns. Co-Authored-By: Claude [ADR-052 src/adapters/__init__.py] --- benchmarks/conformance/baseline.json | 28 ++ src/adapters/__init__.py | 13 + .../apps_script/helpers.expected.json | 19 + .../conformance/apps_script/helpers.gs | 18 + .../apps_script/menu.expected.json | 21 + .../fixtures/conformance/apps_script/menu.gs | 18 + tests/test_scan_defaults_adr052.py | 455 ++++++++++++++++++ tools/conformance_eval.py | 1 + 8 files changed, 573 insertions(+) create mode 100644 tests/fixtures/conformance/apps_script/helpers.expected.json create mode 100644 tests/fixtures/conformance/apps_script/helpers.gs create mode 100644 tests/fixtures/conformance/apps_script/menu.expected.json create mode 100644 tests/fixtures/conformance/apps_script/menu.gs create mode 100644 tests/test_scan_defaults_adr052.py diff --git a/benchmarks/conformance/baseline.json b/benchmarks/conformance/baseline.json index 2d76c1e..b8a6e77 100644 --- a/benchmarks/conformance/baseline.json +++ b/benchmarks/conformance/baseline.json @@ -1,4 +1,18 @@ { + "apps_script": { + "edges_all": { + "precision": 1.0, + "recall": 1.0 + }, + "edges_call": { + "precision": 1.0, + "recall": 1.0 + }, + "symbols": { + "precision": 1.0, + "recall": 1.0 + } + }, "cpp": { "edges_all": { "precision": 1.0, @@ -27,6 +41,20 @@ "recall": 1.0 } }, + "l5x": { + "edges_all": { + "precision": 1.0, + "recall": 1.0 + }, + "edges_call": { + "precision": 1.0, + "recall": 1.0 + }, + "symbols": { + "precision": 1.0, + "recall": 1.0 + } + }, "python": { "edges_all": { "precision": 1.0, diff --git a/src/adapters/__init__.py b/src/adapters/__init__.py index c8f8227..b07f627 100644 --- a/src/adapters/__init__.py +++ b/src/adapters/__init__.py @@ -23,9 +23,21 @@ from adapters.cpp_adapter import CppAdapter from adapters.l5x_adapter import L5xAdapter + +class AppsScriptAdapter(JavaScriptAdapter): + """Apps Script `.gs` (ADR-052 §1): plain JavaScript under the JS grammar. + + Not a bare REGISTRY line because `_WebAdapter.parse` looks the grammar up by + extension, so `.gs` -> `JavaScriptAdapter` raises KeyError on the first parse. + """ + extensions = frozenset({".gs"}) + _grammars = {".gs": JavaScriptAdapter._grammars[".js"]} + + _python = PythonAdapter() _ts = TypeScriptAdapter() _js = JavaScriptAdapter() +_gs = AppsScriptAdapter() _cs = CSharpAdapter() _cpp = CppAdapter() _l5x = L5xAdapter() @@ -36,6 +48,7 @@ ".tsx": _ts, ".js": _js, ".jsx": _js, + ".gs": _gs, ".cs": _cs, ".csproj": _cs, ".sln": _cs, diff --git a/tests/fixtures/conformance/apps_script/helpers.expected.json b/tests/fixtures/conformance/apps_script/helpers.expected.json new file mode 100644 index 0000000..56225a9 --- /dev/null +++ b/tests/fixtures/conformance/apps_script/helpers.expected.json @@ -0,0 +1,19 @@ +{ + "feature": "apps_script/helpers", + "language": "apps_script", + "tier": "A", + "symbols": [ + {"fqn": "getEstimateSheet", "kind": "function"}, + {"fqn": "sumColumn", "kind": "function"}, + {"fqn": "formatCurrency", "kind": "function"} + ], + "edges": [ + {"source": "getEstimateSheet", "target": "getActive", "kind": "call"}, + {"source": "getEstimateSheet", "target": "getSheetByName", "kind": "call"}, + {"source": "sumColumn", "target": "getRange", "kind": "call"}, + {"source": "sumColumn", "target": "getLastRow", "kind": "call"}, + {"source": "sumColumn", "target": "getValues", "kind": "call"}, + {"source": "formatCurrency", "target": "currencySymbol", "kind": "call"}, + {"source": "formatCurrency", "target": "toFixed", "kind": "call"} + ] +} diff --git a/tests/fixtures/conformance/apps_script/helpers.gs b/tests/fixtures/conformance/apps_script/helpers.gs new file mode 100644 index 0000000..f789f13 --- /dev/null +++ b/tests/fixtures/conformance/apps_script/helpers.gs @@ -0,0 +1,18 @@ +// Feature: Apps Script — the other half of menu.gs. Calls back into menu.gs by name. + +function getEstimateSheet() { + return SpreadsheetApp.getActive().getSheetByName("Estimates"); +} + +function sumColumn(sheet, column) { + const values = sheet.getRange(2, column, sheet.getLastRow() - 1, 1).getValues(); + let total = 0; + for (const row of values) { + total += row[0]; + } + return total; +} + +function formatCurrency(amount) { + return currencySymbol() + amount.toFixed(2); +} diff --git a/tests/fixtures/conformance/apps_script/menu.expected.json b/tests/fixtures/conformance/apps_script/menu.expected.json new file mode 100644 index 0000000..3d26dec --- /dev/null +++ b/tests/fixtures/conformance/apps_script/menu.expected.json @@ -0,0 +1,21 @@ +{ + "feature": "apps_script/menu", + "language": "apps_script", + "tier": "A", + "symbols": [ + {"fqn": "onOpen", "kind": "function"}, + {"fqn": "recalculate", "kind": "function"}, + {"fqn": "currencySymbol", "kind": "function"} + ], + "edges": [ + {"source": "onOpen", "target": "getUi", "kind": "call"}, + {"source": "onOpen", "target": "createMenu", "kind": "call"}, + {"source": "onOpen", "target": "addItem", "kind": "call"}, + {"source": "onOpen", "target": "addToUi", "kind": "call"}, + {"source": "recalculate", "target": "getEstimateSheet", "kind": "call"}, + {"source": "recalculate", "target": "sumColumn", "kind": "call"}, + {"source": "recalculate", "target": "getRange", "kind": "call"}, + {"source": "recalculate", "target": "setValue", "kind": "call"}, + {"source": "recalculate", "target": "formatCurrency", "kind": "call"} + ] +} diff --git a/tests/fixtures/conformance/apps_script/menu.gs b/tests/fixtures/conformance/apps_script/menu.gs new file mode 100644 index 0000000..a7fb591 --- /dev/null +++ b/tests/fixtures/conformance/apps_script/menu.gs @@ -0,0 +1,18 @@ +// Feature: Apps Script — top-level functions in one shared global scope. No imports: +// recalculate() calls getEstimateSheet()/sumColumn() from helpers.gs by name, and +// helpers.gs calls currencySymbol() back. Google services (SpreadsheetApp) are globals +// the project does not define. + +function onOpen() { + SpreadsheetApp.getUi().createMenu("Estimates").addItem("Recalculate", "recalculate").addToUi(); +} + +function recalculate() { + const sheet = getEstimateSheet(); + const total = sumColumn(sheet, 3); + sheet.getRange("F1").setValue(formatCurrency(total)); +} + +function currencySymbol() { + return "$"; +} diff --git a/tests/test_scan_defaults_adr052.py b/tests/test_scan_defaults_adr052.py new file mode 100644 index 0000000..2744123 --- /dev/null +++ b/tests/test_scan_defaults_adr052.py @@ -0,0 +1,455 @@ +""" +test_scan_defaults_adr052.py — the ADR-052 scan defaults. + +Four changes to what the gate admits, each pinned with the cases that must still pass +through it: + +- `.gs` (Apps Script) is JavaScript, and its cross-file calls resolve under one global scope; +- `obj/` and MSBuild's `bin//` are build output, `bin/*.py` is not; +- `.cshtml` / `.razor` are indexed as text, through `fallback_token_chunker`, end to end; +- credential files are never scanned, whatever `extensions` says. + +No model load: the tokenizer and the embedder are fakes. +""" +from __future__ import annotations + +import os +import re + +import numpy as np +import pytest + +import scan_policy as sp + + +@pytest.fixture(autouse=True) +def _clean_cache(): + sp.reset() + yield + sp.reset() + + +def _repo(tmp_path, toml_body: str | None = None) -> str: + if toml_body is not None: + (tmp_path / "indexer.toml").write_text(toml_body, encoding="utf-8") + return str(tmp_path) + + +def _write(root, rel: str, text: str = "") -> None: + path = os.path.join(str(root), *rel.split("/")) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as fh: + fh.write(text) + + +# --------------------------------------------------------------------------- +# §1 — Apps Script `.gs` +# --------------------------------------------------------------------------- + +_GS_DIR = os.path.join(os.path.dirname(__file__), "fixtures", "conformance", "apps_script") + + +def test_gs_is_indexable_and_scannable(tmp_path): + policy = sp.scan_policy(_repo(tmp_path)) + assert policy.is_indexable("src/Code.gs") + assert policy.is_scannable("src/Code.gs") + assert policy.is_indexable("Utils.GS") # the suffix is lower-cased + + +def test_gs_parses_with_the_javascript_grammar(): + """A bare `.gs -> JavaScriptAdapter` line raised KeyError: the grammar is per extension.""" + from adapters import get_adapter + from adapters.ts_adapter import JavaScriptAdapter + + adapter = get_adapter(".gs") + assert isinstance(adapter, JavaScriptAdapter) + src = b"function a() { return b(); }\nfunction b() { return 1; }\n" + result = adapter.parse("Code.gs", src) + assert {s.fqn for s in result.symbols} == {"Code.gs::a", "Code.gs::b"} + assert adapter.analyze_tags("Code.gs", src, result.symbols) is not None + + +def _resolve(files: dict[str, str], tmp_path) -> dict[tuple[str, str], str | None]: + """Index `files` through the real parse → upsert → resolve path; {(source, callee): resolved}.""" + from ast_chunker import parse_file + from call_resolver import resolve_call_edges + from db import CodeDB + + db = CodeDB(str(tmp_path / "graph.db")) + try: + for rel, text in files.items(): + result = parse_file(rel, text) + db.upsert_file(rel, "hash:" + rel, result.symbols, result.edges) + resolve_call_edges(db) + rows = db._conn.execute( + "SELECT source_fqn, target, resolved_target FROM edges WHERE kind = 'CALLS'" + ).fetchall() + finally: + db.close() + return {(src, tgt): resolved for src, tgt, resolved in rows} + + +def _fixture_sources() -> dict[str, str]: + out = {} + for name in ("menu.gs", "helpers.gs"): + with open(os.path.join(_GS_DIR, name), encoding="utf-8") as fh: + out[name] = fh.read() + return out + + +def test_gs_cross_file_calls_resolve_under_one_global_scope(tmp_path): + """Apps Script has no imports; every top-level function is one global. A name defined + once in the project is exactly the callee, which is the ADR-021 unique-repo-wide rule.""" + got = _resolve(_fixture_sources(), tmp_path) + + # Bare calls across files, both directions. + assert got[("menu.gs::recalculate", "getEstimateSheet")] == "helpers.gs::getEstimateSheet" + assert got[("menu.gs::recalculate", "sumColumn")] == "helpers.gs::sumColumn" + assert got[("menu.gs::recalculate", "formatCurrency")] == "helpers.gs::formatCurrency" + assert got[("helpers.gs::formatCurrency", "currencySymbol")] == "menu.gs::currencySymbol" + + # Google services and built-ins are defined nowhere in the project: unresolved. + for key in [("menu.gs::onOpen", "getUi"), ("menu.gs::onOpen", "addToUi"), + ("helpers.gs::getEstimateSheet", "getActive"), + ("helpers.gs::sumColumn", "getRange"), ("helpers.gs::formatCurrency", "toFixed")]: + assert got[key] is None, key + + resolved = {k: v for k, v in got.items() if v is not None} + assert len(resolved) == 4, resolved # nothing else bound + + +def test_gs_resolved_edges_read_as_verified(tmp_path): + """The resolved edges clear the ADR-008 verdict floor, as any unique-name resolution.""" + from ast_chunker import parse_file + from call_resolver import resolve_call_edges + from db import CodeDB, EDGE_CONFIDENCE_FLOOR, effective_confidence + + db = CodeDB(str(tmp_path / "graph.db")) + try: + for rel, text in _fixture_sources().items(): + result = parse_file(rel, text) + db.upsert_file(rel, "hash:" + rel, result.symbols, result.edges) + resolve_call_edges(db) + rows = db._conn.execute( + "SELECT candidate, confidence FROM edges " + "WHERE kind = 'CALLS' AND resolved_target IS NOT NULL" + ).fetchall() + finally: + db.close() + assert len(rows) == 4 + assert all(effective_confidence(bool(c), conf) >= EDGE_CONFIDENCE_FLOOR for c, conf in rows) + + +def test_gs_a_name_defined_in_two_files_is_ambiguous_without_a_same_file_copy(tmp_path): + """Two definitions and a caller in a third file: no rule can pick one, so none is bound.""" + got = _resolve({ + "a.gs": "function log(x) { return x; }\n", + "b.gs": "function log(x) { return x + 1; }\n", + "c.gs": "function run() { return log(1); }\n", + }, tmp_path) + assert got[("c.gs::run", "log")] is None + + +@pytest.mark.xfail(strict=True, reason=( + "ADR-052 §1 known gap: Apps Script runs whichever duplicate loads last, everywhere; " + "the resolver's same-file step binds a.gs's caller to a.gs's copy")) +def test_gs_a_duplicate_definition_is_not_bound_to_the_callers_own_file(tmp_path): + got = _resolve({ + "a.gs": "function log(x) { return x; }\nfunction run() { return log(1); }\n", + "b.gs": "function log(x) { return x + 1; }\n", + }, tmp_path) + assert got[("a.gs::run", "log")] is None + + +# --------------------------------------------------------------------------- +# §2 — .NET build output +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("path", [ + "App/obj/Debug/net8.0/App.AssemblyInfo.cs", + "App/obj/Debug/net48/MainWindow.g.cs", + "obj/Release/App.g.i.cs", + "App/bin/Debug/net8.0/App.cs", + "App/bin/Release/net8.0/publish/wwwroot/js/site.js", + "App/bin/x64/Release/Native.cs", + "App/bin/x86/Debug/Native.cs", + "App/Bin/DEBUG/Generated.cs", # case-folded, like every dir rule + "Obj/Foo.cs", +]) +def test_dotnet_build_output_is_excluded(tmp_path, path): + policy = sp.scan_policy(_repo(tmp_path)) + assert not policy.is_indexable(path) + assert not policy.is_scannable(path) + + +@pytest.mark.parametrize("path", [ + "bin/cli.py", # Python console scripts + "tools/bin/run.py", + "bin/www.js", # npm package CLI + "src/bin/main.py", + "bin/debug_helpers.py", # a file, not a Debug directory + "Debug/Tracing.cs", # Debug without bin above it + "App/Views/Home/Release.cshtml", + "App/Form1.Designer.cs", # checked-in generated code stays (§2) + "App/Properties/Resources.Designer.cs", + "App/Generated/Client.g.cs", + "App/Migrations/20241001_Init.cs", # EF migrations are real source + "App/bin/App.csproj", # bin/ itself is not excluded +]) +def test_real_source_near_build_output_is_kept(tmp_path, path): + policy = sp.scan_policy(_repo(tmp_path)) + assert policy.is_scannable(path) + + +def test_the_walk_skips_build_output_and_keeps_bin_scripts(tmp_path): + """Through `scan_disk`, not just the predicate: prune cannot see `bin/Debug` as one + name, so the walk descends and the path check must catch every file under it.""" + import incremental_indexer as ii + + root = _repo(tmp_path) + for rel in ["bin/cli.py", "App/Program.cs", "App/obj/Debug/net8.0/App.g.cs", + "App/bin/Debug/net8.0/App.cs", "App/bin/Release/net8.0/publish/wwwroot/js/site.js", + "App/Migrations/20241001_Init.cs"]: + _write(root, rel, "// placeholder\n") + assert set(ii.scan_disk(root, quiet=True)) == { + "bin/cli.py", "App/Program.cs", "App/Migrations/20241001_Init.cs"} + + +def test_a_custom_configuration_is_one_extra_dirs_line(tmp_path): + policy = sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_dirs = ["bin/Staging", "Migrations"]\n')) + assert not policy.is_indexable("App/bin/Staging/net8.0/App.cs") + assert not policy.is_indexable("App/Migrations/20241001_Init.cs") + assert not policy.is_indexable("App/bin/Debug/App.cs") # defaults survive + assert policy.is_indexable("bin/cli.py") + + +def test_bare_dirs_still_means_exclude_nothing(tmp_path): + """The multi-segment defaults live in the same set, so `dirs = []` drops them too.""" + policy = sp.scan_policy(_repo(tmp_path, "[ignore]\ndirs = []\n")) + assert policy.is_indexable("App/bin/Debug/App.cs") + assert policy.is_indexable("App/obj/App.g.cs") + + +def test_a_multi_segment_entry_matches_consecutive_names_only(tmp_path): + policy = sp.scan_policy(_repo(tmp_path)) + assert policy.is_indexable("bin/tools/Debug/x.py") # not consecutive + assert policy.is_indexable("Debug/bin/x.py") # wrong order + + +# --------------------------------------------------------------------------- +# §3 — Razor and Blazor as text +# --------------------------------------------------------------------------- + +def test_razor_and_blazor_are_indexable_xaml_is_opt_in(tmp_path): + policy = sp.scan_policy(_repo(tmp_path)) + assert policy.is_indexable("Views/Home/Index.cshtml") + assert policy.is_scannable("Views/Home/Index.cshtml") # the watchdog's predicate + assert policy.is_indexable("Components/OrderGrid.razor") + assert not policy.is_scannable("MainWindow.xaml") + + sp.reset() + policy = sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_extensions = [".xaml"]\n')) + assert policy.is_indexable("MainWindow.xaml") + assert policy.is_indexable("Views/Home/Index.cshtml") # defaults survive + + +class _FakeTokenizer: + """Words and whitespace runs as tokens; decode is concatenation. Enough for the + chunker's budget arithmetic, with no download.""" + + def encode(self, text, add_special_tokens=False): + return re.findall(r"\S+|\s+", text) + + def decode(self, tokens): + return "".join(tokens) + + +_CSHTML = """@model EganEstimator.Models.Estimate +@{ + ViewData["Title"] = "Estimate"; +} +

@Model.Name

+ +@foreach (var line in Model.Lines) +{ + +} +
@line.Description@line.Cost.ToString("C")
+

Estimate total: @Model.Total.ToString("C")

+Edit +""" + +_RAZOR = """@page "/orders" +@inject IOrderService Orders + + + +@code { + private List items = new(); + + protected override async Task OnInitializedAsync() + { + items = await Orders.ListAsync(); + } +} +""" + + +def test_razor_files_go_through_the_fallback_path_end_to_end(tmp_path, monkeypatch): + """B-064 level 1: scanned, chunked into every tier, embedded, written — no symbols, + no edges, no crash. `run_incremental` is the real build, with a fake embedder.""" + import core + import incremental_indexer as ii + from ast_chunker import parse_file + from core import embed_dimension + from db import CodeDB + from stable_id import TIER_CONFIGS + + monkeypatch.setattr(core.jina_tokenizer, "_tokenizer", _FakeTokenizer()) + dim = embed_dimension() + monkeypatch.setattr(core, "embed_batch", lambda texts, batch_size=32: np.array( + [[float((hash(t) >> (8 * i)) & 0xFF) + 1.0 for i in range(dim)] for t in texts], + dtype=np.float32)) + monkeypatch.setattr(ii, "summarization_enabled", lambda: False) + + repo = tmp_path / "repo" + repo.mkdir() + _write(repo, "Views/Estimates/Details.cshtml", _CSHTML) + _write(repo, "Components/Orders.razor", _RAZOR) + _write(repo, "MainWindow.xaml", "\n") # not a default + index = tmp_path / "index" + index.mkdir() + monkeypatch.setattr(ii, "INDEX_DIR", str(index)) + monkeypatch.setattr(ii, "DB_PATH", str(index / "graph.db")) + + scanned = ii.scan_disk(str(repo), quiet=True) + assert set(scanned) == {"Views/Estimates/Details.cshtml", "Components/Orders.razor"} + + for rel, text in [("Views/Estimates/Details.cshtml", _CSHTML), ("Components/Orders.razor", _RAZOR)]: + result = parse_file(rel, text) + assert result.symbols == [] and result.edges == [] + tiers = ii.chunk_all_tiers(rel, text) + assert set(tiers) == {name for name, _, _ in TIER_CONFIGS} + assert all(tiers[name] for name in tiers), {n: len(c) for n, c in tiers.items()} + + ii.run_incremental(str(repo), interactive=False) + + with CodeDB(str(index / "graph.db")) as db: + rows = db._conn.execute( + "SELECT f.path, c.tier, c.text FROM chunks c JOIN files f ON f.id = c.file_id" + ).fetchall() + n_symbols = db._conn.execute("SELECT COUNT(*) FROM symbols").fetchone()[0] + n_edges = db._conn.execute("SELECT COUNT(*) FROM edges").fetchone()[0] + assert n_symbols == 0 and n_edges == 0 + by_file: dict[str, set[int]] = {} + for path, tier, _ in rows: + by_file.setdefault(path, set()).add(tier) + assert set(by_file) == {"Views/Estimates/Details.cshtml", "Components/Orders.razor"} + assert all(len(tiers) == len(TIER_CONFIGS) for tiers in by_file.values()), by_file + assert any("Estimate total" in content for _, _, content in rows) + + +# --------------------------------------------------------------------------- +# §4 — credential files +# --------------------------------------------------------------------------- + +# Widened so the exclusion is what decides, not the extension list. +_WIDE = '[ignore]\nextra_extensions = [".json", ".pem", ".key", ".p12", ".pfx", ".local", ".production"]\n' + + +@pytest.mark.parametrize("path", [ + ".env.local", + "config/.env.production", + "credentials.json", + "secrets/application_default_credentials.json", + "client_secret_1234.apps.googleusercontent.com.json", + "service-account.json", + "deploy/my_service_account.json", + "serviceAccountKey.json", + "gantt-app-firebase-adminsdk-abcde-0123456789.json", + "egan-estimator-0123456789ab.json", + "secrets.json", + "certs/server.pem", + "certs/server.key", + "certs/Signing.PFX", + "certs/client.p12", + "Credentials.JSON", +]) +def test_credential_files_are_never_scanned(tmp_path, path): + policy = sp.scan_policy(_repo(tmp_path, _WIDE)) + assert not policy.is_indexable(path) + assert not policy.is_scannable(path) + + +@pytest.mark.parametrize("path", [ + "keyboard.json", # why `*key*.json` was rejected + "monkey.json", + "keymap.json", + "package-lock.json", + "config.json", + "src/auth/credentials.py", # `*credentials*` is .json only + "src/secrets_manager.py", + "src/service_account.py", + "src/keys.ts", + "fixtures/build-12345.json", # not 12 hex digits +]) +def test_files_that_only_look_like_credentials_are_kept(tmp_path, path): + policy = sp.scan_policy(_repo(tmp_path, _WIDE)) + assert policy.is_indexable(path) + + +def test_env_templates_are_re_included(tmp_path): + policy = sp.scan_policy(_repo(tmp_path)) + for name in (".env.example", ".env.sample", ".env.template"): + assert not policy.is_ignored_file(name) + assert policy.is_ignored_file(".env") + assert policy.is_ignored_file("app/.env.local") + + +def test_the_walk_skips_credential_files(tmp_path): + import incremental_indexer as ii + + root = _repo(tmp_path, _WIDE) + _write(root, "config.json", "{}\n") + _write(root, "service-account.json", '{"private_key": "FAKE"}\n') + _write(root, "certs/dev.pem", + "-----BEGIN PRIVATE KEY-----\nFAKE\n-----END PRIVATE KEY-----\n") + assert set(ii.scan_disk(root, quiet=True)) == {"config.json"} + + +def test_extra_files_extends_the_defaults(tmp_path): + policy = sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_files = ["local_settings.py"]\n')) + assert not policy.is_indexable("app/local_settings.py") + assert policy.is_ignored_file("credentials.json") # defaults survive + + +def test_a_bang_pattern_re_includes_one_default(tmp_path): + """An exception without replacing the defaults (and freezing them).""" + policy = sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_files = ["!secrets.json"]\n')) + assert not policy.is_ignored_file("secrets.json") + assert policy.is_ignored_file("credentials.json") + + +def test_bare_files_replaces_the_defaults(tmp_path): + policy = sp.scan_policy(_repo(tmp_path, '[ignore]\nfiles = []\n')) + assert policy.ignore_files == frozenset() + assert not policy.is_ignored_file("credentials.json") + + +def test_files_follows_the_shared_precedence_rules(tmp_path): + with pytest.raises(ValueError, match="extra_files"): + sp.scan_policy(_repo(tmp_path, '[ignore]\nfiles = ["a"]\nextra_files = ["b"]\n')) + sp.reset() + with pytest.raises(ValueError, match="list of strings"): + sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_files = "secrets.json"\n')) + + +def test_a_file_pattern_with_a_path_raises(tmp_path): + with pytest.raises(ValueError, match="file name, not a path"): + sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_files = ["config/secret.json"]\n')) + + +def test_no_config_yields_the_file_defaults(tmp_path): + policy = sp.scan_policy(_repo(tmp_path)) + assert policy.ignore_files == sp.DEFAULT_IGNORE_FILES + assert "ignore_files=" in policy.describe() diff --git a/tools/conformance_eval.py b/tools/conformance_eval.py index e833fab..d71da11 100644 --- a/tools/conformance_eval.py +++ b/tools/conformance_eval.py @@ -56,6 +56,7 @@ ".tsx": "typescript", ".js": "javascript", ".jsx": "javascript", + ".gs": "apps_script", # ADR-052: the JS adapter under Apps Script's extension ".cs": "csharp", ".cpp": "cpp", ".cc": "cpp", From 609b58ffebb3cdb755b722581fdea765768ad544 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:50:39 +0000 Subject: [PATCH 4/8] Record ADR-052 measurements, deviations and open questions Co-Authored-By: Claude [ADR-052] --- .../ADR-052-scan-defaults-and-languages.md | 56 +++++++++++++++++-- 1 file changed, 51 insertions(+), 5 deletions(-) diff --git a/docs/adr/ADR-052-scan-defaults-and-languages.md b/docs/adr/ADR-052-scan-defaults-and-languages.md index e39bcd0..78fa2c5 100644 --- a/docs/adr/ADR-052-scan-defaults-and-languages.md +++ b/docs/adr/ADR-052-scan-defaults-and-languages.md @@ -157,10 +157,56 @@ Rules kept narrow on purpose: ## Implementation Log -- [ ] `scan_policy.py`: `.gs`, `.cshtml`, `.razor`; `obj`, `bin/Debug`, `bin/Release`, `bin/x64`, `bin/x86`; multi-segment `ignore_dirs`; `files`/`extra_files` with `!` exceptions -- [ ] `adapters/__init__.py`: `AppsScriptAdapter` and the `.gs` REGISTRY entry -- [ ] Conformance fixture for `.gs`, scored 1.0/1.0, baseline row -- [ ] Tests: `.gs` scannable + cross-file resolution; `obj`/`bin` incl. Python `bin/*.py`; Razor end to end; credential files and their must-still-index cases; `extra_*`/bare-key behavior for the new knob -- [ ] `src/CLAUDE.md` "The Scan Gate" +- [x] `scan_policy.py`: `.gs`, `.cshtml`, `.razor`; `obj`, `bin/Debug`, `bin/Release`, `bin/x64`, `bin/x86`; multi-segment `ignore_dirs`; `files`/`extra_files` with `!` exceptions. Still a leaf module: the one new import is stdlib `fnmatch`. +- [x] `adapters/__init__.py`: `AppsScriptAdapter` and the `.gs` REGISTRY entry +- [x] Conformance fixture `tests/fixtures/conformance/apps_script/{menu,helpers}.gs`: 1.0/1.0 on symbols (6), all edges (16) and call edges (16); new `apps_script` row in `benchmarks/conformance/baseline.json` +- [x] Tests: `tests/test_scan_defaults_adr052.py` (67 cases, 1 of them a strict xfail) — `.gs` scannable, parsed, and resolved across files; `obj`/`bin` including Python `bin/*.py` and npm `bin/*.js`, through `scan_disk` too; Razor end to end through `run_incremental` with a fake tokenizer and embedder; credential files and the look-alikes that must still be indexed; `extra_files`/`files`/`!`/both-keys/bad-value behavior +- [x] `src/CLAUDE.md` "The Scan Gate" **Notes:** + +- 2026-10-20 — **`.gs` needed an adapter subclass, not a REGISTRY line.** Found before writing + it: `_WebAdapter.parse` and `analyze_tags` index `self._grammars[ext]`. The subclass lives in + `adapters/__init__.py`, so `ts_adapter.py` (graph session) is untouched. +- 2026-10-20 — **Apps Script resolution, measured** (fixture: `menu.gs` + `helpers.gs`, real + `parse_file` → `CodeDB.upsert_file` → `resolve_call_edges`, CPU, no model): 16 call edges, + **4 resolved, all correct** — `recalculate → getEstimateSheet / sumColumn / formatCurrency` + (menu → helpers) and `formatCurrency → currencySymbol` (helpers → menu), each by the + unique-repo-wide rule; the 12 service and built-in calls (`getUi`, `getActive`, `getRange`, + `toFixed`, …) stay unresolved; **0 wrong**. The resolved edges carry no producer confidence + (`confidence` NULL, `candidate` 0), so they read as verified (effective ≥ the 0.5 floor), the same + grade as any unique-name resolution in JS. A name defined in two files with the caller in a third + stays unresolved; with the caller in one of the two, the same-file step binds it to its own copy, + which Apps Script would not do — pinned as `xfail(strict=True)`. +- 2026-10-20 — **Resolution harness not used for `.gs`.** `tools/resolution_eval.py` gates each + language on the typed regime *beating* the hint-stripped baseline. Apps Script has no imports or + receiver types, so the two regimes are identical by construction and a `.gs` row would fail that + gate for a reason that says nothing about correctness. The cross-file measurement is a pytest + instead. +- 2026-10-20 — **Razor end to end.** `.cshtml` and `.razor` through `scan_disk` → `run_incremental` + with a fake tokenizer (words and whitespace) and a fake embedder: both files get chunks in all + three tiers, zero symbols, zero edges, no crash; `.xaml` in the same tree is skipped by default. + `parse_file` returns an empty result for an extension with no adapter, and `chunk_file_ast` + falls straight to `fallback_token_chunker`. +- 2026-10-20 — **Outside this ADR's file list:** `tools/conformance_eval.py`, one line (`".gs": + "apps_script"` in `_LANG_BY_EXT`). Fixture discovery matches `stem + ext` against that table, + so without it the `.gs` fixture is an orphan `expected.json` and the eval raises. +- 2026-10-20 — **Not regenerated:** the README conformance table (`--write-readme`) now lacks the + `apps_script` row. `README.md` belongs to the symbols session (ADR-050) on this round; run + `python tools/conformance_eval.py --write-readme` after both merge. The "Defaults as of this + writing" comment in `indexer.toml`'s `[ignore]` block was already stale (it lists `indexer`, + `public`, `mocks`, which are not defaults) and now also misses this ADR's additions; that file + is not edited this round. + +**Open questions for @edb:** + +1. **Removing one default.** `files` has `!` exceptions, but `dirs` and `extensions` still need a + full replacement to drop a single default (e.g. a repo that wants `obj/` indexed). Worth a + `!` form for those knobs too? +2. **`.xaml`.** Left opt-in (§3). If the B-063 build on AcadeDesignTools shows XAML answers + questions the code-behind can't, flip it. +3. **Apps Script duplicates.** Is the duplicate-definition wrong binding worth a language-aware + rule in `call_resolver.py` (no same-file step for `.gs`), or is it fine as a known gap? +4. **gcloud key-name pattern.** `*-<12 hex>.json` also matches any hyphenated name ending in 12 hex + digits (a content-hashed build manifest). Acceptable loss, or drop it and rely on the other + service-account patterns? From a1e12d4521c2655e668ba0b254d2f72136fe300c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:52:03 +0000 Subject: [PATCH 5/8] Record ADR-052 gate results Co-Authored-By: Claude [ADR-052] --- docs/adr/ADR-052-scan-defaults-and-languages.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/adr/ADR-052-scan-defaults-and-languages.md b/docs/adr/ADR-052-scan-defaults-and-languages.md index 78fa2c5..b1487e5 100644 --- a/docs/adr/ADR-052-scan-defaults-and-languages.md +++ b/docs/adr/ADR-052-scan-defaults-and-languages.md @@ -198,6 +198,15 @@ Rules kept narrow on purpose: `public`, `mocks`, which are not defaults) and now also misses this ADR's additions; that file is not edited this round. +- 2026-10-20 — **Gates** (Linux cloud container, Python 3.12, CPU, PyPI `torch` 2.14.1 run on CPU, + no model, huggingface.co unreachable so `HF_HUB_OFFLINE=1`): flake8 clean; `pytest --co` 775 + collected, no errors (704 on `871b55c`); `pytest` **724 passed, 48 failed, 2 skipped, 1 xfailed** + against `871b55c`'s 654 / 48 / 2 / 0. The 48 failures are the **same 48 test ids on both**, all + from tests that load the embedder's tokenizer from HuggingFace. The +70 passed are this ADR's 66 + passing tests and 4 conformance parametrizations over the new fixtures. + `conformance_eval --check-baseline`, `resolution_eval --check-baseline` and + `l5x_fixture_teeth --check` exit 0. No MCP tool changed, so no Inspector run. + **Open questions for @edb:** 1. **Removing one default.** `files` has `!` exceptions, but `dirs` and `extensions` still need a From 56f116134d380c05fba6cfed15edb824fbf23a58 Mon Sep 17 00:00:00 2001 From: Ethan Blauw Date: Tue, 20 Oct 2026 16:00:39 -0500 Subject: [PATCH 6/8] Leave an Apps Script duplicate unresolved instead of binding the caller's own copy Every .gs file shares one global scope, and the copy that loads last wins everywhere, so a bare call with several candidates skips the same-file and import-scoped steps. Member calls and .js callers are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) [ADR-052 src/call_resolver.py] --- src/call_resolver.py | 8 ++++ tests/test_scan_defaults_adr052.py | 64 ++++++++++++++++++++++++++++-- 2 files changed, 69 insertions(+), 3 deletions(-) diff --git a/src/call_resolver.py b/src/call_resolver.py index bc92b7a..01dbb36 100644 --- a/src/call_resolver.py +++ b/src/call_resolver.py @@ -40,6 +40,12 @@ ADR-044 (``member_call`` NULL, and every edge of an adapter that does not compute it) resolve as before, except that a ``.py`` source skips import-scoping, which it never reached before. +**Apps Script** (ADR-052) — every ``.gs`` file shares one global scope and has no imports, and +when two files define the same top-level function, whichever loads last wins everywhere. So a +bare call from a ``.gs`` file with several candidates skips the positional steps (same-file and +import-scoped) and stays unresolved: binding the caller to its own file's copy would be a guess. +A member call (``this.x()``) still gets the same-file step. + The pass recomputes **all** CALLS edges each run, so a resolution is demoted back to NULL if a later-added symbol makes its name ambiguous — a stale resolution never outlives the uniqueness that justified it (ADR-021 §3). @@ -151,6 +157,8 @@ def resolve_call_edges(db) -> dict: elif len(cands) == 1: resolved = cands[0][0] stats["resolved"] += 1 + elif len(cands) > 1 and src_file.endswith(".gs") and member != 1: + stats["ambiguous"] += 1 # one global scope: no file is nearer (ADR-052) elif len(cands) > 1: src_id = path_to_id.get(src_file) same = [fqn for fqn, fid in cands if fid == src_id] diff --git a/tests/test_scan_defaults_adr052.py b/tests/test_scan_defaults_adr052.py index 2744123..68671ca 100644 --- a/tests/test_scan_defaults_adr052.py +++ b/tests/test_scan_defaults_adr052.py @@ -150,10 +150,9 @@ def test_gs_a_name_defined_in_two_files_is_ambiguous_without_a_same_file_copy(tm assert got[("c.gs::run", "log")] is None -@pytest.mark.xfail(strict=True, reason=( - "ADR-052 §1 known gap: Apps Script runs whichever duplicate loads last, everywhere; " - "the resolver's same-file step binds a.gs's caller to a.gs's copy")) def test_gs_a_duplicate_definition_is_not_bound_to_the_callers_own_file(tmp_path): + # Apps Script runs whichever duplicate loads last, everywhere, so neither copy is + # provably the callee (ADR-052 §1, call_resolver's Apps Script rule). got = _resolve({ "a.gs": "function log(x) { return x; }\nfunction run() { return log(1); }\n", "b.gs": "function log(x) { return x + 1; }\n", @@ -161,6 +160,26 @@ def test_gs_a_duplicate_definition_is_not_bound_to_the_callers_own_file(tmp_path assert got[("a.gs::run", "log")] is None +def test_js_keeps_the_same_file_step_for_a_duplicate(tmp_path): + # The rule is Apps Script's: a module-scoped .js caller still binds to its own copy. + got = _resolve({ + "a.js": "function log(x) { return x; }\nfunction run() { return log(1); }\n", + "b.js": "function log(x) { return x + 1; }\n", + }, tmp_path) + assert got[("a.js::run", "log")] == "a.js::log" + + +def test_gs_member_call_keeps_the_same_file_step(tmp_path): + # this.total() names a method of the caller's own class, which the shared global + # scope does not change. + got = _resolve({ + "a.gs": ("class Sheet {\n total() { return 1; }\n" + " show() { return this.total(); }\n}\n"), + "b.gs": "class Other {\n total() { return 2; }\n}\n", + }, tmp_path) + assert got[("a.gs::Sheet.show", "total")] == "a.gs::Sheet.total" + + # --------------------------------------------------------------------------- # §2 — .NET build output # --------------------------------------------------------------------------- @@ -453,3 +472,42 @@ def test_no_config_yields_the_file_defaults(tmp_path): policy = sp.scan_policy(_repo(tmp_path)) assert policy.ignore_files == sp.DEFAULT_IGNORE_FILES assert "ignore_files=" in policy.describe() + + +# --------------------------------------------------------------------------- +# `!` drops one default from dirs, root_dirs and extensions (ADR-052 §5) +# --------------------------------------------------------------------------- + +def test_a_bang_dir_drops_one_default_and_keeps_the_rest(tmp_path): + policy = sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_dirs = ["!obj", "Migrations"]\n')) + assert policy.is_indexable("App/obj/Generated.cs") # obj dropped + assert not policy.is_indexable("App/bin/Debug/App.cs") # other defaults stay + assert not policy.is_indexable("node_modules/x/index.js") + assert not policy.is_indexable("App/Migrations/0001_Init.cs") # and the addition holds + + +def test_a_bang_drops_a_multi_segment_dir_folded(tmp_path): + policy = sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_dirs = ["!BIN/debug"]\n')) + assert policy.is_indexable("App/bin/Debug/App.cs") + assert not policy.is_indexable("App/bin/Release/App.cs") + + +def test_a_bang_root_dir_drops_one_default(tmp_path): + policy = sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_root_dirs = ["!functions"]\n')) + assert policy.is_indexable("functions/src/index.ts") + + +def test_a_bang_extension_drops_one_default(tmp_path): + policy = sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_extensions = ["!.cshtml"]\n')) + assert not policy.is_indexable("Views/Home/Index.cshtml") + assert policy.is_indexable("Pages/Counter.razor") + + +def test_a_bang_that_names_no_default_raises(tmp_path): + with pytest.raises(ValueError, match="names no built-in default"): + sp.scan_policy(_repo(tmp_path, '[ignore]\nextra_dirs = ["!objj"]\n')) + + +def test_a_bang_in_the_bare_key_raises(tmp_path): + with pytest.raises(ValueError, match="nothing to remove"): + sp.scan_policy(_repo(tmp_path, '[ignore]\ndirs = ["!obj"]\n')) From b7930e242551ba9f2859dc5c2aa1510241f584b7 Mon Sep 17 00:00:00 2001 From: Ethan Blauw Date: Tue, 20 Oct 2026 16:00:40 -0500 Subject: [PATCH 7/8] Let a ! entry drop one default from extra_dirs, extra_root_dirs and extra_extensions Co-Authored-By: Claude Opus 5.5 (1M context) [ADR-052 src/CLAUDE.md src/scan_policy.py] --- .../ADR-052-scan-defaults-and-languages.md | 45 +++++++++++++----- src/CLAUDE.md | 2 +- src/scan_policy.py | 46 +++++++++++++++++-- 3 files changed, 76 insertions(+), 17 deletions(-) diff --git a/docs/adr/ADR-052-scan-defaults-and-languages.md b/docs/adr/ADR-052-scan-defaults-and-languages.md index b1487e5..fcdd801 100644 --- a/docs/adr/ADR-052-scan-defaults-and-languages.md +++ b/docs/adr/ADR-052-scan-defaults-and-languages.md @@ -46,8 +46,10 @@ defaults were filed separately: unresolved. The one wrong binding is a function name defined in **two** `.gs` files: Apps Script runs whichever file loads last, everywhere, while the resolver's *same-file* step binds each caller to its own file's copy. That is a duplicate definition — almost always a bug in the script - — and fixing it needs `call_resolver.py` to know a file's language. It is recorded as a known gap - (an `xfail(strict=True)` test asserting the correct answer), not fixed here. + — and fixing it needs `call_resolver.py` to know a file's language. **Fixed at @edb's request + (open question 3):** a bare call from a `.gs` file with several candidates skips the same-file + and import-scoped steps and stays unresolved. A member call (`this.total()`) keeps the + same-file step, and `.js` callers are unchanged. The strict xfail is now a passing test. ### 2. `obj` everywhere; `bin` only under MSBuild's configuration folders (B-063 item 1) @@ -115,6 +117,17 @@ Rules kept narrow on purpose: - No content check: the scan gate sees only paths (the watchdog never opens the file). Content is the output-redaction pass's job (ADR-049). +### 5. `!` drops one default from `dirs`, `root_dirs` and `extensions` (added at @edb's request) + +`extra_dirs = ["!obj"]` removes that one built-in default and keeps the rest, so a repo that wants +`obj/` indexed no longer has to replace the whole list (which freezes every later default). +- Works in `extra_dirs`, `extra_root_dirs` and `extra_extensions`, alongside additions in the same + list. Matching is case-folded, and a multi-segment default (`!bin/Debug`) can be dropped too. +- A `!name` that matches no default raises: it would remove nothing, and a typo would keep + indexing what the user meant to drop. +- A `!` in a bare key (`dirs = ["!obj"]`) raises: the bare key already replaces the defaults. +- `files` keeps its own `!` meaning (§4): a pattern that re-includes matching names. + ## Consequences **Better:** @@ -209,13 +222,21 @@ Rules kept narrow on purpose: **Open questions for @edb:** -1. **Removing one default.** `files` has `!` exceptions, but `dirs` and `extensions` still need a - full replacement to drop a single default (e.g. a repo that wants `obj/` indexed). Worth a - `!` form for those knobs too? -2. **`.xaml`.** Left opt-in (§3). If the B-063 build on AcadeDesignTools shows XAML answers - questions the code-behind can't, flip it. -3. **Apps Script duplicates.** Is the duplicate-definition wrong binding worth a language-aware - rule in `call_resolver.py` (no same-file step for `.gs`), or is it fine as a known gap? -4. **gcloud key-name pattern.** `*-<12 hex>.json` also matches any hyphenated name ending in 12 hex - digits (a content-hashed build manifest). Acceptable loss, or drop it and rely on the other - service-account patterns? +Answered by @edb, 2026-10-20: + +1. ~~**Removing one default.**~~ Yes: built, Decision §5. +2. ~~**`.xaml`.**~~ Stays opt-in until the B-063 build on AcadeDesignTools shows XAML answers + questions the code-behind can't. +3. ~~**Apps Script duplicates.**~~ Yes, a rule: built, Decision §1. +4. ~~**gcloud key-name pattern.**~~ Keep `*-<12 hex>.json`; losing a content-hashed manifest is + acceptable. + +- [x] Follow-up after @edb answered the open questions (2026-10-20, local, not the cloud session): + - `call_resolver.py`: the Apps Script duplicate rule (§1). Three tests replace the strict xfail: + the `.gs` duplicate stays unresolved, a `.js` duplicate still binds to its own file, and a `.gs` + member call keeps the same-file step. + - `scan_policy.py`: `!` removal in `extra_dirs`/`extra_root_dirs`/`extra_extensions` (§5), six + tests. + - Gates on Windows, Python 3.14.4, CPU, tokenizer cached (`HF_HUB_OFFLINE=1`): `pytest` 782 + passed, 1 skipped, 0 failed; flake8 on `src/` clean; `conformance_eval.py` and + `resolution_eval.py --check-baseline` exit 0. diff --git a/src/CLAUDE.md b/src/CLAUDE.md index 39fb2ad..8d6608a 100644 --- a/src/CLAUDE.md +++ b/src/CLAUDE.md @@ -97,7 +97,7 @@ It is a **leaf module** — `os`, `dataclasses`, `functools`, `fnmatch`, `config Two predicates, and the difference matters: `is_indexable()` is source that gets chunked and embedded; `is_scannable()` is that plus the `.csproj`/`.sln`/`compile_commands.json` descriptors, which are parsed for edges only. The watchdog uses the second — using the first meant editing a `.csproj` never triggered a reindex. -Configuration is the `[ignore]` block. `extra_dirs` / `extra_root_dirs` / `extra_extensions` / `extra_files` **add** to the defaults; the bare `dirs` / `root_dirs` / `extensions` / `files` keys **replace** them; setting both spellings of one knob raises. Defaults live in `scan_policy.py` and are deliberately **not** duplicated into the shipped `indexer.toml` — a bare key there would freeze this repo's gate and silently drop every future default. +Configuration is the `[ignore]` block. `extra_dirs` / `extra_root_dirs` / `extra_extensions` / `extra_files` **add** to the defaults; the bare `dirs` / `root_dirs` / `extensions` / `files` keys **replace** them; setting both spellings of one knob raises. A `!name` entry in `extra_dirs` / `extra_root_dirs` / `extra_extensions` drops that one default and keeps the rest (`!` in `extra_files` re-includes matching names instead). Defaults live in `scan_policy.py` and are deliberately **not** duplicated into the shipped `indexer.toml` — a bare key there would freeze this repo's gate and silently drop every future default. What the defaults decide, and why (ADR-052): diff --git a/src/scan_policy.py b/src/scan_policy.py index ca583a9..67cd23f 100644 --- a/src/scan_policy.py +++ b/src/scan_policy.py @@ -136,7 +136,31 @@ def _as_str_list(value: object, key: str) -> list[str]: return list(value) -def _resolve(block: dict, key: str, defaults: frozenset[str]) -> frozenset[str]: +def _extend(defaults: frozenset[str], values: list[str], key: str) -> frozenset[str]: + """``extra_*``: add each entry to the defaults, and remove each ``!name`` from them. + + ``!`` drops one built-in default without a bare-key replacement, which would freeze + every later default (ADR-052). A ``!name`` that matches no default raises: it removes + nothing, and a typo there would otherwise keep indexing what the user meant to drop. + Matching is folded like every directory comparison. + """ + removed: set[str] = set() + for value in values: + if not value.startswith("!"): + continue + hits = {d for d in defaults if _fold(d) == _fold(value[1:])} + if not hits: + raise ValueError( + f"[ignore].{key}: `{value}` names no built-in default, so it removes " + f"nothing. `!` drops a default; to add an entry, leave the `!` off." + ) + removed |= hits + added = {v for v in values if not v.startswith("!")} + return (frozenset(defaults) - removed) | frozenset(added) + + +def _resolve(block: dict, key: str, defaults: frozenset[str], *, + removable: bool = True) -> frozenset[str]: """Apply the ADR-026 §1 precedence table to one knob. ============ ============= ========================================= @@ -152,6 +176,10 @@ def _resolve(block: dict, key: str, defaults: frozenset[str]) -> frozenset[str]: absent key: ``extra_dirs = []`` is a no-op and ``dirs = []`` means "exclude nothing" — legal, destructive, and exactly what someone asking for it asked for. + With ``removable`` (every knob but ``files``, whose ``!`` is a pattern re-include), a + ``!name`` entry in ``extra_*`` removes that one default (:func:`_extend`). In the bare + key it raises: the bare key already replaces the defaults, so there is nothing to remove. + Both keys set is an error rather than a silent precedence because there is no reading of "extend *and* replace" a user could have meant, and guessing is how the defect this ADR removes was born in the first place. @@ -168,9 +196,19 @@ def _resolve(block: dict, key: str, defaults: frozenset[str]) -> frozenset[str]: ) if bare is not None: - return frozenset(_as_str_list(bare, key)) + values = _as_str_list(bare, key) + negated = sorted(v for v in values if v.startswith("!")) + if removable and negated: + raise ValueError( + f"[ignore].{key} replaces the defaults, so {negated} has nothing to " + f"remove. Use `{extra_key}` to drop one default and keep the rest." + ) + return frozenset(values) if extra is not None: - return frozenset(defaults) | frozenset(_as_str_list(extra, extra_key)) + values = _as_str_list(extra, extra_key) + if removable: + return _extend(defaults, values, extra_key) + return frozenset(defaults) | frozenset(values) return frozenset(defaults) @@ -198,7 +236,7 @@ def _resolve_files(block: dict, defaults: frozenset[str]) -> frozenset[str]: A pattern with a ``/`` raises: these match the file's *name*, so ``config/secret.json`` would match nothing and exclude nothing, silently. """ - resolved = _resolve(block, "files", defaults) + resolved = _resolve(block, "files", defaults, removable=False) bad = sorted(p for p in resolved if "/" in p or "\\" in p) if bad: raise ValueError( From e80f46f6cdd9cf2ef218ef6e0bd02c7c62259054 Mon Sep 17 00:00:00 2001 From: Ethan Blauw Date: Tue, 20 Oct 2026 16:43:22 -0500 Subject: [PATCH 8/8] ADR-052 accepted Co-Authored-By: Claude Opus 5.5 (1M context) [ADR-052] --- docs/adr/ADR-052-scan-defaults-and-languages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/ADR-052-scan-defaults-and-languages.md b/docs/adr/ADR-052-scan-defaults-and-languages.md index fcdd801..7fbd2b5 100644 --- a/docs/adr/ADR-052-scan-defaults-and-languages.md +++ b/docs/adr/ADR-052-scan-defaults-and-languages.md @@ -1,6 +1,6 @@ # ADR-052: Scan Defaults — Apps Script, .NET Output, Razor as Text, Credential Files -**Status:** proposed +**Status:** accepted **Date:** 2026-10-20 **Branch:** `feature/adr-052-scan-defaults-and-languages` **Reviewer:** @edb