From 2688ffc6545466f8a4a1900d0d21d485045abdd2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:03:03 +0000 Subject: [PATCH 1/6] Add ADR-051: change keys that ignore line endings and slice positions Proposed design for B-054, B-048 fix 1 and B-034 tier 1. Co-Authored-By: Claude [ADR-051] --- docs/adr/ADR-051-stable-change-keys.md | 160 +++++++++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 docs/adr/ADR-051-stable-change-keys.md diff --git a/docs/adr/ADR-051-stable-change-keys.md b/docs/adr/ADR-051-stable-change-keys.md new file mode 100644 index 0000000..13dd8c1 --- /dev/null +++ b/docs/adr/ADR-051-stable-change-keys.md @@ -0,0 +1,160 @@ +# ADR-051: Change Keys That Ignore Line Endings and Slice Positions + +**Status:** proposed +**Date:** 2026-10-20 +**Branch:** `feature/adr-051-stable-change-keys` +**Reviewer:** @edb +**Backlog:** [B-054](../backlog.md#b-054) (line endings), [B-048](../backlog.md#b-048) fix 1 only +(the `(Part i/N)` key), [B-034](../backlog.md#b-034) tier 1 only (reuse unchanged tier-1 vectors) +**Depends on:** [ADR-045](./ADR-045-summary-cache-key-without-line-numbers.md), whose +`summary_cache_key` and one-time re-key this extends; [ADR-046](./ADR-046-atomic-migrations-and-reported-failures.md) +for how a migration commits; [ADR-033](./ADR-033-chunker-version.md), whose version this ADR +deliberately does not bump (§2); the unmerged ADR-029, whose §1 design §3 follows. +**Depended on by:** none yet. + +## Context + +An incremental run decides what to redo with three keys. Each one changes in a case where the +code did not: + +| Key | What it is | Changes when the code didn't | +|---|---|---| +| file key | `files.content_hash`: MD5 of the file's raw bytes (`source.md5_file`) | the file's line endings flip (B-054) | +| summary key | `chunk_summaries.text_hash`: `summary_cache_key(chunk text)` (ADR-045) | a tier-2/3 slice's `(Part i/N)` changes because a window was added or dropped elsewhere in the file (B-048) | +| vector | none: a modified file's vectors are all removed and re-embedded | any edit anywhere in the file (B-034) | + +**B-054.** With `core.autocrlf=true` the same file is LF in one checkout and CRLF in another. +InventoryApp-V2's fork reprocessed 239 files, 27 of them identical apart from line endings. + +**Does chunk text keep `\r` today? No.** The backlog left this unverified. Both readers already +normalize: `WorkingTreeSource.read` opens the file in text mode, whose universal newlines turn +`\r\n` and a lone `\r` into `\n`, and `GitCommitSource.read` goes through `source._decode`, which +does the same replacement by hand (ADR-042). So the chunks, the summary keys and the embedded text +of an LF file and its CRLF twin are already identical, and only the file key differs. +`tests/test_change_keys_eol.py` pins this. The consequence is that a CRLF flip costs a re-embed of +the file, never a re-summarize, and that fixing the file key alone fixes B-054. + +**B-048.** `fallback_token_chunker` writes `Scope: (Part {idx + 1}/{total})` into every +slice it makes: every tier-2/3 slice, and the parts of an oversized tier-1 symbol. ADR-045 kept +that line in the key. So an edit that adds or drops one window changes `N` in every slice of the +file, and every slice misses the cache even where its code did not move. These are the long +prompts (about 1,500 and 4,000 tokens). Repacking after an insert, the other half of B-048, is out +of scope here: fixes 2 and 3 change what is retrieved and need a re-measure on a GPU. + +**B-034, tier 1.** A tier-1 chunk's id is `stable_id(tier, file, scope)`, and its summary is +already reused when its code didn't change (ADR-045). Its vector is not: `_run_incremental` +removes every vector of a modified file before re-indexing it. + +## Decision + +1. **The file key is the MD5 of the bytes with line endings normalized** (`source.md5_file`). + `\r\n` and a lone `\r` both become `\n`, which is exactly what the two readers do, so the file + key changes if and only if the text the indexer reads can change. It still reads in 64 KiB + blocks; a `\r` that ends a block is held back until the next block shows whether `\n` follows. + An LF-only file hashes exactly as before. + + **Git mode keeps the blob id as its file key** (ADR-042 §1). Normalizing it would mean reading + every blob on every listing, and ADR-042 chose the blob id precisely so that listing and + diffing read no file contents. `core.autocrlf` changes the working tree, not blobs, so the + B-054 case does not arise there: a blob whose line endings differ was committed that way, and + that commit is re-embedded once. Its chunks and summary keys are EOL-independent already + (`_decode`), so it is never re-summarized. + +2. **Existing worktree indexes re-key in place, once, without re-indexing anything.** + `index_meta.file_key` records the formula (`"md5-lf"`). On the first run of an index without + it, a modified-looking worktree file whose stored hash equals the **old** raw-byte MD5 of the + file as it is now has not changed at all: its stored hash is rewritten to the new key and it + leaves the diff. Every completed run, a no-op included, records the marker, and a run on a + marked index never computes the old hash again. + + **`CHUNKER_VERSION` is not bumped**, and that departs from the backlog item and the task + prompt. ADR-033 says to bump it "in the same commit as any change to what chunks a file + produces (scope, text, count, ids)", and nothing here changes a chunk: the text never held + `\r`, and §3 changes a key, not a text. A bump would make every existing index print "Results + may mix old and new chunks. Run a full re-index" until it was rebuilt, which is untrue and + would cost each user a full re-embed for nothing. The marker in this section is the "re-key + cleanly" the backlog asked for, and it touches only the files that need it. + + **What the upgrade costs a user:** on the first run, one extra raw MD5 per file that looks + modified, which is a handful of files on a quiet repo and the CRLF files of a fork. Nothing is + re-embedded or re-summarized because of the new formula. Without the in-place re-key, every + file with a `\r` in it would be re-embedded once (summaries cached). + +3. **The summary key and the summarizer's input drop the slice position** (ADR-029 §1). + - `db.summary_input(text)` turns the slice header's `(Part i/N)` into `(Part)`. Only the + `Scope:` line that ends the header, right before `Code:`, is touched, as with ADR-045's + `Lines:`. The summarizer is sent `summary_input(text)` in both passes, so a cached summary + cannot quote a position that is no longer true. + - `summary_cache_key(text)` is the MD5 of `summary_input(text)` with the `Lines:` header + removed (ADR-045). Text without a slice header keys exactly as before, so every tier-1 cache + row stays valid. + - **The embedded and stored text is unchanged.** Positions are still shown in search results, + and the vectors need no re-embedding. + +4. **Existing caches carry over by ADR-045's one-time re-key, not ADR-029's lookup fallback.** + `CodeDB._migrate_summary_keys` now copies each indexed chunk's summary from whichever older + key it has (the raw MD5 before ADR-045, or ADR-045's key) to the new key, once per database, + marked by `index_meta.summary_key = "no-lines-no-part"`. The copy and the marker commit in one + transaction (ADR-046). Old rows stay, as in ADR-045. + + ADR-029 proposed trying the old key on a lookup miss instead. That fallback computes the old + key from the *current* text, so it misses exactly when `N` has changed since the summary was + written, which is the case this ADR is for. The migration re-keys from the text the summary + was written for (`chunks`), so after it a slice whose code did not change hits whatever `N` + becomes, and lookups stay one query. + +5. **B-034, tier 1: an unchanged tier-1 chunk keeps its vector.** See the Implementation Log for + whether it was built and how it holds the three invariants (vectors == rows, ADR-037; the + ADR-040 two-pass build is byte-identical with the overlap on or off; the ADR-038 writer lock). + +## Consequences + +**Better:** +- A checkout whose line endings differ from the one indexed reprocesses only the files whose + code differs. +- Adding or dropping a window keeps the cached summary of every slice whose code did not move. +- Upgrading costs no re-embed and no re-summarize. + +**Worse:** +- A summary cached for one slice is shared with any slice of the same file and parent scope that + has the same code at another position. The summary no longer names a position, so this is the + intent. +- The summarizer now sees `(Part)` where it saw `(Part 2/5)`. Unmeasured, as ADR-029 noted; no + cached summary of this repo was checked for quoting a part number. See Open questions. +- `chunk_summaries` keeps one more old row per slice, as ADR-045's re-key did. + +**Neutral:** +- Repacking (B-048 fixes 2 and 3) is untouched: one inserted line still changes the text of every + slice after it. +- A file whose only change is a lone-`\r` (classic Mac) line ending is treated as unchanged, which + is what the readers already made of it. + +## Alternatives Considered + +| Option | Why rejected | +|--------|-------------| +| Bump `CHUNKER_VERSION` for B-054 (the backlog's and prompt's fix) | No chunk changes, so ADR-033's warning would be false on every existing index and would ask for a full re-index (§2) | +| Fold a key version into `files.content_hash` so every file looks modified | ADR-033 rejected this: a silent full re-embed on the next ordinary run | +| Let CRLF files re-key by simply being re-indexed once | Correct but costs a re-embed of every CRLF file; the in-place re-key is a few lines and costs one extra MD5 per changed-looking file, once | +| Hash the decoded text (`read()` output) instead of normalized bytes | Needs the whole file in memory and a UTF-8 decode per file on every scan; bytes with EOL normalized give the same equivalence for the case at hand | +| Normalize the git-mode key too | Reads every blob on every listing, which ADR-042 §1 was designed to avoid, for a case `core.autocrlf` cannot produce | +| ADR-029's lookup-time fallback to the old key | Misses exactly when `N` changed (§4) | +| Strip `(Part i/N)` from the stored chunk text | Changes the embedded text, so every slice re-embeds, and search results lose the part number | +| Key on the summary input but leave the summarizer input as it is | A summary written for "Part 2/5" could be served for a slice that is now part 3/6 | + +## Open questions for @edb + +1. **Not bumping `CHUNKER_VERSION`** (§2) departs from B-054's stated fix. Agree? +2. **The summarizer input change** (§3) is unmeasured. A quick check on a GPU: summarize a few + tier-2 slices with and without the part number and compare. ADR-045 found that no summary of + 7,258 mentioned a line number; nobody has counted part numbers. + +## Implementation Log + +- [ ] B-054: `md5_file` normalizes line endings; `file_key` marker and in-place re-key +- [ ] B-048 fix 1: `summary_input`, `summary_cache_key`, `_migrate_summary_keys` +- [ ] B-034 tier 1: design and build, or a write-up of why not +- [ ] Tests +- [ ] Gates + +**Notes:** From 09f1a814f7755c93bb761e767d139b01394e1ab9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:29:26 +0000 Subject: [PATCH 2/6] Key files by their bytes with line endings normalized B-054: a checkout whose line endings differ from the indexed one no longer reprocesses those files. md5_file normalizes \r\n and a lone \r to \n, as both readers already do for the text they chunk, so chunks and summary keys were never affected; only the file key was. An LF-only file hashes as before. An index built before this re-keys in place on its first run: a file whose stored hash is the raw MD5 of its current bytes gets the new key without being re-indexed. index_meta.file_key marks the index done. Git mode keeps blob ids as its file key. Co-Authored-By: Claude [ADR-051 src/incremental_indexer.py src/source.py] --- src/incremental_indexer.py | 52 ++++++++- src/source.py | 27 ++++- tests/test_change_keys_eol.py | 213 ++++++++++++++++++++++++++++++++++ 3 files changed, 285 insertions(+), 7 deletions(-) create mode 100644 tests/test_change_keys_eol.py diff --git a/src/incremental_indexer.py b/src/incremental_indexer.py index 314d676..bb1ae27 100644 --- a/src/incremental_indexer.py +++ b/src/incremental_indexer.py @@ -9,11 +9,12 @@ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ CHANGE DETECTION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -For each file on disk we compute an MD5 digest of its raw bytes and compare it -against the `content_hash` column in the SQLite `files` table. MD5 is chosen -over SHA-256 because it is ~3× faster and collision resistance is irrelevant -for change detection (we are not using it for security). Reading the file in -64 KiB blocks keeps memory usage constant regardless of file size. +For each file on disk we compute an MD5 digest of its bytes, with line endings +normalized (ADR-051), and compare it against the `content_hash` column in the +SQLite `files` table. MD5 is chosen over SHA-256 because it is ~3× faster and +collision resistance is irrelevant for change detection (we are not using it for +security). Reading the file in 64 KiB blocks keeps memory usage constant +regardless of file size. Three categories result from the comparison: @@ -355,6 +356,44 @@ def chunker_version_warning(db: CodeDB) -> Optional[str]: "(reindex(changed_files_only=False) or a fresh code-indexer build).") +# ADR-051 (B-054): the folder's file key is an MD5 with line endings normalized. An +# index without this marker holds raw-byte MD5s, which differ for every file with a \r. +FILE_KEY_VERSION = "md5-lf" +FILE_KEY_KEY = "file_key" + + +def rekey_unchanged_files(db: CodeDB, diff: DiffResult, disk: dict[str, str], + source) -> DiffResult: + """Drop from `diff.modified` the files an older index only hashed differently. + + ADR-051 §2: a file whose stored hash is the raw-byte MD5 of its bytes as they are + now has not changed, so its stored hash is rewritten to the new key in place and + nothing is re-indexed. Runs until a completed run records `FILE_KEY_VERSION`; a + git source keys on blob ids, which this does not touch. + """ + if (not diff.modified or getattr(source, "commit", None) is not None + or db.meta_get(FILE_KEY_KEY) == FILE_KEY_VERSION): + return diff + from source import md5_file_raw + stored = dict(db._conn.execute("SELECT path, content_hash FROM files")) + same: list[str] = [] + for path in diff.modified: + try: + if stored.get(path) == md5_file_raw(os.path.join(source.repo_path, path)): + same.append(path) + except OSError: + pass # gone since the scan: left modified, as before + if not same: + return diff + with db._tx() as cur: + cur.executemany("UPDATE files SET content_hash = ? WHERE path = ?", + [(disk[p], p) for p in same]) + print(f" [index] {len(same)} file(s) differed only in how line endings were hashed " + f"(ADR-051); re-keyed in place, not re-indexed.") + kept = set(same) + return diff._replace(modified=[p for p in diff.modified if p not in kept]) + + def _write_index_meta(db: CodeDB, repo_path: str, source=None) -> None: """ADR-025 §4/§5: record run-level freshness facts from the shared chokepoint so CLI and MCP agree about what is indexed. Written on EVERY completed run, @@ -366,6 +405,8 @@ def _write_index_meta(db: CodeDB, repo_path: str, source=None) -> None: db.meta_set("last_indexed_commit", head) db.meta_set("source", getattr(source, "label", "worktree")) db.meta_set("last_verified_at", _now_iso()) + # ADR-051 §2: every file a completed run leaves indexed is keyed the new way. + db.meta_set(FILE_KEY_KEY, FILE_KEY_VERSION) try: db.meta_set("files_total", str(db.stats().get("files", 0))) except Exception: @@ -1170,6 +1211,7 @@ def _run_incremental( print("Scanning files...") disk_hashes = source.list() diff = compute_diff(db, disk_hashes) + diff = rekey_unchanged_files(db, diff, disk_hashes, source) # ADR-026 §5: an ordinary run can irreversibly purge most of an index — that is # the same mechanism that makes the ignore-set migration free. Checked before any diff --git a/src/source.py b/src/source.py index 8a254c0..5600f57 100644 --- a/src/source.py +++ b/src/source.py @@ -7,7 +7,8 @@ exists(rel) whether the path is a file in this source ``WorkingTreeSource`` is the folder, as the indexer has always read it: an -``os.walk`` pruned by ``scan_policy`` and a raw-byte MD5 per file. +``os.walk`` pruned by ``scan_policy`` and an MD5 per file, of its bytes with line +endings normalized (ADR-051). ``GitCommitSource`` is a commit's tree, read from git objects with no checkout. The ref is resolved to a commit once, when the source is made, so a whole run @@ -46,15 +47,37 @@ def _decode(data: bytes) -> str: # The folder # ───────────────────────────────────────────────────────────────────────────── +def _lf(data: bytes) -> bytes: + return data.replace(b"\r\n", b"\n").replace(b"\r", b"\n") + + def md5_file(path: str) -> str: """ - MD5 digest of a file's raw bytes, read in 64 KiB blocks. + MD5 digest of a file's bytes with line endings normalized, read in 64 KiB blocks. + + ADR-051 (B-054): ``\\r\\n`` and a lone ``\\r`` hash as ``\\n``, as ``read()`` turns + them, so a checkout whose line endings differ is not a change. A ``\\r`` that ends + a block waits for the next one, which may start with its ``\\n``. Block-reading keeps memory usage constant for large generated files (e.g. bundled JS). MD5 is fast and sufficient for change detection — we are not using it for authentication or integrity guarantees. """ h = hashlib.md5() + carry = b"" + with open(path, "rb") as fh: + for block in iter(lambda: fh.read(65536), b""): + block = carry + block + carry = b"\r" if block.endswith(b"\r") else b"" + h.update(_lf(block[:-1] if carry else block)) + h.update(_lf(carry)) + return h.hexdigest() + + +def md5_file_raw(path: str) -> str: + """MD5 of a file's raw bytes: the file key before ADR-051, used once to re-key an + older index in place.""" + h = hashlib.md5() with open(path, "rb") as fh: for block in iter(lambda: fh.read(65536), b""): h.update(block) diff --git a/tests/test_change_keys_eol.py b/tests/test_change_keys_eol.py new file mode 100644 index 0000000..a976ca9 --- /dev/null +++ b/tests/test_change_keys_eol.py @@ -0,0 +1,213 @@ +"""ADR-051 (B-054): a line-ending-only difference is not a change. + +The file key is an MD5 of the bytes with line endings normalized, as both readers +already normalize the text they chunk. An index built before it is re-keyed in place, +once. Fake embedder, summarizer off: no model loads. +""" +from __future__ import annotations + +import hashlib +import subprocess +from pathlib import Path + +import numpy as np +import pytest + +import core +import incremental_indexer as ii +import source as source_mod +from core import embed_dimension +from db import CodeDB +from source import GitCommitSource, WorkingTreeSource, md5_file, md5_file_raw + +SAMPLE = ( + "import os\n\n\n" + "def alpha(x):\n return x + 1\n\n\n" + "class Beta:\n def gamma(self, y):\n return alpha(y) * 2\n" +) + + +@pytest.fixture(autouse=True) +def _offline_tokens(monkeypatch): + """Token counts by characters, so chunking needs no tokenizer download.""" + monkeypatch.setattr(core.jina_tokenizer, "count_tokens", lambda s: max(1, len(s) // 4)) + + +def _crlf(text: str) -> str: + return text.replace("\n", "\r\n") + + +def _write(path: Path, text: str) -> None: + path.write_bytes(text.encode("utf-8")) + + +# ── the file key ───────────────────────────────────────────────────────────── + +def test_lf_and_crlf_have_one_file_hash(tmp_path): + _write(tmp_path / "lf.py", SAMPLE) + _write(tmp_path / "crlf.py", _crlf(SAMPLE)) + _write(tmp_path / "cr.py", SAMPLE.replace("\n", "\r")) + assert md5_file(str(tmp_path / "lf.py")) == md5_file(str(tmp_path / "crlf.py")) + assert md5_file(str(tmp_path / "lf.py")) == md5_file(str(tmp_path / "cr.py")) + + +def test_an_lf_file_hashes_as_before(tmp_path): + """Only files with a \\r get a new key, so an LF repo is not re-keyed at all.""" + _write(tmp_path / "lf.py", SAMPLE) + assert md5_file(str(tmp_path / "lf.py")) == hashlib.md5(SAMPLE.encode()).hexdigest() + assert md5_file(str(tmp_path / "lf.py")) == md5_file_raw(str(tmp_path / "lf.py")) + + +@pytest.mark.parametrize("pad", [65535, 65536, 65534]) +def test_a_crlf_split_across_blocks_hashes_as_one_newline(tmp_path, pad): + """The \\r of a \\r\\n can end one 64 KiB block and its \\n start the next.""" + body = "x" * pad + "\n" + "tail\n" + "\n" * 3 + _write(tmp_path / "lf.txt", body) + _write(tmp_path / "crlf.txt", _crlf(body)) + assert md5_file(str(tmp_path / "crlf.txt")) == md5_file(str(tmp_path / "lf.txt")) + _write(tmp_path / "end.txt", "x" * 65535 + "\r") # a lone \r at the very end + _write(tmp_path / "end_lf.txt", "x" * 65535 + "\n") + assert md5_file(str(tmp_path / "end.txt")) == md5_file(str(tmp_path / "end_lf.txt")) + + +def test_a_real_change_still_changes_the_hash(tmp_path): + _write(tmp_path / "a.py", _crlf(SAMPLE)) + _write(tmp_path / "b.py", _crlf(SAMPLE.replace("x + 1", "x + 2"))) + assert md5_file(str(tmp_path / "a.py")) != md5_file(str(tmp_path / "b.py")) + + +# ── chunks and summary keys: already EOL-free before ADR-051 ───────────────── + +def _chunk_keys(rel: str, content: str) -> dict[str, list[tuple[str, str, str]]]: + return {tier: [(c.scope, c.text, ii.chunk_text_hash(c.text)) for c in chunks] + for tier, chunks in ii.chunk_all_tiers(rel, content).items()} + + +def test_chunk_text_never_keeps_a_carriage_return(tmp_path): + """The backlog left this unverified: the folder reader opens in text mode, whose + universal newlines drop every \\r, so chunks and summary keys never differed.""" + _write(tmp_path / "lf.py", SAMPLE) + _write(tmp_path / "crlf.py", _crlf(SAMPLE)) + src = WorkingTreeSource(str(tmp_path)) + crlf = src.read("crlf.py") + assert "\r" not in crlf + assert _chunk_keys("m.py", crlf) == _chunk_keys("m.py", src.read("lf.py")) + + +def _git(repo: Path, *args: str) -> str: + done = subprocess.run( + ["git", "-c", "user.email=t@t", "-c", "user.name=t", "-c", "core.autocrlf=false", *args], + cwd=repo, capture_output=True, text=True) + assert done.returncode == 0, done.stderr + return done.stdout.strip() + + +def test_git_mode_reads_a_crlf_blob_into_the_same_chunks(tmp_path): + """Git mode keys files on blob ids (ADR-042), which ADR-051 leaves alone; what it + chunks and keys summaries on is EOL-free.""" + repo = tmp_path / "repo" + repo.mkdir() + _git(repo, "init", "-q", "-b", "main") + _write(repo / "m.py", SAMPLE) + _git(repo, "add", "m.py") + _git(repo, "commit", "-q", "-m", "lf") + _write(repo / "m.py", _crlf(SAMPLE)) + _git(repo, "commit", "-q", "-am", "crlf") + lf, crlf = GitCommitSource(str(repo), "main~1"), GitCommitSource(str(repo), "main") + try: + assert lf.list(quiet=True)["m.py"] != crlf.list(quiet=True)["m.py"] # blob ids + assert _chunk_keys("m.py", crlf.read("m.py")) == _chunk_keys("m.py", lf.read("m.py")) + finally: + lf.close() + crlf.close() + + +# ── an index ───────────────────────────────────────────────────────────────── + +@pytest.fixture +def env(tmp_path, monkeypatch): + repo = tmp_path / "repo" + (repo / "src").mkdir(parents=True) + _write(repo / "src" / "a.py", SAMPLE) + _write(repo / "src" / "b.py", SAMPLE.replace("alpha", "omega")) + index = tmp_path / "index" + index.mkdir() + embedded: list[str] = [] + + def fake_embed(texts, batch_size=32): + embedded.extend(texts) + rng = np.random.default_rng(len(texts)) + return rng.standard_normal((len(texts), embed_dimension())).astype(np.float32) + + monkeypatch.setattr(core, "embed_batch", fake_embed) + monkeypatch.setattr(ii, "summarization_enabled", lambda: False) + monkeypatch.setattr(ii, "INDEX_DIR", str(index)) + monkeypatch.setattr(ii, "DB_PATH", str(index / "graph.db")) + return repo, str(index / "graph.db"), embedded + + +def _to_crlf(repo: Path) -> None: + for path in (repo / "src").glob("*.py"): + _write(path, _crlf(path.read_bytes().decode("utf-8"))) + + +def test_an_index_built_from_lf_does_not_reprocess_crlf(env, capsys): + repo, db_path, embedded = env + ii.run_incremental(str(repo), interactive=False) + assert embedded + embedded.clear() + _to_crlf(repo) + ii.run_incremental(str(repo), interactive=False) + assert embedded == [] + assert "Nothing changed" in capsys.readouterr().out + + +def _as_pre_adr051(db_path: str, repo: Path) -> None: + """What an index built before ADR-051 holds: raw-byte MD5s, and no marker.""" + with CodeDB(db_path) as db: + for (path,) in db._conn.execute("SELECT path FROM files").fetchall(): + db._conn.execute("UPDATE files SET content_hash = ? WHERE path = ?", + (md5_file_raw(str(repo / path)), path)) + db._conn.execute("DELETE FROM index_meta WHERE key = ?", (ii.FILE_KEY_KEY,)) + + +def _stored(db_path: str) -> dict[str, str]: + with CodeDB(db_path) as db: + return dict(db._conn.execute("SELECT path, content_hash FROM files")) + + +def test_an_older_index_rekeys_once_and_not_again(env, monkeypatch, capsys): + repo, db_path, embedded = env + _to_crlf(repo) + ii.run_incremental(str(repo), interactive=False) + _as_pre_adr051(db_path, repo) + assert _stored(db_path)["src/a.py"] != md5_file(str(repo / "src" / "a.py")) + embedded.clear() + capsys.readouterr() + + ii.run_incremental(str(repo), interactive=False) # the upgrade run + out = capsys.readouterr().out + assert embedded == [] # re-keyed, not re-indexed + assert "2 file(s) differed only in how line endings were hashed" in out + assert _stored(db_path) == {p: md5_file(str(repo / p)) for p in ("src/a.py", "src/b.py")} + with CodeDB(db_path) as db: + assert db.meta_get(ii.FILE_KEY_KEY) == ii.FILE_KEY_VERSION + + calls = [] + monkeypatch.setattr(source_mod, "md5_file_raw", lambda p: calls.append(p) or "") + _write(repo / "src" / "a.py", _crlf(SAMPLE.replace("x + 1", "x + 9"))) + ii.run_incremental(str(repo), interactive=False) # the next run + assert calls == [] # the old hash is never computed again + assert any("x + 9" in t for t in embedded) # a real edit is still indexed + + +def test_an_older_index_still_reindexes_a_real_edit(env): + repo, db_path, embedded = env + ii.run_incremental(str(repo), interactive=False) + _as_pre_adr051(db_path, repo) + _write(repo / "src" / "a.py", _crlf(SAMPLE.replace("x + 1", "x + 5"))) + embedded.clear() + ii.run_incremental(str(repo), interactive=False) + assert any("x + 5" in t for t in embedded) + assert not any("omega" in t for t in embedded) # b.py was LF: same key + assert _stored(db_path)["src/a.py"] == md5_file(str(repo / "src" / "a.py")) From cb1d6f1008012372ef249c89c5532c4a282a55ab Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:30:41 +0000 Subject: [PATCH 3/6] Keep a slice's summary key when its part number changes B-048 fix 1, as ADR-029 designed it: summary_cache_key and the text sent to the summarizer turn a slice header's (Part i/N) into (Part), so adding or dropping a window no longer re-summarizes every slice whose code did not move, and a cached summary cannot quote a stale position. The embedded and stored text keep the part number. Existing caches carry over the way ADR-045's did: the one-time re-key now copies each indexed chunk's summary from whichever older key it has to the new one, under index_meta.summary_key = "no-lines-no-part". Co-Authored-By: Claude [ADR-051 src/db.py src/incremental_indexer.py] --- src/db.py | 52 +++++++--- src/incremental_indexer.py | 15 ++- tests/test_change_keys_parts.py | 176 ++++++++++++++++++++++++++++++++ tests/test_summary_cache_key.py | 6 +- 4 files changed, 229 insertions(+), 20 deletions(-) create mode 100644 tests/test_change_keys_parts.py diff --git a/src/db.py b/src/db.py index 465438e..2debd7d 100644 --- a/src/db.py +++ b/src/db.py @@ -47,17 +47,36 @@ # a 57-file GanttWebApp update, 2026-09-29). The summary never depends on them. _LINES_HEADER_RE = re.compile(r"\nLines: \d+-\d+\Z") _CODE_MARKER = "\nCode:\n" -SUMMARY_KEY_VERSION = "no-lines" +# ADR-051 (B-048): a slice's header ends with `Scope: (Part i/N)`. A window +# added or dropped anywhere in the file changes N in every slice of it, and keying on +# N re-summarized every slice whose code had not moved. +_PART_HEADER_RE = re.compile(r"(\nScope: [^\n]*) \(Part \d+/\d+\)\Z") +SUMMARY_KEY_VERSION = "no-lines-no-part" -def summary_cache_key(text: str) -> str: - """The `chunk_summaries` key for a chunk's text: its MD5 with the header's - `Lines: a-b` left out. Text without that header (tier 2/3) keys as before.""" +def summary_input(text: str) -> str: + """The text the summarizer is sent for a chunk: a slice header's `(Part i/N)` + becomes `(Part)`, so a cached summary cannot quote a position that has since + changed (ADR-051 §3). Any other text is returned as is.""" + head, marker, body = text.partition(_CODE_MARKER) + if not marker: + return text + return _PART_HEADER_RE.sub(r"\1 (Part)", head, count=1) + marker + body + + +def _md5_without_lines(text: str) -> str: head, marker, body = text.partition(_CODE_MARKER) if marker: head = _LINES_HEADER_RE.sub("", head, count=1) return hashlib.md5((head + marker + body).encode()).hexdigest() + +def summary_cache_key(text: str) -> str: + """The `chunk_summaries` key for a chunk's text: the MD5 of `summary_input(text)` + with the header's `Lines: a-b` left out (ADR-045). Text with neither header keys + as its plain MD5.""" + return _md5_without_lines(summary_input(text)) + # --------------------------------------------------------------------------- # Return type for graph queries # --------------------------------------------------------------------------- @@ -199,8 +218,9 @@ def effective_confidence(candidate: bool, confidence: Optional[float]) -> float: -- ------------------------------------------------------------------------- -- chunk_summaries — LLM extraction cache keyed by summary_cache_key(chunk_text): MD5 of --- the text without its `Lines:` header (ADR-045), so a symbol that only moved keeps its --- summary. A rename re-summarizes tier 1, whose header names the file. +-- the text without its `Lines:` header (ADR-045) and with a slice's `(Part i/N)` as +-- `(Part)` (ADR-051), so a symbol that only moved, or a slice whose file gained or lost a +-- window, keeps its summary. A rename re-summarizes tier 1, whose header names the file. -- ------------------------------------------------------------------------- CREATE TABLE IF NOT EXISTS chunk_summaries ( text_hash TEXT PRIMARY KEY, @@ -521,20 +541,24 @@ def _seed_index_meta(self) -> None: ) def _migrate_summary_keys(self) -> None: - """ADR-045: copy each cached summary to its line-free key, once per database. - - The old key is MD5 of the chunk text as stored in `chunks`, so every indexed - chunk's summary can be re-keyed without the model. Old rows stay: an older - indexer on the same database keeps finding them, and they cost a few KB. + """ADR-045, ADR-051 §4: copy each cached summary to its current key, once per + database. + + Every older key is a function of the chunk text as stored in `chunks`: its + plain MD5 before ADR-045, its MD5 without `Lines:` before ADR-051. So every + indexed chunk's summary can be re-keyed without the model, from whichever key + it has. Old rows stay: an older indexer on the same database keeps finding + them, and they cost a few KB. """ if self.meta_get("summary_key") == SUMMARY_KEY_VERSION: return pairs = [] for (text,) in self._conn.execute("SELECT text FROM chunks"): new = summary_cache_key(text) - old = hashlib.md5(text.encode()).hexdigest() - if new != old: - pairs.append((new, old)) + # The newer old key first: INSERT OR IGNORE keeps the first copy. + for old in dict.fromkeys((_md5_without_lines(text), hashlib.md5(text.encode()).hexdigest())): + if old != new: + pairs.append((new, old)) with self._tx() as cur: cur.executemany( "INSERT OR IGNORE INTO chunk_summaries(text_hash, summary, created_at) " diff --git a/src/incremental_indexer.py b/src/incremental_indexer.py index bb1ae27..5641b1f 100644 --- a/src/incremental_indexer.py +++ b/src/incremental_indexer.py @@ -636,10 +636,19 @@ def chunk_all_tiers(rel_path: str, content: str) -> dict[str, list]: def chunk_text_hash(text: str) -> str: """Cache key for one chunk's summary. Must agree across both passes. Line numbers - are not part of it (ADR-045): see db.summary_cache_key.""" + are not part of it (ADR-045), nor a slice's position (ADR-051): see + db.summary_cache_key.""" return summary_cache_key(text) +def summary_text(text: str) -> str: + """What the summarizer is sent for one chunk: its text with a slice's `(Part i/N)` + as `(Part)` (ADR-051 §3), so the summary agrees with its key. Both passes call it; + the embedded and stored text keep the position.""" + from db import summary_input + return summary_input(text) + + def dedupe_chunks_by_scope(chunks: list) -> list: """Keep one chunk per scope: the last, which is the row `INSERT OR REPLACE` keeps. @@ -833,7 +842,7 @@ def _prepare_file( print(f" [ingest:{rel_path}] {tier_name}: {len(uncached_idx)} cache misses → LLM", flush=True) if uncached_idx: new_summaries = summarizer.summarize_batch( - [texts[i] for i in uncached_idx] + [summary_text(texts[i]) for i in uncached_idx] ) new_pairs = [ (text_hashes[i], s) @@ -1084,7 +1093,7 @@ def run_summarization_pass( cached = db.get_cached_summaries(hashes) for h, t in zip(hashes, texts): if h not in cached: - pending.setdefault(h, t) + pending.setdefault(h, summary_text(t)) print(f" [summarize] {total_chunks} chunks in {len(to_index)} files, " f"{len(pending)} distinct texts to summarize", flush=True) diff --git a/tests/test_change_keys_parts.py b/tests/test_change_keys_parts.py new file mode 100644 index 0000000..37068d0 --- /dev/null +++ b/tests/test_change_keys_parts.py @@ -0,0 +1,176 @@ +"""ADR-051 (B-048 fix 1): a slice's summary key leaves out its `(Part i/N)`. + +Adding or dropping one window changed N in every slice of the file, so every slice +missed the summary cache. The summarizer is sent the same normalized text; the +embedded and stored text keep the position. No model loads. +""" +from __future__ import annotations + +import hashlib + +import numpy as np +import pytest + +import core +import incremental_indexer as ii +from ast_chunker import fallback_token_chunker +from core import embed_dimension +from db import SUMMARY_KEY_VERSION, CodeDB, summary_cache_key, summary_input + +_SLICE = "File: a.py\nScope: Full File (Part {i}/{n})\nCode:\nimport os\nx = 1\n" +_T1 = ("File: a.py\nEntity: a.py::f (function)\nLines: {lines}\n" + "Code:\ndef f():\n return 1\n") + + +@pytest.fixture(autouse=True) +def _offline_tokens(monkeypatch): + """Token counts by characters, so chunking needs no tokenizer download.""" + monkeypatch.setattr(core.jina_tokenizer, "count_tokens", lambda s: max(1, len(s) // 4)) + + +def _md5(text): + return hashlib.md5(text.encode()).hexdigest() + + +def test_the_key_ignores_the_slice_position(): + assert summary_cache_key(_SLICE.format(i=2, n=5)) == summary_cache_key(_SLICE.format(i=3, n=6)) + assert summary_cache_key(_SLICE.format(i=2, n=5)) == _md5(_SLICE.format(i="", n="") + .replace(" (Part /)", " (Part)")) + + +def test_the_code_and_the_scope_still_count(): + a = _SLICE.format(i=1, n=2) + assert summary_cache_key(a) != summary_cache_key(a.replace("x = 1", "x = 2")) + assert summary_cache_key(a) != summary_cache_key(a.replace("Full File", "a.py::Big")) + + +def test_only_the_header_is_normalized(): + """A line in the code that looks like a slice header is code.""" + text = _SLICE.format(i=1, n=2) + "Scope: Full File (Part 1/2)\n" + assert summary_input(text).endswith("Scope: Full File (Part 1/2)\n") + assert summary_input(text).startswith("File: a.py\nScope: Full File (Part)\nCode:\n") + + +def test_tier1_text_keys_as_under_adr045(): + """No slice header: the key is ADR-045's, so every tier-1 cache row stays valid.""" + text = _T1.format(lines="6-7") + assert summary_input(text) == text + assert summary_cache_key(text) == _md5(text.replace("\nLines: 6-7", "")) + + +@pytest.fixture +def small_tokens(monkeypatch): + """One token per character keeps the windows small and their boundaries obvious.""" + monkeypatch.setattr(core.jina_tokenizer, "count_tokens", lambda s: len(s)) + + +def test_adding_a_window_keeps_the_unchanged_slices_keys(small_tokens): + lines = [f"value_{i:03d} = {i}" for i in range(40)] + before = fallback_token_chunker("\n".join(lines), "a.py", 200, 0, parent_scope="Full File") + after = fallback_token_chunker("\n".join(lines + [f"value_{i:03d} = {i}" for i in range(40, 52)]), + "a.py", 200, 0, parent_scope="Full File") + assert len(after) > len(before) + same_code = [(o, n) for o, n in zip(before, after) + if o.text.partition("\nCode:\n")[2] == n.text.partition("\nCode:\n")[2]] + assert len(same_code) >= len(before) - 1 # all but the old last slice + for old, new in same_code: + assert old.text != new.text # N changed in the text + assert summary_cache_key(old.text) == summary_cache_key(new.text) + assert ii.chunk_text_hash(old.text) == ii.chunk_text_hash(new.text) + +# ── carrying an existing cache over ────────────────────────────────────────── + +def _old_db(tmp_path, texts, old_key, version): + """A database as an older indexer left it: summaries under `old_key(text)`.""" + path = str(tmp_path / "graph.db") + db = CodeDB(path) + with db._tx() as cur: + cur.execute("INSERT INTO files(path, content_hash) VALUES ('a.py', 'h')") + fid = cur.execute("SELECT id FROM files").fetchone()[0] + for i, t in enumerate(texts): + cur.execute("INSERT INTO chunks(file_id, scope, tier, start_line, end_line, text, tags)" + " VALUES (?, ?, 2, 1, 2, ?, '')", (fid, f"s{i}", t)) + cur.execute("INSERT INTO chunk_summaries(text_hash, summary) VALUES (?, ?)", + (old_key(t), f"summary {i}")) + if version is None: + cur.execute("DELETE FROM index_meta WHERE key = 'summary_key'") + else: + cur.execute("UPDATE index_meta SET value = ? WHERE key = 'summary_key'", (version,)) + db.close() + return path + + +def _adr045_key(text): + head, marker, body = text.partition("\nCode:\n") + import re + head = re.sub(r"\nLines: \d+-\d+\Z", "", head) + return _md5(head + marker + body) + + +@pytest.mark.parametrize("old_key,version", [(_adr045_key, "no-lines"), (_md5, None)], + ids=["from-adr045", "from-raw-md5"]) +def test_an_existing_cache_carries_over_once(tmp_path, old_key, version): + texts = [_SLICE.format(i=1, n=2), _SLICE.format(i=2, n=2).replace("x = 1", "y = 2"), + _T1.format(lines="3-4")] + path = _old_db(tmp_path, texts, old_key, version) + with CodeDB(path) as db: + assert db.meta_get("summary_key") == SUMMARY_KEY_VERSION + got = db.get_cached_summaries([summary_cache_key(t) for t in texts]) + assert got == {summary_cache_key(t): f"summary {i}" for i, t in enumerate(texts)} + assert db.get_cached_summaries([old_key(texts[0])]) # old rows stay + n = db._conn.execute("SELECT COUNT(*) FROM chunk_summaries").fetchone()[0] + with CodeDB(path) as db: # runs once + assert db._conn.execute("SELECT COUNT(*) FROM chunk_summaries").fetchone()[0] == n + + +def test_a_carried_over_slice_hits_after_a_window_is_added(tmp_path): + """The pre-upgrade row was written for "Part 1/2"; the file now has three parts.""" + path = _old_db(tmp_path, [_SLICE.format(i=1, n=2)], _adr045_key, "no-lines") + with CodeDB(path) as db: + assert db.get_cached_summaries([summary_cache_key(_SLICE.format(i=1, n=3))]) == { + summary_cache_key(_SLICE.format(i=1, n=3)): "summary 0"} + + +# ── what the summarizer and the embedder see ───────────────────────────────── + +class _Recorder: + def __init__(self): + self.seen: list[str] = [] + + def summarize_batch(self, codes): + self.seen.extend(codes) + return [f"summary {len(c)}" for c in codes] + + def shutdown(self): + pass + + +def test_the_summarizer_gets_normalized_text_and_the_embedder_the_original( + tmp_path, monkeypatch, small_tokens): + repo = tmp_path / "repo" + repo.mkdir() + (repo / "big.py").write_text("\n".join(f"v{i:04d} = {i}" for i in range(600)), encoding="utf-8") + index = tmp_path / "index" + index.mkdir() + embedded: list[str] = [] + + def fake_embed(texts, batch_size=32): + embedded.extend(texts) + return np.ones((len(texts), embed_dimension()), dtype=np.float32) + + recorder = _Recorder() + monkeypatch.setattr(core, "embed_batch", fake_embed) + monkeypatch.setattr(ii, "summarization_enabled", lambda: True) + monkeypatch.setattr(ii, "summarizer_model_id", lambda: "fake") + monkeypatch.setattr("model_client.make_summarizer", lambda: recorder) + monkeypatch.setattr(ii, "INDEX_DIR", str(index)) + monkeypatch.setattr(ii, "DB_PATH", str(index / "graph.db")) + ii.run_incremental(str(repo), interactive=False) + + sent = [s for s in recorder.seen if s.startswith("File: big.py\nScope: Full File")] + assert len(sent) > 1 + assert all(s.startswith("File: big.py\nScope: Full File (Part)\nCode:\n") for s in sent) + sliced = [t for t in embedded if t.startswith("File: big.py\nScope: Full File")] + assert len(sliced) == len(sent) + assert all(t.startswith("File: big.py\nScope: Full File (Part ") for t in sliced) + assert not any("(Part)" in t for t in sliced) # positions kept diff --git a/tests/test_summary_cache_key.py b/tests/test_summary_cache_key.py index 880a721..a119540 100644 --- a/tests/test_summary_cache_key.py +++ b/tests/test_summary_cache_key.py @@ -37,9 +37,9 @@ def test_other_header_lines_still_count(): def test_text_without_the_header_keys_as_before(): - """Tier 2/3 slices have no `Lines:` header: their existing cache rows stay valid, - even when the code itself contains a line that looks like one.""" - assert summary_cache_key(_T2) == _md5(_T2) + """Tier 2/3 slices have no `Lines:` header, even when the code itself contains a + line that looks like one. Their part number is left out since ADR-051.""" + assert summary_cache_key(_T2) == _md5(_T2.replace("(Part 1/1)", "(Part)")) assert summary_cache_key("no header at all") == _md5("no header at all") From f954fc035ea763ad014324103d6981e1c15352ec Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:31:57 +0000 Subject: [PATCH 4/6] Keep the vector of an edited file's unchanged tier-1 chunks B-034, tier 1: before a modified file's vectors are purged, its tier-1 vectors are read back from the FAISS id map with their chunk_text_hash. A re-ingested chunk with the same id and hash gets that vector instead of an embedding; a moved symbol keeps its vector and gets its new line range. Vectors are still added in _write_plan, before the file's rows, so vectors equal rows and the overlapped build writes the same bytes as the serial one. Summary vectors and tiers 2/3 are embedded as before. Co-Authored-By: Claude [ADR-051 src/incremental_indexer.py] --- src/incremental_indexer.py | 110 +++++++++++++++++-- tests/test_change_keys_reuse.py | 183 ++++++++++++++++++++++++++++++++ 2 files changed, 284 insertions(+), 9 deletions(-) create mode 100644 tests/test_change_keys_reuse.py diff --git a/src/incremental_indexer.py b/src/incremental_indexer.py index 5641b1f..bd3b3a0 100644 --- a/src/incremental_indexer.py +++ b/src/incremental_indexer.py @@ -554,6 +554,48 @@ def reconcile_vectors( return sorted(missing_files), n_removed +# --------------------------------------------------------------------------- +# Tier-1 vector reuse (ADR-051 §5, B-034) +# --------------------------------------------------------------------------- + +TIER1 = "tier1_surgical" + + +def tier1_vectors_to_reuse( + db: CodeDB, + index: faiss.Index, + paths: list[str], +) -> dict[int, tuple[str, np.ndarray]]: + """{chunk id: (text hash, vector)} for every tier-1 chunk of `paths` that has a + vector, read before `purge_stale_vectors` removes them. + + A modified file's chunks are all purged and re-ingested. One whose id and + `chunk_text_hash` come back unchanged gets this vector back instead of a new + embedding (`_prepare_file`). The hash leaves out `Lines:`, so a symbol that only + moved keeps a vector embedded with its old line numbers, as an unchanged file's + chunks already do. Tiers 2 and 3 are not reused: a slice's text changes with any + edit above it (B-048). + """ + id_map = getattr(index, "id_map", None) + if not paths or id_map is None or index.ntotal == 0: + return {} + position = {int(i): p for p, i in enumerate(faiss.vector_to_array(id_map))} + flat = index.index + out: dict[int, tuple[str, np.ndarray]] = {} + for start in range(0, len(paths), 500): + part = paths[start:start + 500] + rows = db._conn.execute( + "SELECT f.path, c.scope, c.text FROM chunks c JOIN files f ON f.id = c.file_id " + f"WHERE c.tier = ? AND f.path IN ({','.join('?' * len(part))})", + [TIER_NUM[TIER1], *part], + ).fetchall() + for path, scope, text in rows: + fid = stable_id(TIER1, path, scope) + if fid in position: + out[fid] = (chunk_text_hash(text), flat.reconstruct(position[fid])) + return out + + # --------------------------------------------------------------------------- # Project-descriptor ingest: parse edges only — no chunking or embedding # --------------------------------------------------------------------------- @@ -676,7 +718,7 @@ class _FilePlan: "rel_path", "content_hash", "content_changed_at", "authored_at", "symbols", "edges", "references", "symbol_types", "tier_chunks", "tier_ids", "tier_texts", "summary_items", - "tier_vectors", "summary_vectors", + "tier_reused", "tier_vectors", "summary_vectors", ) def __init__( @@ -693,6 +735,7 @@ def __init__( tier_ids: dict[str, list[int]], tier_texts: dict[str, list[str]], summary_items: dict[str, list[tuple[int, str]]], + tier_reused: Optional[dict[str, dict[int, np.ndarray]]] = None, ) -> None: self.rel_path = rel_path self.content_hash = content_hash @@ -706,6 +749,9 @@ def __init__( self.tier_ids = tier_ids self.tier_texts = tier_texts self.summary_items = summary_items + # ADR-051 §5 (B-034): {tier: {position in tier_texts: vector}} for chunks that + # keep the vector they already had; those texts are not embedded again. + self.tier_reused = tier_reused or {} self.tier_vectors: dict[str, np.ndarray] = {} self.summary_vectors: dict[str, np.ndarray] = {} @@ -715,20 +761,35 @@ def pending_texts(self) -> list[str]: summary texts (same order).""" texts: list[str] = [] for tier_name, _, _ in TIER_CONFIGS: - texts.extend(self.tier_texts.get(tier_name, ())) + reused = self.tier_reused.get(tier_name, {}) + texts.extend(t for i, t in enumerate(self.tier_texts.get(tier_name, ())) + if i not in reused) for tier_name, _, _ in TIER_CONFIGS: texts.extend(s for _fid, s in self.summary_items.get(tier_name, ())) return texts - def scatter(self, vectors: np.ndarray) -> None: + def scatter(self, vectors: Optional[np.ndarray]) -> None: """Split `vectors` — already normalized and aligned to `pending_texts()` — - into this file's per-tier code and summary vectors.""" + into this file's per-tier code and summary vectors, putting reused vectors + back in their places. `None` when nothing was pending.""" + if vectors is None: + vectors = np.empty((0, 0), dtype=np.float32) offset = 0 for tier_name, _, _ in TIER_CONFIGS: n = len(self.tier_texts.get(tier_name, ())) - if n: - self.tier_vectors[tier_name] = vectors[offset:offset + n] - offset += n + reused = self.tier_reused.get(tier_name, {}) + fresh = vectors[offset:offset + n - len(reused)] + offset += n - len(reused) + if not n: + continue + if not reused: + self.tier_vectors[tier_name] = fresh + continue + full = np.empty((n, len(next(iter(reused.values())))), dtype=np.float32) + rest = iter(fresh) + for i in range(n): + full[i] = reused[i] if i in reused else next(rest) + self.tier_vectors[tier_name] = full for tier_name, _, _ in TIER_CONFIGS: items = self.summary_items.get(tier_name, ()) n = len(items) @@ -767,6 +828,7 @@ def _prepare_file( summarizer: Optional[object] = None, content_changed_at: Optional[str] = None, authored_at: Optional[str] = None, + reuse: Optional[dict[int, tuple[str, np.ndarray]]] = None, ) -> _FilePlan: """Everything `ingest_file` did before it touched the embedder: AST parse, three-tier chunking, import-edge resolution, and summary-cache lookups @@ -777,6 +839,10 @@ def _prepare_file( The DocumentStore text/tag cache is populated here, same as it always was: it does not depend on the embedding and MCP reads should see it as soon as the chunk exists. + + `reuse` is `tier1_vectors_to_reuse`'s map: a tier-1 chunk whose id is in it with + the same text hash keeps that vector instead of being embedded (ADR-051 §5). + Entries are popped as they are looked at. """ print(f" [ingest:{rel_path}] chunking...", flush=True) tier_chunks: dict[str, list] = chunk_all_tiers(rel_path, content) @@ -805,6 +871,7 @@ def _prepare_file( tier_ids: dict[str, list[int]] = {} tier_texts: dict[str, list[str]] = {} summary_items: dict[str, list[tuple[int, str]]] = {} + tier_reused: dict[str, dict[int, np.ndarray]] = {} for tier_name, chunks in tier_chunks.items(): ids: list[int] = [] @@ -827,6 +894,17 @@ def _prepare_file( tier_ids[tier_name] = ids tier_texts[tier_name] = texts + if reuse and tier_name == TIER1: + kept = {} + for i, (fid, text) in enumerate(zip(ids, texts)): + old = reuse.pop(fid, None) + if old is not None and old[0] == chunk_text_hash(text): + kept[i] = old[1] + if kept: + tier_reused[tier_name] = kept + print(f" [ingest:{rel_path}] {tier_name}: {len(kept)} of {len(texts)} " + f"vector(s) reused, code unchanged", flush=True) + if not texts: print(f" [ingest:{rel_path}] {tier_name}: no chunks, skipping", flush=True) continue @@ -879,6 +957,7 @@ def _prepare_file( tier_ids=tier_ids, tier_texts=tier_texts, summary_items=summary_items, + tier_reused=tier_reused, ) @@ -970,6 +1049,8 @@ def ingest_file( faiss.normalize_L2(vectors) print(f" [ingest:{rel_path}] embedding done, shape={vectors.shape}", flush=True) plan.scatter(vectors) + else: + plan.scatter(None) return _write_plan(plan, faiss_indexes, doc_store, db) @@ -1286,6 +1367,13 @@ def _run_incremental( stale_paths = diff.modified + diff.deleted stale_ids = get_stale_ids(db, stale_paths) if stale_paths else np.empty(0, dtype=np.int64) + # ADR-051 §5 (B-034): read the modified files' tier-1 vectors before the purge, so + # a chunk whose code did not change keeps its vector. Not on a pre-ADR-030 index, + # whose code vectors have summaries folded in. + _reuse = (tier1_vectors_to_reuse(db, faiss_indexes[TIER1], diff.modified) + if db.meta_get("embed_layout") == EMBED_LAYOUT else {}) + n_reused = 0 + if len(stale_ids) > 0: n_removed = purge_stale_vectors(faiss_indexes, doc_store, stale_ids) print(f" Purged {n_removed} stale vector IDs ({len(stale_paths)} file(s)).") @@ -1402,8 +1490,7 @@ def _write_window(job: tuple) -> None: offset = 0 for plan, n in zip(plans, counts_by_plan): try: - if n: - plan.scatter(vectors[offset:offset + n]) + plan.scatter(vectors[offset:offset + n] if n else None) counts = _write_plan(plan, faiss_indexes, doc_store, db) t1 = counts.get("tier1_surgical", 0) t2 = counts.get("tier2_component", 0) @@ -1473,7 +1560,9 @@ def _flush_pending() -> None: summarizer=summarizer, content_changed_at=_cc_at, authored_at=_auth_at, + reuse=_reuse, ) + n_reused += len(plan.tier_reused.get(TIER1, ())) timing["prep"] += time.monotonic() - _t_prep texts = plan.pending_texts() pending_plans.append(plan) @@ -1506,6 +1595,9 @@ def _flush_pending() -> None: f"prepare {timing['prep']:.1f}s, write {timing['write']:.1f}s, " f"wall {time.monotonic() - pass2_start:.1f}s " f"(overlap {'on' if embedder is not None else 'off'})", flush=True) + if n_reused: + print(f" Reused {n_reused} tier-1 vector(s) whose code did not change (ADR-051).") + _reuse.clear() # ADR-021: resolve CALLS-edge bare callee names to in-repo FQNs so the graph # Traverse step has real neighbours to walk. Runs once here, over the now-complete diff --git a/tests/test_change_keys_reuse.py b/tests/test_change_keys_reuse.py new file mode 100644 index 0000000..58238d1 --- /dev/null +++ b/tests/test_change_keys_reuse.py @@ -0,0 +1,183 @@ +"""ADR-051 §5 (B-034, tier 1): an edited file's unchanged tier-1 chunks keep their vectors. + +The fake embedder counts what it is asked to embed and is deterministic per text, so +the overlapped and back-to-back builds can be compared byte for byte (ADR-040). The +summarizer is off. No model loads. +""" +from __future__ import annotations + +import os + +import faiss +import numpy as np +import pytest + +import core +import incremental_indexer as ii +from core import DocumentStore, embed_dimension +from db import CodeDB +from stable_id import TIER_CONFIGS + +SAMPLE = ( + "def alpha(x):\n return x + 1\n\n\n" + "def beta(y):\n return y * 2\n\n\n" + "def gamma(z):\n return z - 3\n" +) + + +@pytest.fixture(autouse=True) +def _offline_tokens(monkeypatch): + """Token counts by characters, so chunking needs no tokenizer download.""" + monkeypatch.setattr(core.jina_tokenizer, "count_tokens", lambda s: max(1, len(s) // 4)) + + +def _fake_embed(texts, batch_size=32): + dim = embed_dimension() + return np.array([[float((hash(t) >> (8 * i)) & 0xFF) + 1.0 for i in range(dim)] for t in texts], + dtype=np.float32) + + +@pytest.fixture +def env(tmp_path, monkeypatch): + embedded: list[str] = [] + + def counting_embed(texts, batch_size=32): + embedded.extend(texts) + return _fake_embed(texts) + + monkeypatch.setattr(core, "embed_batch", counting_embed) + monkeypatch.setattr(ii, "summarization_enabled", lambda: False) + return embedded + + +def _repo(tmp_path, name="repo"): + repo = tmp_path / name + (repo / "src").mkdir(parents=True) + (repo / "src" / "m.py").write_text(SAMPLE, encoding="utf-8") + (repo / "src" / "other.py").write_text("def other():\n return 0\n", encoding="utf-8") + return repo + + +def _run(tmp_path, monkeypatch, repo, name="index", overlap=True): + index = tmp_path / name + index.mkdir(exist_ok=True) + monkeypatch.setattr(ii, "INDEX_DIR", str(index)) + monkeypatch.setattr(ii, "DB_PATH", str(index / "graph.db")) + monkeypatch.setattr(ii, "embed_overlap", lambda: overlap) + ii.run_incremental(str(repo), interactive=False) + return str(index) + + +def _tier1(texts): + return [t for t in texts if "\nEntity: " in t] + + +def _vectors(index_dir: str) -> dict[str, dict[int, bytes]]: + out: dict[str, dict[int, bytes]] = {} + for name, _, _ in TIER_CONFIGS: + idx = faiss.read_index(os.path.join(index_dir, f"{name}.faiss")) + ids = faiss.vector_to_array(idx.id_map) + flat = idx.index.reconstruct_n(0, idx.ntotal) + out[name] = {int(i): v.tobytes() for i, v in zip(ids, flat)} + return out + + +def _vectors_match_rows(index_dir: str) -> None: + db_path = os.path.join(index_dir, "graph.db") + mgr = core.MultiIndexManager(index_dir) + indexes = {name: mgr.load_or_create(name) for name, _, _ in TIER_CONFIGS} + with CodeDB(db_path) as db: + missing, surplus = ii.reconcile_vectors(db, indexes, DocumentStore(db_path)) + n_rows = db._conn.execute("SELECT COUNT(*) FROM chunks").fetchone()[0] + assert (missing, surplus) == ([], 0) + assert sum(idx.ntotal for idx in indexes.values()) == n_rows + + +def test_a_one_function_edit_embeds_only_that_functions_tier1_chunk(tmp_path, monkeypatch, env): + embedded = env + repo = _repo(tmp_path) + index = _run(tmp_path, monkeypatch, repo) + before = _vectors(index) + embedded.clear() + + (repo / "src" / "m.py").write_text(SAMPLE.replace("y * 2", "y * 7"), encoding="utf-8") + _run(tmp_path, monkeypatch, repo) + + t1 = _tier1(embedded) + assert len(t1) == 1 and "beta" in t1[0] and "y * 7" in t1[0] + assert not any("other" in t for t in embedded) # other.py untouched + after = _vectors(index) + unchanged = [i for i in before["tier1_surgical"] if i in after["tier1_surgical"] + and before["tier1_surgical"][i] == after["tier1_surgical"][i]] + assert len(unchanged) == len(after["tier1_surgical"]) - 1 # alpha, gamma, other + _vectors_match_rows(index) + + +def test_a_moved_function_keeps_its_vector_and_gets_its_new_lines(tmp_path, monkeypatch, env): + embedded = env + repo = _repo(tmp_path) + index = _run(tmp_path, monkeypatch, repo) + before = _vectors(index)["tier1_surgical"] + embedded.clear() + + (repo / "src" / "m.py").write_text("import os\n\n\n" + SAMPLE, encoding="utf-8") + _run(tmp_path, monkeypatch, repo) + + assert not any("alpha" in t or "gamma" in t for t in _tier1(embedded)) + after = _vectors(index)["tier1_surgical"] + assert all(after[k] == v for k, v in before.items()) # same ids, same bytes + with CodeDB(os.path.join(index, "graph.db")) as db: + lines = db._conn.execute( + "SELECT c.start_line FROM chunks c JOIN files f ON f.id = c.file_id " + "WHERE f.path = 'src/m.py' AND c.scope LIKE '%alpha' AND c.tier = 1").fetchone() + assert tuple(lines) == (4,) + _vectors_match_rows(index) + + +def test_reuse_writes_the_same_bytes_with_the_overlap_on_or_off(tmp_path, monkeypatch, env): + monkeypatch.setattr(ii, "_EMBED_WINDOW_TEXTS", 3) # several windows + built = {} + for overlap in (False, True): + repo = _repo(tmp_path, f"repo-{overlap}") + (repo / "src" / "n.py").write_text(SAMPLE.replace("alpha", "delta"), encoding="utf-8") + _run(tmp_path, monkeypatch, repo, f"index-{overlap}", overlap) + (repo / "src" / "m.py").write_text(SAMPLE.replace("y * 2", "y * 7"), encoding="utf-8") + (repo / "src" / "n.py").write_text("\n" + SAMPLE.replace("alpha", "delta"), encoding="utf-8") + built[overlap] = _run(tmp_path, monkeypatch, repo, f"index-{overlap}", overlap) + assert _vectors(built[False]) == _vectors(built[True]) + for index in built.values(): + _vectors_match_rows(index) + + +def test_an_all_reused_file_still_writes_its_rows(tmp_path, monkeypatch, env): + """A file whose pending texts are all tier 2/3 still gets its reused tier-1 vectors + added: `scatter` runs even when nothing of the file was embedded in tier 1.""" + embedded = env + repo = _repo(tmp_path) + index = _run(tmp_path, monkeypatch, repo) + embedded.clear() + (repo / "src" / "m.py").write_text(SAMPLE + "\n\n", encoding="utf-8") + _run(tmp_path, monkeypatch, repo) + assert _tier1(embedded) == [] + _vectors_match_rows(index) + + +def test_ingest_file_with_nothing_pending_writes_reused_vectors(tmp_path, monkeypatch, env): + """The single-file path: every vector reused, so no embed call at all.""" + repo = _repo(tmp_path) + index = _run(tmp_path, monkeypatch, repo) + mgr = core.MultiIndexManager(index) + indexes = {name: mgr.load_or_create(name) for name, _, _ in TIER_CONFIGS} + db_path = os.path.join(index, "graph.db") + with CodeDB(db_path) as db: + reuse = ii.tier1_vectors_to_reuse(db, indexes["tier1_surgical"], ["src/other.py"]) + assert len(reuse) == 1 + plan = ii._prepare_file("src/other.py", "def other():\n return 0\n", "h", + DocumentStore(db_path), db, reuse=reuse) + assert plan.pending_texts() == [t for tier in ("tier2_component", "tier3_architectural") + for t in plan.tier_texts[tier]] + plan.tier_texts = {"tier1_surgical": plan.tier_texts["tier1_surgical"]} + plan.tier_ids = {"tier1_surgical": plan.tier_ids["tier1_surgical"]} + assert plan.pending_texts() == [] + plan.scatter(None) + assert plan.tier_vectors["tier1_surgical"].shape == (1, embed_dimension()) From dd2ab92978a5751a35d29b9b937ad393427cba24 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:33:14 +0000 Subject: [PATCH 5/6] Record ADR-051's B-034 design, tests and measurements Co-Authored-By: Claude [ADR-051] --- docs/adr/ADR-051-stable-change-keys.md | 117 +++++++++++++++++++++++-- 1 file changed, 109 insertions(+), 8 deletions(-) diff --git a/docs/adr/ADR-051-stable-change-keys.md b/docs/adr/ADR-051-stable-change-keys.md index 13dd8c1..c328d47 100644 --- a/docs/adr/ADR-051-stable-change-keys.md +++ b/docs/adr/ADR-051-stable-change-keys.md @@ -103,9 +103,37 @@ removes every vector of a modified file before re-indexing it. was written for (`chunks`), so after it a slice whose code did not change hits whatever `N` becomes, and lookups stay one query. -5. **B-034, tier 1: an unchanged tier-1 chunk keeps its vector.** See the Implementation Log for - whether it was built and how it holds the three invariants (vectors == rows, ADR-037; the - ADR-040 two-pass build is byte-identical with the overlap on or off; the ADR-038 writer lock). +5. **B-034, tier 1: an unchanged tier-1 chunk keeps its vector.** + - **Where the vector comes from: the FAISS id map, not a cache.** Right before + `purge_stale_vectors`, `tier1_vectors_to_reuse` reads each modified file's tier-1 rows from + `chunks` and, for each one with a vector, `reconstruct`s it from the `IndexIDMap`'s flat + index at its id-map position. It keeps `{chunk id: (chunk_text_hash, vector)}` in memory for + the run. A cache keyed by text hash would be a second store to keep in step with FAISS and + to prune; the vector is already where the purge is about to drop it. + - **When it is reused.** `_prepare_file` gives a tier-1 chunk the old vector when its id + (file, scope) and its `chunk_text_hash` both match. The hash leaves out `Lines:` (and a + split part's position), so a symbol that only moved keeps its vector and gets its new line + range in its row. Its vector was embedded with the old line numbers; an unchanged file's + chunks already carry vectors embedded with whatever their header said then. + - **What is not reused.** Tiers 2 and 3: a slice's text changes with any edit above it, so + a reuse there almost never fires (B-048 fixes 2 and 3 are the way in). Summary vectors: + their text is the cached summary, which is the same for an unchanged key, but a reuse + would need the summary index's vectors read the same way and the condition proven for every + cache history. Left as a follow-up; with summaries on, an unchanged chunk now costs one + embed instead of two. + - **Not on a pre-ADR-030 index** (`embed_layout` unset): its code vectors have summaries folded + in, and ADR-030 §5 asks for a full re-index there anyway. + - **The three invariants:** + - *vectors == rows (ADR-037).* A reused vector goes through the same `_write_plan` as an + embedded one: `add_with_ids` under the chunk's id, then the file's rows. The commit point + and what a kill leaves behind are unchanged, and `reconcile_vectors` heals the same way. + A healed file's missing vectors are not in the id map, so they are embedded. + - *ADR-040's build is byte-identical with the overlap on or off.* Which vectors are reused + is decided in `_prepare_file` on the main thread from a snapshot taken before pass 2; + `scatter` puts reused and embedded vectors back in chunk order. The embed thread never + sees a reused text. A test builds and edits the same repo both ways and compares bytes. + - *The writer lock (ADR-038).* The snapshot is taken inside `_run_incremental`, under the + lock, and lives only for that run. ## Consequences @@ -113,6 +141,12 @@ removes every vector of a modified file before re-indexing it. - A checkout whose line endings differ from the one indexed reprocesses only the files whose code differs. - Adding or dropping a window keeps the cached summary of every slice whose code did not move. + Replaying this repo's last 40 commits that touched `src/*.py`, 665 of 1,578 tier-2/3 slices + hit the cache with the new key against 550 with ADR-045's: 115 fewer long prompts (see + Measurements for the stack and its caveat). +- An edit re-embeds only the tier-1 chunks it changed. In the same replay, 4,252 of 4,774 + tier-1 chunks of the edited files (89%) came back unchanged and keep their vectors. Switching + `[indexer].source` (ADR-042), which makes every file look modified, keeps every tier-1 vector. - Upgrading costs no re-embed and no re-summarize. **Worse:** @@ -122,6 +156,12 @@ removes every vector of a modified file before re-indexing it. - The summarizer now sees `(Part)` where it saw `(Part 2/5)`. Unmeasured, as ADR-029 noted; no cached summary of this repo was checked for quoting a part number. See Open questions. - `chunk_summaries` keeps one more old row per slice, as ADR-045's re-key did. +- A modified file's tier-1 vectors are held in memory between the purge and the end of pass 2: + about 6 KB each at 1,536 dimensions, so 120 MB for 20,000 chunks when a source switch makes + every file modified. A normal edit holds a few files' worth. +- A moved symbol's vector was embedded with its old `Lines:`, so an incremental index and a + fresh build of the same tree can differ in those vectors. They differed already: an unchanged + file keeps its vectors across any number of edits elsewhere. **Neutral:** - Repacking (B-048 fixes 2 and 3) is untouched: one inserted line still changes the text of every @@ -141,6 +181,9 @@ removes every vector of a modified file before re-indexing it. | ADR-029's lookup-time fallback to the old key | Misses exactly when `N` changed (§4) | | Strip `(Part i/N)` from the stored chunk text | Changes the embedded text, so every slice re-embeds, and search results lose the part number | | Key on the summary input but leave the summarizer input as it is | A summary written for "Part 2/5" could be served for a slice that is now part 3/6 | +| B-034: cache vectors by text hash in SQLite | A second store of every tier-1 vector to keep in step with FAISS and prune; `reconstruct` from the id map needs neither | +| B-034: reuse tiers 2 and 3 too | Any edit above a slice changes its text, so it would rarely fire until slices stop repacking (B-048 fixes 2, 3) | +| B-034: skip it as not worth the complexity | About 90 lines, and it removes most of an edit's tier-1 embeds (89% in the replay) and all of a source switch's | ## Open questions for @edb @@ -148,13 +191,71 @@ removes every vector of a modified file before re-indexing it. 2. **The summarizer input change** (§3) is unmeasured. A quick check on a GPU: summarize a few tier-2 slices with and without the part number and compare. ADR-045 found that no summary of 7,258 mentioned a line number; nobody has counted part numbers. +3. **Reusing summary vectors** too (§5) would halve the remaining tier-1 embeds with summaries on. + Worth a follow-up? +4. **The replay's numbers** used a stand-in tokenizer (Measurements). Re-running + the replay with bge-code-v1's tokenizer would give the real slice counts; the tier-1 number + barely depends on it (only oversized symbols split by tokens). ## Implementation Log -- [ ] B-054: `md5_file` normalizes line endings; `file_key` marker and in-place re-key -- [ ] B-048 fix 1: `summary_input`, `summary_cache_key`, `_migrate_summary_keys` -- [ ] B-034 tier 1: design and build, or a write-up of why not -- [ ] Tests -- [ ] Gates +- [x] B-054: `source.md5_file` normalizes line endings, block-safe; `md5_file_raw` keeps the old + formula for the re-key. `incremental_indexer.rekey_unchanged_files` and the + `index_meta.file_key` marker, written by `_write_index_meta` on every completed run. +- [x] B-048 fix 1: `db.summary_input`, `summary_cache_key` on top of it, `_migrate_summary_keys` + copying from either older key; `incremental_indexer.summary_text` sends the normalized text to + the summarizer in both passes. +- [x] B-034 tier 1: `tier1_vectors_to_reuse`, `_FilePlan.tier_reused`, `scatter` reassembling + reused and embedded vectors; `scatter(None)` when a file has nothing pending. +- [x] Tests, 25 new: + - `tests/test_change_keys_eol.py` (11): LF, CRLF and lone-CR files hash alike, including a + `\r\n` split across 64 KiB blocks; an LF file hashes as before; chunk text never keeps + `\r`, in worktree and git mode; an LF-built index does not reprocess its CRLF twin; an older + index re-keys once without re-embedding and never computes the old hash again; a real edit + on an older index is still re-indexed. + - `tests/test_change_keys_parts.py` (9): the key ignores the position, not the code or scope; + only the header is touched; tier-1 keys are ADR-045's; adding a window keeps the unchanged + slices' keys; a cache from either older key carries over once; a carried-over row hits after + N changes; the summarizer gets `(Part)` and the embedder the original text. + - `tests/test_change_keys_reuse.py` (5): a one-function edit embeds that function's tier-1 + chunk only, and vectors == rows after; a moved function keeps its vector bytes and gets its + new lines; overlap on and off write identical bytes with reuse; an all-reused file still + writes its vectors, through `run_incremental` and through `_prepare_file` + `scatter(None)`. + - `tests/test_summary_cache_key.py`: one assertion changed. It pinned a `(Part 1/1)` slice's + key to its plain MD5, which this ADR changes on purpose. +- [x] Gates: flake8 clean; `pytest --co` 729 collected; `conformance_eval --check-baseline`, + `resolution_eval --check-baseline`, `l5x_fixture_teeth --check` all pass. MCP Inspector not + run: no tool changed. +- [x] `CHUNKER_VERSION` left at 3 (§2). **Notes:** + +- **Test environment (2026-10-20):** Linux, Python 3.12, CPU only, torch from PyPI (the CPU wheel + index was blocked), and no network to huggingface.co. Every test that chunks needs + bge-code-v1's tokenizer, so 48 existing tests fail here, identically on `871b55c` and on this + branch (654 → 679 passed, 48 failed). The new tests patch `count_tokens` with a character + count, so they run offline. To check the 48, both trees were also run with a stand-in + word/punctuation tokenizer patched into `core.jina_tokenizer` by a pytest plugin outside the + repo: `871b55c` 702 passed, this branch 727 passed, 2 skipped each, no failures. +- **Chunk text and `\r`:** verified, never kept (Context). So B-054 was a file-key bug only. +- **Not bumping `CHUNKER_VERSION`** departs from B-054's text and the task prompt (§2). +- **ADR-029's carry-over** is done as ADR-045's migration, not as its lookup fallback (§4). +- **`src/summarizer.py`'s module docstring** still describes the key as "MD5 of the chunk text + without its `Lines:` header". It is in another session's file, so it is left for a follow-up. + +### Measurements + +**Replay of this repo's history, 2026-10-20.** Stack: the sandbox above, the chunker at this +branch, **a stand-in tokenizer** (one token per word, run of whitespace or punctuation mark) +because bge-code-v1's could not be downloaded. Token counts decide tier-2/3 window boundaries and +which tier-1 symbols are split, so the slice numbers are indicative, not production. For the last +40 non-merge commits before `871b55c` touching `src/*.py` (90 file edits), each edited file was +chunked before and after: + +| | Chunks after the edit | Unchanged by the key | Share | +|---|---|---|---| +| Tier 1, same (scope, `chunk_text_hash`): vector kept (§5) | 4,774 | 4,252 | 89% | +| Tiers 2+3, cache hit with ADR-045's key | 1,578 | 550 | 35% | +| Tiers 2+3, cache hit with this ADR's key (§3) | 1,578 | 665 | 42% | + +The script is not committed; it is about 30 lines over `chunk_all_tiers` and `git show`. From 2f8c803769e854d6e690df63c4f0234e8e10b607 Mon Sep 17 00:00:00 2001 From: Ethan Blauw Date: Tue, 20 Oct 2026 16:45:26 -0500 Subject: [PATCH 6/6] ADR-051 accepted Co-Authored-By: Claude Opus 5.5 (1M context) [ADR-051] --- docs/adr/ADR-051-stable-change-keys.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/ADR-051-stable-change-keys.md b/docs/adr/ADR-051-stable-change-keys.md index c328d47..a92be38 100644 --- a/docs/adr/ADR-051-stable-change-keys.md +++ b/docs/adr/ADR-051-stable-change-keys.md @@ -1,6 +1,6 @@ # ADR-051: Change Keys That Ignore Line Endings and Slice Positions -**Status:** proposed +**Status:** accepted **Date:** 2026-10-20 **Branch:** `feature/adr-051-stable-change-keys` **Reviewer:** @edb