Repository navigation
ADR-051: change keys that ignore line endings and slice positions (B-054, B-048, B-034) - #91
Merged
Merged
Conversation
Proposed design for B-054, B-048 fix 1 and B-034 tier 1. Co-Authored-By: Claude <noreply@anthropic.com> [ADR-051]
B-054: a checkout whose line endings differ from the indexed one no longer reprocesses those files. md5_file normalizes \r\n and a lone \r to \n, as both readers already do for the text they chunk, so chunks and summary keys were never affected; only the file key was. An LF-only file hashes as before. An index built before this re-keys in place on its first run: a file whose stored hash is the raw MD5 of its current bytes gets the new key without being re-indexed. index_meta.file_key marks the index done. Git mode keeps blob ids as its file key. Co-Authored-By: Claude <noreply@anthropic.com> [ADR-051 src/incremental_indexer.py src/source.py]
B-048 fix 1, as ADR-029 designed it: summary_cache_key and the text sent to the summarizer turn a slice header's (Part i/N) into (Part), so adding or dropping a window no longer re-summarizes every slice whose code did not move, and a cached summary cannot quote a stale position. The embedded and stored text keep the part number. Existing caches carry over the way ADR-045's did: the one-time re-key now copies each indexed chunk's summary from whichever older key it has to the new one, under index_meta.summary_key = "no-lines-no-part". Co-Authored-By: Claude <noreply@anthropic.com> [ADR-051 src/db.py src/incremental_indexer.py]
B-034, tier 1: before a modified file's vectors are purged, its tier-1 vectors are read back from the FAISS id map with their chunk_text_hash. A re-ingested chunk with the same id and hash gets that vector instead of an embedding; a moved symbol keeps its vector and gets its new line range. Vectors are still added in _write_plan, before the file's rows, so vectors equal rows and the overlapped build writes the same bytes as the serial one. Summary vectors and tiers 2/3 are embedded as before. Co-Authored-By: Claude <noreply@anthropic.com> [ADR-051 src/incremental_indexer.py]
Co-Authored-By: Claude <noreply@anthropic.com> [ADR-051]
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> [ADR-051]
…ble-change-keys [ADR-051 src/hybrid_retriever.py src/iterative_retriever.py]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements ADR-051 (backlog B-054, B-048 fix 1, B-034 tier 1).
What changes
\r\nand\rnormalized to\n, so an LF-only file hashes as before. Git mode keeps the blob id.index_meta.file_key = "md5-lf". Nothing is re-embedded or re-summarized.CHUNKER_VERSIONis not bumped (§2). No chunk's text changes, and a bump would tell every user to run a full rebuild for nothing.(Part i/N)becomes(Part).index_meta.summary_key = "no-lines-no-part".chunk_text_hashmatch. The vectors-equal-rows invariant (ADR-037) is kept.Tests
Still open (left in the ADR)
🤖 Generated with Claude Code