Repository navigation
perf(core): write the FTS index once per ingest, in rowid order - #23
Merged
Merged
Conversation
FTS5 flushes its pending terms at each statement that writes it, and ingest inserts one search row per statement. The AFTER INSERT trigger on search_document therefore wrote one index segment per row. Migration 0015 drops that trigger. persist_rows and append_rows now read the highest search row id after their deletes, and index every row above it with one INSERT ... SELECT ... ORDER BY id at the end. The DELETE and UPDATE triggers stay: each delete is one statement, so it already flushes once, and nothing updates search_document.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
FTS5 flushes its pending terms at each statement that writes it. Ingest inserts one search row per statement, so the insert trigger on
search_documentwrote one index segment per row. A profile of a large session replace put half its samples in that flush.What changed
search_document_aitrigger.persist_rowsandappend_rowsread the highest search row id after their deletes. At the end they index every row above it with oneINSERT ... SELECT ... ORDER BY id.Scope
search_document.Tradeoffs
'delete'was measured and rejected. A 20k-row replace took 1.47 CPU-s either way, and every delete path would have needed its own FTS write.Blast Radius
Every ingest write path changes how it reaches the index. A missed insert would drop rows from search. The tests run FTS5's content integrity check after ingest, append, title change, replace, reassignment, forget, and migration.
Verification
cargo test --workspacepasses, including the newfts_synctests. Each test searches for a word unique to each row and runsintegrity-checkwithrank = 1, which compares the index withsearch_document. The plain check misses an unindexed row.lorectl scan, full replace of a 20k-message Claude session: 2.60 → 1.47 CPU-s, median of 5 interleaved runs per side.