Skip to content

perf(core): write the FTS index once per ingest, in rowid order - #23

Merged
hsusul merged 2 commits into
mainfrom
p1-3b-batch-fts
Oct 26, 2026
Merged

hsusul merged 2 commits into
mainfrom
p1-3b-batch-fts

Conversation

@hsusul

@hsusul hsusul commented Oct 26, 2026

Copy link
Copy Markdown
Owner

Why

FTS5 flushes its pending terms at each statement that writes it. Ingest inserts one search row per statement, so the insert trigger on search_document wrote one index segment per row. A profile of a large session replace put half its samples in that flush.

What changed

  • Migration 0015 drops the search_document_ai trigger.
  • persist_rows and append_rows read the highest search row id after their deletes. At the end they index every row above it with one INSERT ... SELECT ... ORDER BY id.
  • The append path reads that id after it deletes a changed title, since the delete can free the top id for reuse.

Scope

  • Covers the full replace path and the append path. Those are the only places that insert search rows.
  • The DELETE and UPDATE triggers stay. Each delete is one statement, so it already flushes once. Nothing updates search_document.
  • First-scan time barely moves. It is bound by per-session commits, not FTS.

Tradeoffs

  • Replacing the DELETE trigger with one bulk FTS 'delete' was measured and rejected. A 20k-row replace took 1.47 CPU-s either way, and every delete path would have needed its own FTS write.

Blast Radius

Every ingest write path changes how it reaches the index. A missed insert would drop rows from search. The tests run FTS5's content integrity check after ingest, append, title change, replace, reassignment, forget, and migration.

Verification

  • cargo test --workspace passes, including the new fts_sync tests. Each test searches for a word unique to each row and runs integrity-check with rank = 1, which compares the index with search_document. The plain check misses an unindexed row.
  • Release lorectl scan, full replace of a 20k-message Claude session: 2.60 → 1.47 CPU-s, median of 5 interleaved runs per side.
  • First scan of 3000 Claude and 1000 Codex synthetic sessions: 5.57 → 5.28 CPU-s, median of 5 interleaved runs.

FTS5 flushes its pending terms at each statement that writes it, and
ingest inserts one search row per statement. The AFTER INSERT trigger
on search_document therefore wrote one index segment per row.

Migration 0015 drops that trigger. persist_rows and append_rows now read
the highest search row id after their deletes, and index every row above
it with one INSERT ... SELECT ... ORDER BY id at the end. The DELETE and
UPDATE triggers stay: each delete is one statement, so it already
flushes once, and nothing updates search_document.
@hsusul
hsusul merged commit 587b7cb into main Oct 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant