Repository navigation
paperless.2: filtered-KNN performance PRs and AVX cosine dispatch - #2
Merged
Merged
Conversation
_cmp returned the i64 difference of its arguments as an int, and it orders the `rowid in (...)` list of a KNN query before each row binary-searches it. Two rowids 2**32 apart narrow to 0 and compare equal, so a row that is not in the list matches: `rowid in (1, 2)` also returned rowid 2**32 + 1. A difference wider than an int also changes sign, so the qsort leaves the list out of order and the search misses rows that are in it: of 150 rowids drawn from +/-2**62, 29 to 54 were missing per run. A difference wider than i64 overflows. _cmp now compares its arguments. test_vec0_knn_rowid_in_far_apart covers both: an extra row 2**32 away, and five rowids 3e9 apart with alternating signs, one of which the old comparator loses. It fails on the old _cmp and passes on this one.
A KNN query's `n in (...)` on an integer metadata column compared every row of every chunk against every value in the list, so its cost grew with rows times list length. The list is now sorted once when vec0Filter_knn reads it, and each row binary-searches it, as `rowid in (...)` already does. test_vtab_in_integer_list checks the rows a 16-value list keeps, with values 2**32 apart; it fails on a build that pairs this search with the comparator _cmp had before the previous commit.
…rows A KNN query opened and read each chunk's vectors blob before applying its rowid in (...) and metadata constraints, so a filter that leaves a chunk with no candidates still paid the full blob read and only saved the distance computation. The filters are now applied first, and a chunk whose bitmap is empty is skipped before its vectors are opened.
min_idx() had two bodies: by default a scan of the whole chunk once per result, k times n comparisons, and under SQLITE_VEC_EXPERIMENTAL_MIN_IDX a binary heap of the k kept so far, n log k. With the default chunk size of 1024, a KNN query with k of 1024 or more spent about a million comparisons per chunk: on an in-memory 10,000-row float[768] table, k = 1000 and k = 4096 took 22.3 and 22.8 ms against 8.1 ms for k = 1. The heap compares distances alone, so it returns tied rows in another order than the scan: over 1,000 KNN queries on tables built to tie heavily, a build with the option returns other rows or another order for 779 of them. This keeps the heap as min_idx()'s only body and orders it with min_idx_after(): by distance, between equal distances the larger index first, as the scan's <= chose, and a NaN distance after every number. The cosine distance to a zero vector is 0 / 0, a NaN, and the scan returned such a row first whenever it was the first candidate left in its chunk: with a zero vector in slot 0, a k = 2 query for [1, 0] returned it ahead of an exact match and dropped the next nearest row. The bitmap of taken slots the scan needed goes with it, from vec0's KNN loop, the rescore KNN, and the declarations in tests/sqlite-vec-internal.h and tests/unittest.rs. merge_sorted_lists(), which merges each chunk's nearest rows into the ones kept from the chunks before it, ranks a NaN after every number too, as mceachen's photostructure/sqlite-vec@a791836 does in that fork: it took a later chunk's NaN ahead of the numbers still unmerged, so a zero vector in the second chunk came back ahead of the first chunk's rows. Measured on that table (medians of 5 interleaved runs of 7 queries), k = 1000 and k = 4096 take 8.6 and 8.9 ms, k = 100 8.2 ms instead of 8.6, and k = 1 8.1 ms either way; the heap as the option builds it, without the order, takes 8.5 ms at both. Over 4,000 random KNN queries on tables built to tie heavily (vectors on a 5 by 3 grid, chunk sizes 8 to 64, deletes, metadata filters, k from 1 to 1000), every result matches the default build row for row. tests/test-knn-topk.py checks the k nearest against a brute-force order for k below, at and above the chunk size, with and without a filter, pins the order of tied rows, and checks that a zero vector in a cosine column comes back last, within its chunk and behind an earlier chunk's rows; on main both of those tests fail, with the zero vector first.
A vec0 full scan read each row's vector through vec0_get_vector_data(), which for a flat column looks up the row's chunk, opens the chunk's vectors blob, reads one vector and closes the blob again. A chunk's blob holds chunk_size vectors (3 MiB for 1024 float[768]), and the first read through a newly opened handle walks the blob's overflow pages up to the row's offset, so the opens dominated the scan. The scan already visits rows in (chunk_id, chunk_offset) order, and for a flat column the cursor now keeps one blob handle per vector column, moved to the next chunk with sqlite3_blob_reopen(). Each row's chunk position is looked up when its vector is read: the scan's rowid statement sorts every row when the scan starts, so its positions would miss a delete or insert the same connection runs during the scan. A write that replaces the chunk's row expires the handle, and a rollback of a transaction that changed the schema invalidates it (SQLITE_ABORT, or SQLITE_ABORT_ROLLBACK with extended result codes on); the read is then retried once on a newly opened handle. Any other failure to move the handle, such as an interrupt, fails the read. Rescore, IVF and DiskANN columns keep their vectors in tables of their own and are read row by row as before. The position lookup and that failure handling follow photostructure/sqlite-vec@f793711 by Matthew McEachen, which corrected this change after it merged there as photostructure#1. On a 10,000-row float[768] table (medians of 3 interleaved runs of 7 queries), selecting every vector takes 12.6 ms instead of 762.4 ms in a database file and 5.4 ms instead of 104.7 ms in memory, and vec_distance_cosine() against each 33.8 ms instead of 853.5 ms and 24.5 ms instead of 111.8 ms. tests/test-fullscan.py checks every row's vectors against point lookups across chunks with gaps and two vector columns, scans rescore and DiskANN columns beside a flat one, reads a chunk rewritten in the middle of a scan, fails on a row deleted during the scan instead of returning another row's vector, and fails a scan cancelled while its handle moves to the next chunk; tests/test-unit.c scans across a rollback that changed the schema, with extended result codes off and on.
vec0Update_Delete_ClearValidity() tested `bx >> (chunk_offset % CHAR_BIT)`, which is nonzero whenever a later row's bit in the same byte of the chunk's validity bitmap is set. Deleting a row whose own bit was already clear therefore succeeded instead of failing with "validity bit is not set". Only a damaged _chunks table has such a row. The check now tests the row's bit alone, `(bx >> (chunk_offset % CHAR_BIT)) & 1`. Ported from photostructure/sqlite-vec@f6a3244 by Matthew McEachen. test_vec0_delete_errors sets a chunk's validity byte to x'02', clearing row 1's bit while row 2's stays set, and expects the delete of row 1 to fail. It fails on main and passes here. The full suite passes (239 passed, 66 skipped, 204 snapshots).
Every DELETE cleared the row's validity bit through one blob handle, and vec0Update_Delete_DeleteChunkIfEmpty() then opened the chunk's validity blob a second time and read all of it to learn whether the chunk was now empty. A byte that still holds another row's bit keeps the chunk, as it does for most deletes, so vec0Update_Delete_ClearValidity() now reports whether the chunk is empty, reading the rest of the bitmap through the handle it already holds and only once the cleared byte is zero, and vec0Update_Delete_DeleteChunk() deletes the chunk when it is. Ported from photostructure/sqlite-vec@4a8ffe0 by Matthew McEachen. Deleting 20,000 rows of a float[8] table one statement at a time takes 48,232 instructions per DELETE instead of 52,557 under callgrind, 8.2% fewer, where two copies of main's build count the same. Deleting 100,000 rows took 0.54 s of CPU instead of 0.64 s (medians of 11 interleaved runs, where two copies of main's build differed by 0.0%). test_delete_keeps_chunk_while_another_validity_byte_has_rows clears either byte of a 16-row chunk's bitmap and checks that the chunk stays with the other byte's rows, then goes once both are clear. The full suite passes (240 passed, 66 skipped, 204 snapshots), also with the IVF index compiled in (302 passed, 4 skipped) and with vec0 built with AddressSanitizer and UBSan.
vec0 checks a KNN query's constraints itself and sets aConstraintUsage[].omit, which tells SQLite a constraint "is assumed to be fully handled by the virtual table and might not be checked again" (sqlite3.h). Each constraint vec0 claims therefore has to keep exactly the rows SQLite would keep for the same WHERE clause. They did not: - `n = ?` bound to NULL or 'abc' kept n = 0, `n = 5.5` kept n = 5, `n < 5.5` dropped n = 5, and `n in (5.5, 6)` kept n = 5. - `f > 'abc'` kept every positive f, though every number is less than any text. - `b = 2` kept the true rows. - `t in (null)` kept t = '', and `t = x'78'` kept t = 'x'. - `t < ?` bound to NULL crashed (a NULL pointer read) once a chunk held a text longer than 12 bytes, and <, <=, > and >= failed with "Could not filter metadata fields" once a chunk held a text of exactly 12 bytes. - Text comparisons ignored the query's collation: `t = 'abc' COLLATE NOCASE` missed 'ABC', and `COLLATE RTRIM` missed 'abc '. - A partition key compared as stored: `p = '5'` missed p = 5, `s = 5` missed s = '5', and collations were ignored. - `rowid in (5.5, 'abc', null)` matched rowids 5 and 0, and a point lookup `rowid = ?` matched rowid 0 for NULL, 'abc' and a blob, and rowid 5 for 5.5. With a text primary key, an id in `id in (...)` that names no row emptied the whole result. - `distance > ?` bound to NULL kept every row and `distance < 'abc'` none, and distances compared in f32 (the TODO beside that cast) kept the row at distance 2 for `distance <= 1.999999999999`, which SQLite drops. vec0 now follows SQLite's R-Tree, the extension that checks its own constraints: - It declares its columns with their types, as rtreeInit() does: metadata columns boolean, integer, float or text, partition keys and a text primary key with theirs, rowid integer and distance real. So SQLite's own checks give a value the affinity vec0 gives it. - Each filter gives the constraint's value its column's affinity and compares by https://www.sqlite.org/datatype3.html#comparison_expressions: NULL compares with nothing, INTEGER and REAL compare exactly, as sqlite3IntFloatCompare() does, every number sorts before any text, and every text before any blob. A rowid lookup takes the value as rtreeFilter() does: numeric affinity, then an integer or a real equal to one. - Text compares under the constraint's collation, from sqlite3_vtab_collation(): BINARY, NOCASE and RTRIM as binCollFunc, nocaseCollatingFunc and rtrimCollFunc do, and any other collation is refused with an error. SQLite names no collation for !=, so vec0 keeps the rows unequal under BINARY and leaves != for SQLite to check again. SQLite orders text under BINARY by the bytes of the database's encoding, so in a UTF-16 database vec0 compares the bytes SQLite stores for each value, converting its UTF-8 with SQLite's own READ_UTF8, WRITE_UTF16LE and WRITE_UTF16BE (src/utf.c), and orders a text longer than its 12-byte prefix by its full text. vec0 reads the encoding when it opens a table, from sqlite3_value_encoding() of a string literal on SQLite 3.41 and later, which needs no PRAGMA an authorizer might deny, and from PRAGMA encoding before that, where no row means UTF-8 only in a build with SQLITE_OMIT_UTF16. A read that fails fails the constructor with its error, and an interrupted one reports as an interrupt. This read follows photostructure/sqlite-vec@8156329 by Matthew McEachen, which corrected it after this change merged there as photostructure#4. - A distance compares as the double SQLite receives. The tests compare each filter with a plain table declaring the same types and with SQLite's own scan of the vtab: every operator over NULL, integers, reals, numeric and other text and blobs, texts around the 12 bytes a chunk keeps inline, each collation, IN lists, partition keys, rowid and text id lookups, distances a hair either side of a row's, and UTF-16 databases, where every filter is compared at each k and with LIMIT. A KNN query whose k is the number of matching rows must return all of them, so a filter that keeps any other row fails. A connection whose authorizer denies PRAGMAs opens the table and keeps its text range filters, as on main, and creating a table fails when the encoding cannot be read. The tests fail on main and pass here. The full suite passes (247 passed, 66 skipped, 204 snapshots), also with vec0 built with AddressSanitizer and UBSan. make test-unit does not build on main (tests/sqlite-vec-internal.h redeclares the rescore types), and this change adds no error to it.
distance_cosine_float() had a NEON kernel but no AVX one, so x86 builds with SQLITE_VEC_ENABLE_AVX ran the scalar loop for cosine while L2 used l2_sqr_float_avx(). cosine_float_avx() takes 16 floats per step into two independent sets of dot, |a|^2 and |b|^2 accumulators, uses FMA when the build has it (__FMA__) and mul+add otherwise, and finishes the tail of fewer than 16 elements in scalar code. It is used for 16 or more dimensions. test_distance_cosine_float() now checks lengths around the 16-element block (15 to 389, odd tails included) against a double-precision reference.
… KNN filters keep none of its rows
…n rowids are far apart
…n's `in (...)` list
… the min_idx() heap
…rough its validity handle
…blob handle per chunk # Conflicts: # tests/test-unit.c
The kernel had no target attribute and was called unconditionally, so a CPU without AVX2 would SIGILL (same class as asg017#307). Also add CHANGELOG-paperless.md. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… reinsert Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…an jobs The paperless line ships Linux and macOS wheels only. Also move the macOS arm64 job to macos-15, since macos-14 jobs were being cancelled without a runner. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
macos-15 jobs sat unassigned until cancelled, while macos-latest jobs get runners. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the upstream performance work into the integration line for
v0.1.10-paperless.2. SeeCHANGELOG-paperless.md.Merged as separate merge commits: asg017/sqlite-vec asg017#330, asg017#327, asg017#328, asg017#329, asg017#364, asg017#337, asg017#358, asg017#359, asg017#336.
Fork-only changes:
target("avx,avx2")plus__builtin_cpu_supports("avx2")). As merged it had neither, the same pre-AVX SIGILL class as Fix SIGILL on pre-AVX x86_64: function-level target attributes + runtime CPU detection asg017/sqlite-vec#307.tests/test-loadable.py:rowid INwith adistanceconstraint, and filtered KNN after delete and reinsert.CHANGELOG-paperless.md.Measured, 3072-dim, vs the 0.1.9 wheel, ids and order identical: one-document filtered KNN warm 54 to 79 ms -> about 0.5 ms; cold on HDD ZFS 2.8 to 3.4 s -> 0.6 to 1.3 s (1024-row chunks), 5 to 20 s -> 0.4 to 1.2 s (32-row chunks).
Left out: asg017#338, asg017#316, asg017#339, asg017#243, asg017#269.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com