feat(postgres): vector indexes and SearchVectors - #307
Merged
Conversation
yesyayen
marked this pull request as ready for review
August 25, 2026 16:21
yesyayen
requested review from
LeeroyHannigan,
amrith,
c33howard,
jcshepherd,
pdf-amzn and
robinnsc
as code owners
August 25, 2026 16:21
| // is no steady-state traffic that could strand it. | ||
| let queue_empty = if delay_ms == 0 && metas.iter().any(|(_, status)| status == "ACTIVE") { | ||
| let pending: Option<(i32,)> = | ||
| sqlx::query_as("SELECT 1 FROM gsi_pending WHERE table_id = $1 LIMIT 1") |
Collaborator
There was a problem hiding this comment.
Should-fix - maintenance runs through the gsi_pending queue, so a search right after a write can come back empty until it drains. Worth a line in the docs so nobody reads that as data loss.
Collaborator
Author
There was a problem hiding this comment.
Agreed, worth documenting. It's the same eventual-consistency model as a GSI, so the write is never lost, just queued. Added a note to the vector propagation row in differences-from-dynamodb.md saying exactly that: the window is ordering, not data loss. Done in commit 941821e.
yesyayen
force-pushed
the
feat/vector-shared-groundwork
branch
from
August 26, 2026 23:34
08ad04d to
85b8f60
Compare
yesyayen
force-pushed
the
feat/vector-postgres
branch
from
August 26, 2026 23:35
d3c307b to
941821e
Compare
Catalog migration 002 stores vector index metadata at catalog version 0.0.3. pgvector is installed opportunistically at init (CREATE EXTENSION IF NOT EXISTS, failure tolerated with a notice) and probed once at startup; the cached capability is what as_vector_search reads, so a server without the extension refuses vector features fail-closed instead of failing mid-request.
CreateTable, DescribeTable, DeleteTable, and UpdateTable-delete for vector indexes, gated on the startup capability. Hardening folded in: the catalog migration is replay-safe, the catalog decode is shared and refuses an unknown status, only the undefined-object SQLSTATE classifies as missing pgvector, an empty search schema collapses once for every backend, and the engine maps an unsupported backup operation to a validation error. Restore refuses a backup that carries vector indexes. Covered by storage-level tests, a live-deployment migration test, version-gate tests in both directions, and a CI job that keeps the refusal surface tested after the pgvector image flip.
Vector index data tables typed as pgvector vector(N) columns, with the pgvector crate for typed embedding binds. Maintenance runs at all six write sites through one entry point, and the capability is declared where the server can serve it, so PostgreSQL answers SearchVectors. The three design-outs are binding. Index membership comes from a fresh catalog read per write, never from cached key info, and that read also decides whether the write needs a transaction at all. Stored bytes that cannot enter an index are non-indexable rather than fatal. A write to a CREATING index always enqueues, at any delay, because the backfill holds an older snapshot of the same item and its plain INSERT would collide. Data migration 004 adds the hold that keeps the queue off a table whose index is building; the catalog cannot be joined from a claim transaction because it is a different database.
Backfill and status sequencing on the shared lifecycle primitives, with keyset pagination over the full primary key. Build ownership is a session-scoped advisory lock; the build session is owned for the whole build, and every route that abandons a build gives back its queue hold, with a self-healing sweep for the routes nothing enumerated. Stuck builds rebuild at runtime, not only at startup, and the scanning phase is re-asserted before a recovery rebuild. Query norms compute in f64 and bind as parameters.
Deleting a vector index during the resource-allocation phase refuses with the measured ResourceInUseException; during backfilling the delete is accepted. The phase is observable, a test can hold the allocation phase open, and the delete-phase wire test is backend-blind.
Query norms compute in f64 on both backends and pgvector's NaN cannot reach the client. Also covers the delayed queue path for a vector-only table and closes the remaining review nits.
Corrects comments and manual sections that described code as it used to be: the out-of-pool session count, the underflow degradation, the backend split hidden by two vague rows, and a backend-author section that was wrong about a shipped backend. Adds a test asserting the ranking a tiny query vector must produce.
… constants The guard sees every literal form it claims to cover, reaches every file, and does not arm on an inline marker. Includes the documentation corrections the guard and the audit caught, and tells users the feature exists.
…on samples The capability refusal is two strings, kept greppable in the source and stated in the ADR; F-19 and F-20 carry measured bounds; the extension-install comments claim only what the measurement supports; sample version output follows the workspace version.
Five behavior fixes from the review of this PR, on both backends wherever the rule is shared. A write no longer applies inline while its table has queued rows. Queue order only ordered queued rows against each other, so once an index flipped to ACTIVE and released its hold, an older queued row could be applied after a newer inline write to the same item and leave the index disagreeing with the base table until that item was written again. Completing a build whose index was deleted now drops the data table the build recreated. A rebuild recreates that table after reloading the definition, so a delete landing in between left a table nothing referenced. UpdateTable refuses an index name another index family already holds. CreateTable already enforced this across families; UpdateTable consulted only its own catalog table, so a GSI and a vector index could share a name and therefore an ARN. A write whose vector data table vanishes mid-apply is skipped under a savepoint. The failed statement used to abort the whole transaction, so the write answered InternalServerError where the service answers normally. update_table reads the stored billing mode once, under the row lock it already holds, rather than re-querying it twice.
Six tests for the fixes in the previous commit, each proven to fail against the unfixed code first: the queue ordering gate and its empty-queue control, deleted-index build completion on both backends, and the two cross-family index name directions. Build completion is reachable from a test through a hidden entry point, alongside the two that already exist for the same reason, because the branch it exercises only runs when the catalog row is already gone and the alternative is timing the race. Two SQLite assertions compared sqlite_master.name against a name that arrives already quoted for DDL use, so they matched nothing and passed whatever the real state was. Both now use the bare name through one helper, and each carries a positive control asserting presence where the table must exist, so an assertion that cannot fail would itself fail. The allocation-phase test resets its global lever even when the body panics, which previously left every later test in the serial run with a four second phase and failures pointing at the wrong cause. Ledger: F-20's fix is coupled to the queue gate, since gating SQLite's inline branch on ACTIVE is what first produces the queued rows the gate protects, and landing it alone would swap one race for another.
…rate The extension attempt ran as the application role, which the code's own doc comment proves can never succeed on a stock server: pgvector's control file does not mark it trusted, so a non-superuser is refused even on a database its role owns. init and migrate hold admin credentials the whole time and never used them for this, so a deployment initialised with a superuser admin still came up with vector search off, and every vector wire test failed in CI while the suite pinned EXTENDDB_EXPECT_VECTORS=1. The attempt now runs as the admin role first and falls back to the application role, which is the shape a managed platform allowlisting pgvector for the database owner needs. Refusal of both stays a notice with the same hint, and serve-time code still never attempts it.
Carried across the rebase onto the reviewed groundwork tip, which itself moved onto a main that gained the vector lifecycle and capacity fixes: - the vector phase refusal produces IndexesInUse, the folded in-use variant that carries the whole wire message - the PostgreSQL UpdateTable-create path applies the same min-CREATING floor SQLite gained on main, passed to the shared driver beside the flip it delays - CreateTable's response initializer reports restore_summary: None, the field the restore work added to TableDescription - doc sample version literals follow the workspace to 0.1.10, caught by the branch's own doc guard - the F-20 ledger entry follows main: the wedge it originally tracked is fixed by KeepExisting, and what remains is the ordering coupling to the queue-emptiness gate - the propagation row in the differences doc states the queue-emptiness consequence plainly: a search-after-write window exists even at delay zero while queued rows drain, and it is ordering, not data loss
LeeroyHannigan
force-pushed
the
feat/vector-postgres
branch
from
August 27, 2026 12:39
941821e to
cba3550
Compare
LeeroyHannigan
changed the base branch from
feat/vector-shared-groundwork
to
main
August 27, 2026 12:39
LeeroyHannigan
approved these changes
Aug 27, 2026
yesyayen
added a commit
to yesyayen/extenddb
that referenced
this pull request
Aug 28, 2026
Replay of the feat/wasm-shared-crates gating pass onto current main: - auth takes HeaderMap from http, not axum - cache: moka/tokio native-only; wasm32 uses a pass-through SWR shim - core: uuid js + time wasm-bindgen features on wasm32 - engine: tokio native-only; import/export surface gated with it - storage: tokio/tokio-util/bcrypt/rand/aes-gcm native-only; bootstrapper, server_components, backend registry and hooks gated off wasm32 New on top of that pass (post-ExtendDB#306/ExtendDB#307 call sites): - storage vector_lifecycle build driver (tokio::time) is native-only; the pure row-shape helpers stay on both targets
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
PostgreSQL implements the vector index surface end to end: CreateTable / UpdateTable create and delete with a real backfill, DescribeTable status reporting, SearchVectors exact scan (pgvector, one
vector(N)data table per index), index maintenance at all six write sites through the sharedgsi_pendingqueue, crash recovery, and cross-process build ownership (advisory lock plus heartbeat; the lock holds its own session because a pooled connection's session outlives its checkout, which would otherwise defeat the stuck-build sweep).pgvector is runtime-detected:
initattemptsCREATE EXTENSIONand tolerates failure;serveprobes once and caches. Without the extension every vector operation fails closed at validation with a typed error, proven over the wire by a CI job on a plain postgres image. The catalog migrates 0.0.2 -> 0.0.3, and the version gate refuses symmetrically in both directions.Because the shared wire suite is backend-blind, SQLite gains the same measured rules in this PR: the allocation-phase delete refusal, the observable CREATING with
Backfilling: falsewindow, and f64 score arithmetic. Zero vectors index and score 1.0 as measured. Ties break on the base key; the service's tie order is measured-unstable, so any order conforms.One deliberate residual is documented in
docs/differences-from-dynamodb.md: pgvector's internal f32 accumulation collapses cosine distance to three values for query vectors whose squares underflow; SQLite computes the true value. Known follow-ups are tracked indocs/technical-debt.md(F-19: SQLite restore silently drops vector indexes where PostgreSQL refuses; F-20: SQLite's zero-delay inline maintenance bypasses the backfill hold).Behavior: today vs Amazon DynamoDB
Every value below was captured from Amazon DynamoDB via raw signed requests (us-west-2, 2026-08-19).
ResourceInUseException: Attempt to change a resource which is still in use: Index creation is in resource allocation phase. Retry deletion during backfilling phase or when the index is active. Table: {table} Index: {index}"Score": nullon HTTP 2001.0(SQLite; PostgreSQL n/a)1.0, zero on either sideWhy
The second storage backend for vector search, built on the lifecycle primitives extracted in #306 (ADR-0005): the backend implements a cursor and apply hooks; the shared drivers own backfill, status sequencing, recovery, and holds. Storage and scoring decisions (vector(N) layout, operator-derived scores, f64 norms, fail-closed detection) are recorded in ADR-0006.
Stacked on #306: the first 19 commits are that PR; review the last 9 here.
Closes # n/a
Testing done
Measured against the live service throughout: 91 recorded wire pairs, including the phase-dependent delete, net-state billing, sparse-index semantics, and the dimension-mismatch message family.
EXTENDDB_EXPECT_VECTORS=1(58 vector tests). The PostgreSQL leg was run against PostgreSQL 15.18 with pgvector 0.8.0; CI runs the same suite on thepgvector/pgvector:pg16image, and a separate job runs the fail-closed refusal leg on a plainpostgres:16image (the new CI job shape)."Score": nullfix was verified by watching the test fail against the unfixed build.cargo test --workspace,cargo fmt --all -- --check, andcargo clippy --all-targets -- -D warningspass at the final tip.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).ADR / RFC: docs/adr/0006-pgvector-storage-and-scoring.md (lifecycle ownership: docs/adr/0005-index-build-lifecycle-ownership.md, via #306; RFC context: #236)
Breaking changes
n/a
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.