feat(postgres): vector indexes and SearchVectors

.agents/skills/extenddb/SKILL.md

@@ -143,7 +143,7 @@ This skill presents commands but does not execute state-changing operations (`ex
 | `references/samples/02-sample-app.md` | Nine-stage lifecycle walkthrough |
 | `references/samples/03-stream-consumer.md` | Streams demo, two-client pattern |
 | **Troubleshooting** | |
-| `references/troubleshooting/01-symptom-index.md` | 16-symptom keyword-to-category lookup |
+| `references/troubleshooting/01-symptom-index.md` | 17-symptom keyword-to-category lookup |
 | `references/troubleshooting/02-postgres-symptoms.md` | Connection refused, password auth, migration |
 | `references/troubleshooting/03-catalog-symptoms.md` | Version mismatch, not initialized, already exists |
 | `references/troubleshooting/04-startup-symptoms.md` | Address in use, TLS, permissions, daemonize |

.agents/skills/extenddb/references/troubleshooting/01-symptom-index.md

@@ -2,7 +2,7 @@
 
 ## 1. Purpose
 
-This index maps each of the 16 known extenddb symptoms to the category file that holds the verbatim Cause and Fix from `docs/troubleshooting.md`. To use it, grep this file for the user's error text, follow the link to the category file, and present the entry to the user. The skill never executes a remediation command on the user's behalf. Requirement 14.4 applies to every entry.
+This index maps each of the 17 known extenddb symptoms to the category file that holds the verbatim Cause and Fix from `docs/troubleshooting.md`. To use it, grep this file for the user's error text, follow the link to the category file, and present the entry to the user. The skill never executes a remediation command on the user's behalf. Requirement 14.4 applies to every entry.
 
 ## 2. Symptom table
 
@@ -24,6 +24,7 @@ This index maps each of the 16 known extenddb symptoms to the category file that
 | 14 | UnrecognizedClientException | `06-auth-symptoms.md#unrecognizedclientexception` | `UnrecognizedClientException: The security token included in the request is invalid` |
 | 15 | AccessDeniedException | `06-auth-symptoms.md#accessdeniedexception` | `AccessDeniedException: User: <ARN> is not authorized to perform: <action>` |
 | 16 | Connection pool exhausted / HTTP 500 under load | `07-runtime-symptoms.md#connection-pool-exhausted` | `HTTP 500 on all requests under heavy load` |
+| 17 | Vector indexes are not supported / SearchVectors is not supported | `05-feature-gate-symptoms.md#vector-unsupported` | `Vector indexes are not supported by this storage backend` |
 
 
 ## 3. Per-entry summaries
@@ -142,6 +143,13 @@ The cause and fix summaries below are paraphrased for quick scanning. The catego
 **Fix summary:** Raise `pool_size` under `[storage.postgres]` in `extenddb.toml` and investigate long-running queries in `pg_stat_activity`.
 **Full entry:** `references/07-runtime-symptoms.md#connection-pool-exhausted`
 
+### Vector indexes are not supported
+
+**Error text:** `Vector indexes are not supported by this storage backend` (or `SearchVectors is not supported by this storage backend`)
+**Cause summary:** Vector indexes need the pgvector extension on the PostgreSQL data database, and the server probes for it once at startup and caches the answer.
+**Fix summary:** Install the extension for the server version, run `CREATE EXTENSION vector;` on the data database, then restart ExtendDB, because the probe result is cached at startup.
+**Full entry:** `references/05-feature-gate-symptoms.md#vector-unsupported`
+
 ## 4. Unknown-symptom fallback
 
 If the user's error text does not match any entry above, ask the user to pull the last 100 lines of the extenddb log, then retry the lookup on the new text.

.agents/skills/extenddb/references/troubleshooting/05-feature-gate-symptoms.md

@@ -39,3 +39,25 @@ paths = ["/path/to/exports"]
 ```
 
 **Source:** `docs/troubleshooting.md`, section "`Export is disabled. Configure [export] paths in extenddb.toml to enable.`", last synced 2026-05-12.
+
+### Vector indexes are not supported by this storage backend
+
+<a name="vector-unsupported"></a>
+
+**Error text:**
+```
+Vector indexes are not supported by this storage backend
+SearchVectors is not supported by this storage backend
+```
+
+**Cause:** Vector indexes need the pgvector extension on the PostgreSQL **data**
+database. Support is a property of the server, not of the ExtendDB build, and the
+server probes for the extension once at startup and caches the answer.
+
+**Fix:** Install the extension for the server version (for example
+`postgresql-16-pgvector`), run `CREATE EXTENSION vector;` on the data database, then
+**restart ExtendDB**, because the probe result is cached at startup. The startup log
+line `pgvector ... detected` or `pgvector not installed ...` says which answer the
+running server is serving.
+
+**Source:** `docs/troubleshooting.md`, section "`Vector indexes are not supported by this storage backend`", last synced 2026-08-20.

.github/workflows/integration.yml

@@ -13,7 +13,11 @@ jobs:
     runs-on: ubuntu-latest
     services:
       postgres:
-        image: postgres:16
+        # pgvector's own image, which is postgres:16 plus the extension. The
+        # backend serves vector indexes where the extension is present, so this is
+        # what makes the positive paths reachable in CI. The novector job below
+        # keeps the refusal paths covered on a plain image.
+        image: pgvector/pgvector:pg16
         env:
           POSTGRES_PASSWORD: devpass
         options: >-
@@ -182,7 +186,7 @@ jobs:
     runs-on: ubuntu-latest
     services:
       postgres:
-        image: postgres:16
+        image: pgvector/pgvector:pg16
         env:
           POSTGRES_PASSWORD: devpass
         options: >-
@@ -247,14 +251,96 @@ jobs:
           AWS_DEFAULT_REGION: us-east-1
           EXTENDDB_ADMIN_USER: admin
           EXTENDDB_ADMIN_PASSWORD: ${{ steps.init.outputs.admin_password }}
-          # PostgreSQL implements no vector search, so this is the job where the
-          # wire refusal tests must actually run. Both vector suites adapt to
-          # whatever the backend reports, so the refusal suite could skip every
-          # assertion here and still report green. "0" asserts the backend
-          # refuses vector indexes, making those tests mandatory rather than
-          # optional, and turns a silent skip into a failure.
-          EXTENDDB_EXPECT_VECTORS: "0"
+          # PostgreSQL now serves vector search against a server with pgvector,
+          # which this job's image has. Both vector suites adapt to whatever the
+          # backend reports, so without a pinned expectation the positive suite
+          # could skip every assertion and still report green. "1" makes those
+          # tests mandatory. The refusal suite moves to the novector job, which is
+          # the only place those assertions still mean anything.
+          EXTENDDB_EXPECT_VECTORS: "1"
         run: devtools/run-tests --extenddb --rust-integration --release
+
+      # The control plane for vector indexes is not reachable over the wire while
+      # this backend declares no vector search capability, so its tests drive the
+      # storage layer directly against this job's PostgreSQL. They build their own
+      # throwaway databases; the connection string is the server, not a database.
+      - name: Run PostgreSQL storage-level tests
+        env:
+          EXTENDDB_TEST_PG_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432
+        run: cargo test --release -p extenddb-storage-postgres --test vector_control_plane
+
+  # The runtime-detection proof: the same binary, against a PostgreSQL with no
+  # pgvector, must refuse vector indexes over the wire. Today the job above runs
+  # on a plain image too, so this looks like a duplicate of it; it is not, because
+  # that job moves to the pgvector image when the search path lands and its
+  # EXTENDDB_EXPECT_VECTORS flips to "1". Without this job the refusal surface
+  # would stop being tested at exactly that point.
+  run-rust-integration-postgres-novector:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_PASSWORD: devpass
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          - 5432:5432
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - uses: dtolnay/rust-toolchain@stable
+
+      - uses: Swatinem/rust-cache@v2
+        with:
+          cache-on-failure: true
+
+      - name: Build release
+        run: cargo build --release
+
+      - name: Initialize ExtendDB
+        id: init
+        run: |
+          output=$(./target/release/extenddb init --config extenddb.toml \
+            --pg-host 127.0.0.1 --pg-port 5432 --pg-user postgres --pg-pass devpass 2>&1)
+          echo "$output"
+          echo "admin_password=$(echo "$output" | grep -oP 'Password: \K\S+')" >> "$GITHUB_OUTPUT"
+
+      - name: Start ExtendDB
+        run: |
+          ./target/release/extenddb serve --config extenddb.toml --foreground --write-pid-file &
+          for i in $(seq 1 30); do
+            if curl -sk https://127.0.0.1:18443/health | grep -q healthy; then
+              echo "Server ready"
+              exit 0
+            fi
+            sleep 1
+          done
+          echo "Server failed to start"
+          exit 1
+
+      - name: Run the vector refusal suite
+        env:
+          EXTENDDB_TEST_ENDPOINT: https://127.0.0.1:18443
+          AWS_DEFAULT_REGION: us-east-1
+          EXTENDDB_ADMIN_USER: admin
+          EXTENDDB_ADMIN_PASSWORD: ${{ steps.init.outputs.admin_password }}
+          # "0" makes the refusal assertions mandatory: the suite adapts to what
+          # the backend reports, so without this it could skip every assertion
+          # and still report green.
+          EXTENDDB_EXPECT_VECTORS: "0"
+        run: >-
+          devtools/run-tests --extenddb --rust-integration --release
+          --filter vector_index_unsupported
+
+      # No storage-level step here on purpose. Those tests build their own
+      # extension-free scratch databases, so they behave identically on either
+      # image and the job above already runs them; repeating them here would buy
+      # a second release build and no coverage.
   run-rust-integration-sqlite:
     runs-on: ubuntu-latest
 
@@ -323,6 +409,7 @@ jobs:
         run-integration-sqlite,
         run-integration-dev-mode,
         run-rust-integration,
+        run-rust-integration-postgres-novector,
         run-rust-integration-sqlite,
       ]
     if: always()
@@ -332,6 +419,7 @@ jobs:
              [ "${{ needs.run-integration-sqlite.result }}" != "success" ] || \
              [ "${{ needs.run-integration-dev-mode.result }}" != "success" ] || \
              [ "${{ needs.run-rust-integration.result }}" != "success" ] || \
+             [ "${{ needs.run-rust-integration-postgres-novector.result }}" != "success" ] || \
              [ "${{ needs.run-rust-integration-sqlite.result }}" != "success" ]; then
             exit 1
           fi

AGENTS.md

@@ -10,7 +10,7 @@ by AWS engineers. It is not a fork of DynamoDB and contains no DynamoDB source c
 protocol: any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged.
 
 - **Language:** Rust (edition 2024, MSRV 1.88+)
-- **Storage backends:** PostgreSQL 14+ (default), MongoDB 7.0+ (feature flag `mongodb`)
+- **Storage backends:** PostgreSQL 14+ (default), SQLite (bundled, no server or client install required; feature flag `sqlite`, also `sqlite-memory` for dev and CI), MongoDB 7.0+ (feature flag `mongodb`). One backend per binary, selected by mutually exclusive Cargo features.
 - **Architecture:** Async (tokio), trait-based storage abstraction
 - **Authentication:** Mandatory SigV4 with built-in IAM (users, groups, roles, policies)
 - **TLS:** Mandatory (self-signed cert generated by default)
@@ -32,6 +32,11 @@ extenddb/
 │   ├── engine/           # DynamoDB operation handlers (PutItem, Query, etc.)
 │   ├── storage/          # Storage trait definitions (TableEngine trait)
 │   ├── storage-postgres/ # PostgreSQL implementation of TableEngine
+│   ├── storage-sqlite/   # SQLite implementation (dev, CI, and embedded use)
+│   ├── storage-mongodb/  # MongoDB implementation (feature flag)
+│   ├── cache/            # Auth and authz caches
+│   ├── config/           # Config parsing shared by the CLI and the server
+│   ├── app/              # Shared CLI surface the thin bin dispatches to
 │   ├── auth/             # SigV4 verification, IAM policy engine
 │   ├── server/           # HTTP server, management API, web console
 │   └── bin/              # CLI, config, daemon lifecycle (extenddb binary)
@@ -52,6 +57,7 @@ extenddb/
 The `TableEngine` trait in `crates/storage/src/lib.rs` defines the storage interface. All storage backends implement this trait:
 
 - `storage-postgres` (PostgreSQL) — default backend
+- `storage-sqlite` (SQLite) — feature flag `sqlite`, and `sqlite-memory` for dev and CI
 - `storage-mongodb` (MongoDB) — feature flag `mongodb`
 
 The trait uses RPITIT (return-position impl Trait in traits) for async methods — no `#[async_trait]` macro.
@@ -68,12 +74,15 @@ extenddb (bin)
        │    └─> extenddb-storage (trait definitions)
        ├─> extenddb-auth
        ├─> extenddb-storage-postgres (feature: postgres)
+       ├─> extenddb-storage-sqlite   (feature: sqlite / sqlite-memory)
        └─> extenddb-storage-mongodb  (feature: mongodb)
 ```
 
 - **extenddb-core:** Pure synchronous Rust. No async, no I/O. Types, validation, expression parsing.
 - **extenddb-storage:** Trait definitions only. No implementation.
 - **extenddb-storage-postgres:** Concrete PostgreSQL implementation.
+- **extenddb-storage-sqlite:** Concrete SQLite implementation, the dev and CI backend.
+  It serves the same wire surface, vector search included, and needs no server.
 - **extenddb-storage-mongodb:** Concrete MongoDB implementation.
 - **extenddb-engine:** Operation handlers that call storage traits.
 - **extenddb-server:** HTTP server, management API, web console.
@@ -329,6 +338,9 @@ Query, Scan (key conditions, filters, projections, pagination, index selection)
 ### Batch & Transactions
 BatchGetItem (100 keys), BatchWriteItem (25 ops), TransactGetItems (100 items), TransactWriteItems (100 ops)
 
+### Vector Search
+SearchVectors, and vector indexes on CreateTable and UpdateTable (COSINE, EUCLIDEAN, DOT_PRODUCT; on-demand tables only; five per table). Served by the PostgreSQL and SQLite backends; MongoDB provides no implementation and refuses every vector operation. PostgreSQL additionally requires the pgvector extension on the data database, probed once at startup, and refuses without it. The refusal is two strings, which matters when grepping logs: `Vector indexes are not supported by this storage backend` for CreateTable and UpdateTable, and `SearchVectors is not supported by this storage backend` for a search.
+
 ### Streams
 ListStreams, DescribeStream, GetShardIterator, GetRecords
 
@@ -430,7 +442,7 @@ walkthroughs, and troubleshooting. It dispatches to domain-specific reference fi
     ├── postgres/         PostgreSQL readiness and installation
     ├── first-request/    AWS CLI/SDK configuration, first CRUD
     ├── samples/          sample_app.py and stream_consumer.py
-    └── troubleshooting/  Symptom-to-fix lookup (16 indexed errors)
+    └── troubleshooting/  Symptom-to-fix lookup (17 indexed errors)
 ```
 
 Activate when the user asks about installing, configuring, running, or debugging ExtendDB.

Cargo.toml

@@ -64,6 +64,10 @@ moka = { version = "0.12", features = ["future"] }
 
 # Database
 sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "tls-rustls-aws-lc-rs", "postgres", "json", "time", "uuid", "bigdecimal", "derive"] }
+# Vector column type for the PostgreSQL backend. The `sqlx` feature gives a typed
+# `Vector` that encodes and decodes pgvector's binary format, so embeddings keep
+# f32 bit-exactness instead of going through a hand-written text parser.
+pgvector = { version = "0.4", default-features = false, features = ["sqlx"] }
 mongodb = "3"
 bson = "2.13"
 dashmap = "6"

README.md

@@ -196,6 +196,9 @@ Query, Scan (key conditions, filters, projections, pagination, index selection)
 ### Batch & Transactions
 BatchGetItem (100 keys), BatchWriteItem (25 ops), TransactGetItems (100 items), TransactWriteItems (100 ops)
 
+### Vector Search
+SearchVectors, and vector indexes on CreateTable and UpdateTable (cosine, Euclidean and dot product; on-demand tables only). On PostgreSQL this requires the pgvector extension on the data database; without it every vector operation is refused and nothing else is affected.
+
 ### Streams
 ListStreams, DescribeStream, GetShardIterator, GetRecords
 

crates/app/Cargo.toml

@@ -15,7 +15,6 @@ dev-mode = ["extenddb-config/dev-mode"]
 
 [dependencies]
 extenddb-auth = { workspace = true }
-extenddb-cache = { workspace = true }
 extenddb-core = { workspace = true }
 extenddb-engine = { workspace = true }
 extenddb-storage = { workspace = true }

crates/cache/src/lib.rs

@@ -1,7 +1,7 @@
 // Copyright 2026 ExtendDB contributors
 // SPDX-License-Identifier: Apache-2.0
 
-//! Stale-while-revalidate (SWR) cache primitive used by the auth and storage layers.
+//! Stale-while-revalidate (SWR) cache primitive used by the auth and server layers.
 //!
 //! See `docs/design/12-auth-authz-cache.md` for the full design rationale.
 //!