Amazon bought a database. We put it in the database.

[quinnypig]

Forward

This is the fourth time.

#54 put the items in Route 53. #174 put them in S3 Object Annotations. #205 put them in DynamoDB, which was the anti-joke, and which was closed, which was the joke.

Then Amazon acquired DuckDB Labs.

I have no inside information about the deal and I'm not going to invent any. I only know that a DynamoDB-compatible database whose maintainers were ex-AWS engineers now has, available to it, an embedded analytics engine owned by the company that makes DynamoDB. Declining to wire those together would have been the one irresponsible option.

This PR adds DuckDB as a storage backend. Your key-value store is now a columnar OLAP engine wearing a key-value costume. Nobody at the acquiring company asked for this. That is the traditional posture of this series.

What is actually in here

Unlike three of its four siblings, this backend works. cargo build -p extenddb --no-default-features --features duckdb, init, serve, point boto3 at it. 932 tests in the main suite and 331 in the comprehensive suite pass against a DuckDB-served instance. Zero failures.

It is a port of the SQLite backend, which is the most flattering thing I can say about the SQLite backend: it survived transplant into an engine that disagrees with SQLite about nearly everything below the SQL grammar.

The disagreements are the content of the PR.

DuckDB has no sqlx driver. The SQLite crate is 17,500 lines of sqlx::query(...).bind(...).fetch_all(&pool). So src/db.rs is a small async facade that impersonates the exact slice of sqlx the crate uses: a pool, the three query builders, and a transaction that rolls back on drop. duckdb::Connection is Send but not Sync and every call blocks, so each statement lifts the connection out of its slot, runs it on a spawn_blocking thread, and puts it back. The port then becomes, mostly, sqlx:: → db::. I would like to say this was the plan from the start. It was the plan from about twenty minutes in.

DuckDB does not cascade deletes. It will parse ON DELETE CASCADE. It will then tell you, at CREATE TABLE, that it does not do that. The catalog relied on cascades in eleven places, so the schema now has no foreign keys at all and src/referential.rs is a file that does, by hand, what a database is for.

DuckDB has no SAVEPOINT. The GSI propagation worker used one per queued row so a poison row couldn't roll back its siblings. It now runs one transaction per row. Same guarantees, more commits, a ritual repeated a hundred times per batch.

DuckDB's INTEGER is 32 bits. table_size_bytes INTEGER would have overflowed at 2 GiB, which is the kind of bug that ships. Every integer is now BIGINT. REAL is a 32-bit float, so it's DOUBLE. char() is chr(), which took begins_with down until it wasn't. SQLite's scalar MIN(a, b) is an aggregate in DuckDB and is not allowed in an UPDATE; the metrics upsert now says LEAST. rowid starts at 0. Every one of these was found by a failing test, which is the argument for having tests.

DuckDB will not index an expression that calls an extension function. The TTL sweep wanted an index on json_extract_string(item_data, '$.ttl.N'). Left alone, the engine retried that index forever and never marked the table TTL-ready, and nothing ever expired. Now it doesn't try. TTL is a filtered scan. Analytics engine; it can scan.

Why this is not as deranged as it sounds

• The file on disk is a real DuckDB database. duckdb extenddb.duckdb then SELECT COUNT(*) FROM "_ddb_..." GROUP BY ... and you are doing analytics against your DynamoDB-compatible table with no export step, no Iceberg, no ETL, and no invoice. This is the single most useful property of any backend in this series and I resent that.
• In-memory mode is better than SQLite's. A DuckDB in-memory database is shared by every connection cloned from it, so :memory: gets a real read pool where SQLite had to pin one connection.
• MVCC. Readers never block on the writer. SQLite's BEGIN IMMEDIATE dance is gone.
• It is a second embedded backend, and it found real semantics the storage seam had been letting SQLite define by default. That's what a second implementation is for.

These are defensible reasons to merge this PR, which is as usual damaging to the artistic intent.

Why it is exactly as deranged as it sounds

• DuckDB is a vectorized columnar engine tuned for scanning billions of rows. Every PutItem is a one-row INSERT ... ON CONFLICT DO UPDATE into that engine. It is a combine harvester being used to trim a bonsai.
• One process at a time. DuckDB takes a process-exclusive lock on the file. While the server is running, extenddb settings set from another shell cannot open the database. Not "returns stale data." Cannot open it. The CLI and the server now have to take turns, which is a concurrency model I last encountered in a household with one bathroom.
• The catalog has no foreign keys because the database refused to enforce them, so the Rust code enforces them, so the database has been demoted to a place where the Rust code keeps its files.
• A cold build compiles DuckDB from source. Budget several minutes. It is cached afterward, in the same sense that the acquisition is cached.
• This backend's only dependency is now owned by the company whose product it exists to impersonate. The supply chain is a loop.

Known limits

Documented in crates/storage-duckdb/README.md and docs/design/14-storage-duckdb.md: the process lock (run settings/catalog-check/verify with the server stopped), no TTL expression index, cold build time. The test_gsi_async module, which drives the server out-of-band through settings set, is excluded from the DuckDB CI job for the lock reason; the paths it exercises have crate-level unit tests.