Repository navigation
Python: Add DuckDB vector store connector - #8675
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Store-level collection checks mishandle identifier casing, and the new tests are not registered with pull-request CI.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
Adds an alpha DuckDB vector-store connector supporting persistent typed records, CRUD, filtering, transactions, and exact vector search.
Changes:
- Adds
DuckDBStoreandDuckDBCollection. - Registers dependencies, workspace metadata, and alpha status.
- Adds tests, documentation, and sample usage.
| File | Summary |
|---|---|
python/uv.lock |
Locks DuckDB package dependencies. |
python/pyproject.toml |
Registers the workspace package; PR CI test registration remains unresolved. |
python/packages/duckdb/tests/duckdb/test_settings.py |
Tests configuration and connection ownership. |
python/packages/duckdb/tests/duckdb/test_collection.py |
Tests CRUD, filtering, search, transactions, and concurrency. |
python/packages/duckdb/tests/duckdb/conftest.py |
Provides shared test fixtures. |
python/packages/duckdb/samples/duckdb_vectors.py |
Demonstrates persistent vector storage. |
python/packages/duckdb/README.md |
Documents installation, usage, and limitations. |
python/packages/duckdb/pyproject.toml |
Defines package metadata and dependencies. |
python/packages/duckdb/LICENSE |
Adds the package license. |
python/packages/duckdb/agent_framework_duckdb/py.typed |
Marks the package as typed. |
python/packages/duckdb/agent_framework_duckdb/_vector_store.py |
Implements the connector; case-insensitive store-level collection checks need correction. |
python/packages/duckdb/agent_framework_duckdb/__init__.py |
Exports the public API. |
python/PACKAGE_STATUS.md |
Registers the package as alpha. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
MAF Automated Review — Iteration 1
Result: Findings reported
Scope: full PR (1 commit(s)): 78893fd709a1
Model: gpt-5.6-sol-fast
Overview
This PR adds a self-contained alpha DuckDB vector-store connector with strong type validation, parameterized SQL, serialized off-loop access, explicit connection ownership, transactional owned writes, and focused lifecycle/search tests. Two residual defects remain: connection failures can expose a secret-wrapped connection string through the chained driver exception, and store-level case-sensitive existence checks can silently skip deletion of a valid DuckDB table.
Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (2 medium) across 1 file. Details are attached to the affected lines below.
Affected areas: python/packages/duckdb/agent_framework_duckdb/_vector_store.py
There was a problem hiding this comment.
MAF Automated Review — Iteration 2
Result: Findings reported
Scope: 1 net-new commit(s): 84228281b257
Model: gpt-5.6-sol-fast
Overview
This incremental update sanitizes connection-time exceptions, aligns store-level collection checks with DuckDB's ASCII case behavior, adds regression tests, and wires dedicated PR CI. The exception-chain assertions, parameterized catalog lookup, and ownership-aware cleanup are strong guardrails. Residual risk remains for non-ASCII identifier semantics and for core-only PRs that bypass the new DuckDB test job.
Reviewed the supplied incremental change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (2 medium) across 2 files. Details are attached to the affected lines below.
Affected areas: .github/workflows/python-merge-tests.yml, python/packages/duckdb/agent_framework_duckdb/_vector_store.py

Motivation & Context
Agent Framework does not yet have a DuckDB vector store. DuckDB offers a durable, local on-disk option for typed records and exact vector search; the same Python client also supports MotherDuck and other DuckDB connection URIs. This adds a standalone connector without changing the existing providers.
Description & Review Guide
agent-framework-duckdbworkspace package withDuckDBStore/DuckDBCollectionover the current vector-store abstractions. It defaults to a persistent local file; URI/configuration or a caller-owned DuckDB connection use the same implementation. It includes typed table lifecycle, transactional owned-client batch upserts, CRUD, parameterized scalar filters, exact ARRAY-based vector search, a sample/README, package registration, and focused tests.core[all]dependencies. The connector serializes DuckDB work off the event loop on one worker thread per store, closes only connections it creates, and leaves caller-managed transactions alone. Search defaults to cosine distance (smaller is better), with SQL-side filtering, scoring, thresholds, and paging. MotherDuck and other service auth remains with the DuckDB client; the README documents credentials and limitations.Related Issue
Part of #4169. DuckDB is an additional independent connector; that umbrella issue still has Oracle and SQL Server work, so this PR intentionally does not close it. No other open DuckDB PR or PR referencing that issue was found.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after a language prefix) — a workflow keeps the label and title prefix in sync automatically.