Skip to content

Python: Add DuckDB vector store connector - #8675

Merged
westey (westey-m) merged 3 commits into
microsoft:mainfrom
eavanvalkenburg:duckdb-vector-connector
Sep 24, 2026
Merged

westey (westey-m) merged 3 commits into
microsoft:mainfrom
eavanvalkenburg:duckdb-vector-connector

Conversation

@eavanvalkenburg

Copy link
Copy Markdown
Member

Motivation & Context

Agent Framework does not yet have a DuckDB vector store. DuckDB offers a durable, local on-disk option for typed records and exact vector search; the same Python client also supports MotherDuck and other DuckDB connection URIs. This adds a standalone connector without changing the existing providers.

Description & Review Guide

  • What are the major changes? Add the alpha agent-framework-duckdb workspace package with DuckDBStore/DuckDBCollection over the current vector-store abstractions. It defaults to a persistent local file; URI/configuration or a caller-owned DuckDB connection use the same implementation. It includes typed table lifecycle, transactional owned-client batch upserts, CRUD, parameterized scalar filters, exact ARRAY-based vector search, a sample/README, package registration, and focused tests.
  • What is the impact of these changes? No changes to core or core[all] dependencies. The connector serializes DuckDB work off the event loop on one worker thread per store, closes only connections it creates, and leaves caller-managed transactions alone. Search defaults to cosine distance (smaller is better), with SQL-side filtering, scoring, thresholds, and paging. MotherDuck and other service auth remains with the DuckDB client; the README documents credentials and limitations.
  • What do you want reviewers to focus on? Connection ownership and batch-transaction behavior, portable-filter/score semantics, and whether the persistent default filename and generic DuckDB URI/configuration surface feel right.

Related Issue

Part of #4169. DuckDB is an additional independent connector; that umbrella issue still has Oracle and SQL Server work, so this PR intentionally does not close it. No other open DuckDB PR or PR referencing that issue was found.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after a language prefix) — a workflow keeps the label and title prefix in sync automatically.

Copilot AI balanced review requested due to automatic review settings September 23, 2026 10:20
@agent-framework-automation agent-framework-automation Bot added documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python labels Sep 23, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Store-level collection checks mishandle identifier casing, and the new tests are not registered with pull-request CI.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds an alpha DuckDB vector-store connector supporting persistent typed records, CRUD, filtering, transactions, and exact vector search.

Changes:

  • Adds DuckDBStore and DuckDBCollection.
  • Registers dependencies, workspace metadata, and alpha status.
  • Adds tests, documentation, and sample usage.
File Summary
python/​uv.lock Locks DuckDB package dependencies.
python/​pyproject.toml Registers the workspace package; PR CI test registration remains unresolved.
python/​packages/​duckdb/​tests/​duckdb/​test_settings.py Tests configuration and connection ownership.
python/​packages/​duckdb/​tests/​duckdb/​test_collection.py Tests CRUD, filtering, search, transactions, and concurrency.
python/​packages/​duckdb/​tests/​duckdb/​conftest.py Provides shared test fixtures.
python/​packages/​duckdb/​samples/​duckdb_vectors.py Demonstrates persistent vector storage.
python/​packages/​duckdb/​README.md Documents installation, usage, and limitations.
python/​packages/​duckdb/​pyproject.toml Defines package metadata and dependencies.
python/​packages/​duckdb/​LICENSE Adds the package license.
python/​packages/​duckdb/​agent_framework_duckdb/​py.typed Marks the package as typed.
python/​packages/​duckdb/​agent_framework_duckdb/​_vector_store.py Implements the connector; case-insensitive store-level collection checks need correction.
python/​packages/​duckdb/​agent_framework_duckdb/​__init__.py Exports the public API.
python/​PACKAGE_STATUS.md Registers the package as alpha.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAF Automated Review — Iteration 1

Result: Findings reported
Scope: full PR (1 commit(s)): 78893fd709a1
Model: gpt-5.6-sol-fast

Overview

This PR adds a self-contained alpha DuckDB vector-store connector with strong type validation, parameterized SQL, serialized off-loop access, explicit connection ownership, transactional owned writes, and focused lifecycle/search tests. Two residual defects remain: connection failures can expose a secret-wrapped connection string through the chained driver exception, and store-level case-sensitive existence checks can silently skip deletion of a valid DuckDB table.

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (2 medium) across 1 file. Details are attached to the affected lines below.

Affected areas: python/packages/duckdb/agent_framework_duckdb/_vector_store.py

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAF Automated Review — Iteration 2

Result: Findings reported
Scope: 1 net-new commit(s): 84228281b257
Model: gpt-5.6-sol-fast

Overview

This incremental update sanitizes connection-time exceptions, aligns store-level collection checks with DuckDB's ASCII case behavior, adds regression tests, and wires dedicated PR CI. The exception-chain assertions, parameterized catalog lookup, and ownership-aware cleanup are strong guardrails. Residual risk remains for non-ASCII identifier semantics and for core-only PRs that bypass the new DuckDB test job.

Reviewed the supplied incremental change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (2 medium) across 2 files. Details are attached to the affected lines below.

Affected areas: .github/workflows/python-merge-tests.yml, python/packages/duckdb/agent_framework_duckdb/_vector_store.py

@westey-m
westey (westey-m) added this pull request to the merge queue Sep 24, 2026
Merged via the queue into microsoft:main with commit 7227261 Sep 24, 2026
51 checks passed

This branch was previously deployed

1 inactive deployment
github-app-auth — fea5580a Deployed Sep 24, 2026 by eavanvalkenburg via add_label #23710
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants