Python: Add DuckDB vector store connector

.github/workflows/python-merge-tests.yml

@@ -39,6 +39,7 @@ jobs:
       foundryChanged: ${{ steps.filter.outputs.foundry }}
       foundryHostingChanged: ${{ steps.filter.outputs.foundry_hosting }}
       cosmosChanged: ${{ steps.filter.outputs.cosmos }}
+      duckdbChanged: ${{ steps.filter.outputs.duckdb }}
       qdrantChanged: ${{ steps.filter.outputs.qdrant }}
       mongodbChanged: ${{ steps.filter.outputs.mongodb }}
       redisChanged: ${{ steps.filter.outputs.redis }}
@@ -88,6 +89,9 @@ jobs:
               - 'python/packages/foundry_hosting/**'
             cosmos:
               - 'python/packages/azure-cosmos/**'
+            duckdb:
+              - 'python/packages/duckdb/**'
+              - '.github/workflows/python-merge-tests.yml'
             qdrant:
               - 'python/packages/qdrant/**'
               - '.github/workflows/python-merge-tests.yml'
@@ -115,6 +119,29 @@ jobs:
       - name: not python tests
         if: steps.filter.outputs.python != 'true'
         run: echo "NOT python file"
+
+  # The aggregate unit-test job is skipped for pull requests, so run DuckDB tests directly.
+  python-tests-duckdb:
+    name: Python Tests - DuckDB
+    needs: paths-filter
+    if: >
+      github.event_name == 'pull_request' &&
+      (needs.paths-filter.outputs.duckdbChanged == 'true' ||
+       needs.paths-filter.outputs.coreChanged == 'true')
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: python
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - name: Set up python and install the project
+        uses: ./.github/actions/python-setup
+        with:
+          python-version: ${{ env.UV_PYTHON }}
+          os: ${{ runner.os }}
+      - name: Test DuckDB package
+        run: uv run poe test -P duckdb
+
   python-tests-qdrant:
     name: Python Integration Tests - Qdrant
     needs: paths-filter
@@ -947,6 +974,7 @@ jobs:
     runs-on: ubuntu-latest
     needs:
       [
+        python-tests-duckdb,
         python-tests-unit,
         python-tests-openai,
         python-tests-azure-openai,

python/PACKAGE_STATUS.md

@@ -30,6 +30,7 @@ Status is grouped into these buckets:
 | `agent-framework-core` | `python/packages/core` | `released` |
 | `agent-framework-declarative` | `python/packages/declarative` | `released` |
 | `agent-framework-devui` | `python/packages/devui` | `beta` |
+| `agent-framework-duckdb` | `python/packages/duckdb` | `alpha` |
 | `agent-framework-foundry` | `python/packages/foundry` | `released` |
 | `agent-framework-foundry-hosting` | `python/packages/foundry_hosting` | `beta` |
 | `agent-framework-foundry-local` | `python/packages/foundry_local` | `beta` |

python/packages/duckdb/LICENSE

@@ -0,0 +1,21 @@
+    MIT License
+
+    Copyright (c) Microsoft Corporation.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.

python/packages/duckdb/README.md

@@ -0,0 +1,154 @@
+# Agent Framework DuckDB vector store
+
+Store and search typed records in DuckDB using the
+[Microsoft Agent Framework](https://learn.microsoft.com/agent-framework/)
+vector-store APIs. This alpha package uses the **DuckDB Python client only**:
+local files, DuckDB-supported connection URIs (including MotherDuck), and
+caller-provided `duckdb.DuckDBPyConnection` objects use the same implementation.
+
+## Installation
+
+```bash
+pip install agent-framework-duckdb --pre
+```
+
+Python 3.10+ and DuckDB 1.4.1–1.5.x are required. The package also installs
+`pytz`, which DuckDB uses when returning timezone-aware timestamps. Import
+`DuckDBStore` and `DuckDBCollection` directly from `agent_framework_duckdb`;
+this alpha package is not part of `agent-framework-core[all]`.
+
+## Connections and persistence
+
+`DuckDBStore()` (and a directly constructed `DuckDBCollection`) defaults to
+**`agent-framework.duckdb` in the current working directory**. It is a normal
+persistent database file, not an in-memory database or temporary file. Give
+each application a suitable writable location with
+`connection_string="path/to/vectors.duckdb"`; parent directories must already
+exist. The connection is opened lazily on the first async operation.
+`aclose()` or an async context manager drains pending operations and releases
+the file. Opening a *new* store against the same filename restores its tables
+and records.
+
+`DUCKDB_CONNECTION_STRING` can instead provide a filename or a URI. Settings
+precedence is **explicit `connection_string` > selected `.env` file >
+environment > persistent default**. Select a file with `env_file_path` and
+optionally `env_file_encoding`; `.env` files are never discovered implicitly.
+An explicitly empty connection string is an error. The URI is held as an AF
+`SecretString` and never printed or logged by the connector.
+
+To use [MotherDuck](https://motherduck.com/docs/getting-started/interfaces/client-apis/python/installation-authentication/),
+configure the *same* DuckDB client with a `md:` URI and its usual credentials,
+for example:
+
+```bash
+export DUCKDB_CONNECTION_STRING='md:my_db'
+export MOTHERDUCK_TOKEN='<your access token>'
+```
+
+```python
+from agent_framework_duckdb import DuckDBStore
+
+
+async def list_remote_tables() -> None:
+    async with DuckDBStore() as store:
+        print(await store.list_collection_names())
+```
+
+For an application-managed token, pass
+`DuckDBStore(connection_string="md:my_db",
+config={"motherduck_token": SecretString(token)})` (import `SecretString`
+from `agent_framework`). `config` is forwarded to `duckdb.connect`; it is
+not loaded from the connector's `.env` file. Keep tokens outside source control
+and avoid embedding them in a URI. Other DuckDB-supported services use their
+documented URI/configuration or an injected client; no service-specific
+adapter is installed. DuckDB/MotherDuck may need network access and compatible
+client/extension versions; authentication and service availability are managed
+by DuckDB, not by this connector.
+
+Alternatively, use `DuckDBStore(client=duckdb.connect(...))` or
+`DuckDBCollection(Record, client=...)` to **borrow** an existing connection.
+The caller owns and closes that connection; `client` cannot be combined with
+`connection_string`, `config`, or `.env` options. Collections obtained from a
+store borrow its client, so keep the store open while using them. All connector
+operations on one store/collection run serially on one worker thread, not on
+the event loop. Coordinate any *external* use of an injected connection
+yourself. Cancelling an awaiter does not stop a DuckDB query already running
+on that thread; `aclose()` waits for it to finish.
+
+## Example
+
+```python
+from dataclasses import dataclass
+from typing import Annotated
+
+from agent_framework import Filter, VectorStoreField, vectorstoremodel
+from agent_framework_duckdb import DuckDBStore
+
+
+@vectorstoremodel(collection_name="articles")
+@dataclass
+class Article:
+    id: Annotated[str, VectorStoreField("key")]
+    text: Annotated[str, VectorStoreField("data")]
+    embedding: Annotated[list[float] | None, VectorStoreField("vector", dimensions=3)] = None
+
+
+async def save_and_search() -> None:
+    async with DuckDBStore(connection_string="articles.duckdb") as store:
+        collection = store.get_collection(Article)
+        await collection.ensure_collection_exists()
+        await collection.upsert(
+            [Article("one", "DuckDB persists records", [1, 0, 0])],
+            generate_vectors=False,
+        )
+
+    async with DuckDBStore(connection_string="articles.duckdb") as reopened:
+        collection = reopened.get_collection(Article)
+        assert (await collection.get(["one"]))[0].text == "DuckDB persists records"
+        results = await collection.search(
+            vector=[1, 0, 0],
+            filter=Filter("text", "contains_text", "persists"),
+        )
+        async for result in results:
+            print(result["record"].text, result["score"])
+```
+
+Run a complete local example with
+`uv run --package agent-framework-duckdb python packages/duckdb/samples/duckdb_vectors.py`
+from the `python/` directory. It writes a default database file, reopens it,
+and cleans up only its sample table.
+
+## Capabilities and limits
+
+- Tables are created by `ensure_collection_exists()`; `collection_exists()`,
+  `list_collection_names()`, and `ensure_collection_deleted()` provide the
+  corresponding lifecycle. Existing tables are **not migrated or reindexed**.
+  Identifiers are quoted, values are parameterized, and collection names refer
+  to tables in the connection's current database and schema.
+- Batch upsert, retrieval, filtered/paged listing, and delete support string,
+  signed 64-bit integer, and UUID keys; string and UUID keys can be generated
+  when declared auto-generated. Records support string, integer, float,
+  boolean, UUID, bytes, date, timezone-aware datetime, JSON list/dict data,
+  and nullable dense vectors. Use `generate_vectors=False` for supplied
+  embeddings or pass a local `embedding_generator`. Retrieval excludes vectors
+  unless `include_vectors=True`.
+- Exact SQL vector search supports `DEFAULT`/`cosine_distance`,
+  `cosine_similarity`, `euclidean_distance`, `dot_prod`, and
+  `negative_dot_prod`. The default score is cosine **distance**, so lower is
+  better and `score_threshold` is a maximum; similarity/dot-product scores
+  use a minimum threshold. Null vectors are omitted. Results are filtered
+  and paged **in DuckDB**, with the primary key as a stable tie-breaker.
+- Portable filters support scalar equality/inequality, null/presence checks,
+  ordered scalar comparisons, `in`/`not_in`, text contains/prefix/suffix,
+  and AND/OR/NOT groups. They do not coerce booleans into numbers, and text
+  wildcards are literal. JSON equality/collection membership and nested
+  paths are not supported.
+- No approximate vector indexes, keyword/hybrid search, full-text or explicit
+  data indexes, binary/sparse vectors, auto-generated integer keys, or
+  server-side vectorization are provided. Unsupported options raise errors.
+  DuckDB is an in-process database with file locking: multiple writer
+  **processes** cannot concurrently write the same local file. Local records
+  are unencrypted; choose and protect the database file appropriately.
+  Connector-created batch upserts are transactional. On a borrowed connection
+  the connector does not start/commit/roll back a caller transaction; without
+  one, a failed batch may have partially persisted.

python/packages/duckdb/agent_framework_duckdb/__init__.py

@@ -0,0 +1,14 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""DuckDB vector stores for Microsoft Agent Framework."""
+
+import importlib.metadata
+
+from ._vector_store import DuckDBCollection, DuckDBSettings, DuckDBStore
+
+try:
+    __version__ = importlib.metadata.version(__name__)
+except importlib.metadata.PackageNotFoundError:
+    __version__ = "0.0.0"
+
+__all__ = ["DuckDBCollection", "DuckDBSettings", "DuckDBStore", "__version__"]