.NET: Postgres Memory

.vscode/settings.json

@@ -1,3 +1,3 @@
 {
-	"dotnet.defaultSolution": "dotnet/agent-framework-src-and-tests.slnf"
+    "dotnet.defaultSolution": "dotnet/agent-framework-src-and-tests.slnf"
 }

docs/decisions/0041-postgres-memory-provider.md

@@ -0,0 +1,52 @@
+---
+status: superseded by [ADR-0042](0042-postgres-context-memory.md)
+contact: jaredmeade
+date: 2026-09-14
+deciders: jaredmeade
+---
+
+# PostgreSQL-backed memory context provider
+
+> **Superseded by [ADR-0042](0042-postgres-context-memory.md).** The final design retains the
+> PostgreSQL package and native .NET implementation but replaces the provider-owned, message-level
+> pipeline with a reusable `PostgresMemoryClient` and thin `PostgresMemoryContextProvider`.
+
+## Context and Problem Statement
+
+Many customers run PostgreSQL as their primary database and want durable, cross-session agent
+memory backed by it, using pgvector for semantic recall and a full-text ranking mechanism for
+lexical recall, without introducing another datastore. How should a PostgreSQL memory provider be
+designed and shipped, and should it be available for both .NET and Python?
+
+## Decision Drivers
+
+- Integrate through Agent Framework's existing context-provider extension points.
+- Avoid requiring a separate always-on service; the provider should work with a plain PostgreSQL database plus optional extensions.
+- Support both .NET and Python with parity, since both are first-class Agent Framework languages.
+- Ship an initial, reviewable increment rather than the full memory pipeline (raw turns → facts → summaries → profile) in one PR.
+
+## Considered Options
+
+- Native per-language packages (`Microsoft.Agents.AI.Postgres` for .NET, with a matching Python package), each talking to PostgreSQL directly.
+- A single shared backend service (one implementation, both SDKs as thin HTTP/gRPC clients) — avoids duplicating retrieval/extraction logic across languages at the cost of an extra deployable component.
+- Reuse the existing `agent-framework-postgres` (Python) pgvector connector directly as the context provider, without a dedicated memory-shaped package.
+
+## Decision Outcome
+
+Chosen option: "Native per-language packages", because it keeps the integration in-process without
+an additional service to deploy. Behavioral parity between the two languages is enforced through
+documentation and shared expectations in this ADR rather than shared code.
+
+### v1 scope
+
+- Store conversation turns and derived memories with embeddings.
+- Retrieve relevant memories via hybrid search: pgvector cosine similarity fused with PostgreSQL full-text ranking (native `tsvector`/`ts_rank`; BM25 extensions such as `pg_textsearch`/ParadeDB `pg_search` are a documented future upgrade, not a v1 requirement).
+- Scope memories by application, agent, thread, and user identifiers.
+
+### Deferred to a later iteration
+
+- The reusable client and context-provider separation described in
+  [0042-postgres-context-memory.md](0042-postgres-context-memory.md), which supersedes the initial
+  provider-owned design.
+- BM25 ranking via `pg_textsearch`/ParadeDB `pg_search` (native `tsvector`/`ts_rank` is the v1 fallback and default).
+- The Python package (`agent-framework-postgres-memory` or similar); this ADR covers the .NET package (`Microsoft.Agents.AI.Postgres`) first.

docs/decisions/0042-postgres-context-memory.md

@@ -0,0 +1,102 @@
+---
+status: proposed
+contact: jaredmeade
+date: 2026-09-14
+deciders: jaredmeade
+---
+
+# PostgreSQL ContextMemory architecture
+
+## Context and Problem Statement
+
+[0041-postgres-memory-provider.md](0041-postgres-memory-provider.md) proposed a PostgreSQL-backed
+Agent Framework provider, but its first implementation placed persistence, retrieval, extraction,
+summarization, and profile maintenance inside the provider itself. This coupled the reusable memory
+engine to one Agent Framework lifecycle adapter. The PostgreSQL integration should separate those
+responsibilities so the memory engine is useful independently and the .NET and Python adapters share
+the same behavioral contract.
+
+## Decision Drivers
+
+- Preserve the distinction between an Agent Framework context adapter and a reusable memory engine.
+- Support turns, facts, procedural memories, episodic memories, thread summaries, and cross-thread
+  user summaries.
+- Support confidence filtering, hybrid retrieval, memory reconciliation, and background processing,
+  not only fact extraction.
+- Use provider-agnostic chat and embedding abstractions in both language implementations.
+- Keep PostgreSQL as the only durable dependency; pgvector supplies semantic retrieval and native
+  PostgreSQL full-text search supplies lexical retrieval.
+
+## Decision Outcome
+
+The package contains two public layers:
+
+1. `PostgresMemoryClient` is the reusable PostgreSQL ContextMemory engine. It owns storage,
+   retrieval, transformation, cadence-aware background processing, explicit processing, and
+   background-work flushing.
+2. `PostgresMemoryContextProvider` is a thin Agent Framework lifecycle adapter. Before a run it
+   searches the client and retrieves the user summary; after a run it writes conversation turns.
+   It does not implement extraction or summarization itself.
+
+The .NET implementation is distributed in `Microsoft.Agents.AI.Postgres`. The Python implementation
+is part of `agent-framework-postgres` and uses Psycopg 3.
+
+### Memory model
+
+The configured base table name produces four tables:
+
+- `{base}_turns`: append-only raw conversation turns.
+- `{base}_memories`: active and superseded fact, procedural, and episodic memories, with confidence,
+  salience, tags, embeddings, full-text indexes, expiry, and reconciliation metadata.
+- `{base}_summaries`: versioned thread summaries and cross-thread user summaries.
+- `{base}_processing`: per-scope processing checkpoints and extraction cadence state.
+
+### Lifecycle
+
+- The provider's pre-run hook performs hybrid vector/full-text search over typed derived memories
+  and independently retrieves the user summary. Both are injected as user-role, explicitly
+  untrusted reference information.
+- The provider's post-run hook writes supported input and output messages as turns. A turn write may
+  schedule processing, but the provider does not wait for extraction.
+- Both providers support configurable search/request/response message filters and independent
+  storage and search scopes.
+- `PostgresMemoryClient` extracts typed memories, incrementally updates thread/user summaries,
+  applies episodic TTL, folds exact/vector-similar duplicates, and reconciles semantic
+  contradictions while retaining supersession history.
+- `FlushAsync`/`flush` drains in-flight processing before shutdown. `ProcessNowAsync`/`process_now`
+  is available through the provider and client layers, giving tests, operators, and applications
+  with automatic cadence disabled an explicit processing hook.
+
+### Scope
+
+`PostgresMemoryScope.UserId` is required for durable operations and follows a user across sessions.
+`ThreadId` is required when storing or processing turns. The provider defaults its search scope to a
+copy without `ThreadId`, enabling cross-thread recall for the same user. Optional application and
+agent identifiers provide additional isolation and are matched exactly; null identifiers do not
+act as wildcards across other applications or agents.
+
+### Retrieval
+
+Search defaults to fact, procedural, and episodic memories. It filters by minimum confidence and
+combines pgvector cosine ranking with native `tsvector`/`ts_rank` using Reciprocal Rank Fusion.
+Superseded and expired records are excluded. The initial implementation uses native PostgreSQL
+full-text ranking; optional BM25 extensions remain a future enhancement.
+
+## Consequences
+
+- The memory engine is independently usable and testable instead of being coupled to Agent
+  Framework lifecycle types.
+- The provider API is named `PostgresMemoryContextProvider` to make its lifecycle role explicit.
+- Model clients and Npgsql/Psycopg data sources are supplied and owned by the caller. A convenience
+  provider constructor owns only the `PostgresMemoryClient` it creates, not those dependencies.
+- Summary versions are serialized with PostgreSQL advisory transaction locks, and processing
+  checkpoints are monotonic so concurrent workers cannot move completed work backward. Stale or
+  equal summary coverage is rejected while holding the same lock.
+- Extraction and reconciliation require a complete, schema-valid JSON array response. One malformed
+  item invalidates the full response, and processing checkpoints do not advance.
+- Direct and scheduled thread processing share per-scope locks, and user-summary turn cutoffs are
+  captured before model execution so concurrent turn writes are not checkpointed prematurely.
+- In-process background processing is appropriate for development and low-throughput deployments.
+  A durable PostgreSQL processor (for example, an outbox plus worker) is deferred.
+- PostgreSQL `vector` HNSW indexing supports up to 2,000 dimensions; callers using larger embedding
+  outputs must request reduced dimensions or a future half-precision/index strategy.

docs/decisions/0043-postgres-workflow-checkpoints.md

@@ -0,0 +1,100 @@
+---
+status: proposed
+contact: jaredmeade
+date: 2026-09-17
+deciders: jaredmeade
+---
+
+# PostgreSQL workflow checkpoints with a shared storage schema
+
+## Context and Problem Statement
+
+The PostgreSQL integration already persists agent memory. Workflows also need exact
+execution snapshots to survive a process restart or wait for human approval. These are
+not derived memories and must not pass through extraction, summarization, or vector search.
+Python and .NET should share a storage contract without implying that their different
+execution-state formats can resume across runtimes.
+
+## Decision Drivers
+
+- Reuse the existing checkpoint extension points without modifying the workflow engine.
+- Support plain PostgreSQL, including deployments without pgvector or a model client.
+- Share one relational schema and default table across both SDKs.
+- Preserve scoped access, immutable snapshots, parent history, and committed save order.
+
+## Considered Options
+
+- **Separate language-specific tables:** simple payload separation, but duplicates database
+  administration and creates unnecessary schema divergence.
+- **Shared table with an explicit payload format:** common storage and operations, with
+  independent SDK codecs. Does not provide cross-language execution recovery.
+- **Universal checkpoint codec:** could enable portable recovery, but requires a framework-wide
+  contract for graphs, executor identities, messages, types, and state restoration. A storage
+  adapter cannot supply that guarantee by converting JSON field names.
+
+## Decision Outcome
+
+Implement `PostgresCheckpointStorage` in Python and `PostgresCheckpointStore` in .NET.
+Both default to the same `agent_framework_checkpoints` table, in an existing configured
+schema, with this common envelope:
+
+| Column | Meaning |
+| --- | --- |
+| `application_id`, `tenant_id` | Required trusted application and authorization scope |
+| `run_id` | Required execution identity; .NET's workflow session ID |
+| `payload_format` | SDK-specific codec identifier |
+| `checkpoint_id` | Identity within the scope and format |
+| `workflow_name` | Optional definition metadata; used by Python filtering |
+| `parent_checkpoint_id` | Previous checkpoint identity, when present |
+| `version` | Storage-envelope version, initially integer 1 |
+| `sequence` | Database-generated identity used for save ordering |
+| `created_at` | Database timestamp for operational inspection |
+| `payload` | Framework-serialized JSONB snapshot |
+
+The composite primary key is `(application_id, tenant_id, run_id, payload_format, checkpoint_id)`.
+A common B-tree index covers `(application_id, tenant_id, run_id, payload_format, sequence)`.
+Each adapter filters every retrieval and deletion by its complete scope and format:
+
+- Python: `agent-framework.python.checkpoint.v1`.
+- .NET: `agent-framework.dotnet.checkpoint.v1`.
+
+The discriminator versions the codec contract, not the installed package. Python retains
+the checkpoint's own format version inside its payload. The .NET provider treats the
+framework's `JsonElement` as opaque. A format change requires a new discriminator or an
+explicit compatibility strategy; changing the envelope requires a schema migration.
+
+Both providers serialize saves within a run and format using transaction-scoped PostgreSQL
+advisory locks. The database identity is assigned after obtaining the lock, so commit order
+does not depend on worker clocks or iteration counts. Python re-saving an identical
+checkpoint ID is idempotent and does not change history order; conflicting content is
+rejected. The .NET contract generates a new checkpoint ID for each creation.
+
+## Consequences
+
+- The same physical table can contain both runtimes' checkpoints, even for identical
+  application, tenant, run, and checkpoint IDs. Neither runtime decodes the other's rows.
+- SDK-specific payloads are **not** cross-runtime resumable. Python uses the framework's
+  restricted pickle/JSON codec; .NET uses its JSON checkpoint marshaller.
+- Scope values must be authorized by the host. They are query isolation, not a replacement
+  for database access controls. Checkpoints must remain trusted, private, untampered state.
+- Callers own injected connections and pools. A borrowed Python connection's outer
+  transaction must commit before its checkpoint is durable; use pools for parallel writes.
+- Table initialization creates no schema or extension and does not migrate incompatible
+  existing tables. Provision the table before concurrent workers start. Retention and
+  cleanup remain application responsibilities.
+- Checkpoint persistence does not add scheduling, worker leases, distributed workflow
+  ownership, or exactly-once external side effects. Hosts must avoid concurrent execution
+  of the same run and use idempotency or an outbox for repeatable side effects.
+
+## Verification
+
+Focused tests cover round trips, scope and payload-format isolation, history ordering,
+duplicate-save behavior, parent lookup, rollback, and concurrent writes. Model-free approval
+samples demonstrate preserving a human wait across separate processes. Validation uses
+an isolated PostgreSQL database without the vector extension.
+
+## References
+
+- [PostgreSQL ContextMemory architecture](0042-postgres-context-memory.md)
+- [Workflow checkpoint documentation](https://learn.microsoft.com/agent-framework/workflows/checkpoints)
+- [Durable execution extension](https://learn.microsoft.com/agent-framework/hosting/azure-functions)

docs/specs/feature-usage-bit-registry.md

@@ -248,7 +248,8 @@ only to approved first-party endpoints.
 | 72 | `local_codeact` | Local Python CodeAct provider | `Microsoft.Agents.AI.LocalCodeAct.LocalCodeActProvider` |
 | 73 | `hosting.a2a` | A2A hosting endpoints | `Microsoft.AspNetCore.Builder.A2AEndpointRouteBuilderExtensions.MapA2AJsonRpc` |
 | 74 | `hosting.openai` | OpenAI-compatible hosting endpoints | `Microsoft.AspNetCore.Builder.MicrosoftAgentAIHostingOpenAIEndpointRouteBuilderExtensions.MapOpenAIResponses` |
-| 75–127 | _reserved_ | future packages | — |
+| 75 | `postgres.memory` | PostgreSQL ContextMemory client/provider | `Microsoft.Agents.AI.Postgres.PostgresMemoryClient`; `agent_framework_postgres.PostgresMemoryClient` |
+| 76–127 | _reserved_ | future packages | — |
 
 ## Opt-out
 

dotnet/Directory.Packages.props

@@ -140,6 +140,9 @@
     <PackageVersion Include="Microsoft.PowerFx.Interpreter" Version="1.8.1" />
     <!-- Valkey -->
     <PackageVersion Include="Valkey.Glide" Version="1.1.0" />
+    <!-- PostgreSQL -->
+    <PackageVersion Include="Npgsql" Version="10.0.3" />
+    <PackageVersion Include="Pgvector" Version="0.3.2" />
     <!-- Console UX -->
     <PackageVersion Include="Spectre.Console" Version="0.49.1" />
     <!-- AWS -->

dotnet/agent-framework-dotnet.slnx

@@ -193,6 +193,7 @@
     <Project Path="samples/02-agents/AgentWithMemory/AgentWithMemory_Step06_MemoryUsingAgentMemory/AgentWithMemory_Step06_MemoryUsingAgentMemory.csproj" />
     <Project Path="samples/02-agents/AgentWithMemory/AgentWithMemory_Step07_FileMemoryProvider/AgentWithMemory_Step07_FileMemoryProvider.csproj" />
     <Project Path="samples/02-agents/AgentWithMemory/AgentWithMemory_Step08_MemoryUsingCosmosNoSql/AgentWithMemory_Step08_MemoryUsingCosmosNoSql.csproj" />
+    <Project Path="samples/02-agents/AgentWithMemory/AgentWithMemory_Step10_MemoryUsingPostgres/AgentWithMemory_Step10_MemoryUsingPostgres.csproj" />
   </Folder>
   <Folder Name="/Samples/02-agents/AgentProviders/openai/">
     <File Path="samples/02-agents/AgentProviders/openai/README.md" />
@@ -278,6 +279,7 @@
     <Project Path="samples/03-workflows/Checkpoint/CheckpointAndRehydrate/CheckpointAndRehydrate.csproj" />
     <Project Path="samples/03-workflows/Checkpoint/CheckpointAndResume/CheckpointAndResume.csproj" />
     <Project Path="samples/03-workflows/Checkpoint/CheckpointWithHumanInTheLoop/CheckpointWithHumanInTheLoop.csproj" />
+    <Project Path="samples/03-workflows/Checkpoint/PostgresCheckpointing/PostgresCheckpointing.csproj" />
   </Folder>
   <Folder Name="/Samples/03-workflows/HumanInTheLoop/">
     <Project Path="samples/03-workflows/HumanInTheLoop/HumanInTheLoopBasic/HumanInTheLoopBasic.csproj" />
@@ -630,6 +632,7 @@
     <Project Path="src/Microsoft.Agents.AI.AgentHooks/Microsoft.Agents.AI.AgentHooks.csproj" />
     <Project Path="src/Microsoft.Agents.AI.Mem0/Microsoft.Agents.AI.Mem0.csproj" />
     <Project Path="src/Microsoft.Agents.AI.OpenAI/Microsoft.Agents.AI.OpenAI.csproj" />
+    <Project Path="src/Microsoft.Agents.AI.Postgres/Microsoft.Agents.AI.Postgres.csproj" />
     <Project Path="src/Microsoft.Agents.AI.Purview/Microsoft.Agents.AI.Purview.csproj" />
     <Project Path="src/Microsoft.Agents.AI.Tools.Shell/Microsoft.Agents.AI.Tools.Shell.csproj" />
     <Project Path="src/Microsoft.Agents.AI.Valkey/Microsoft.Agents.AI.Valkey.csproj" />
@@ -642,6 +645,7 @@
   </Folder>
   <Folder Name="/Tests/">
     <Project Path="tests/Microsoft.Agents.AI.Hosting.OpenAI.IntegrationTests/Microsoft.Agents.AI.Hosting.OpenAI.IntegrationTests.csproj" />
+    <Project Path="tests/Microsoft.Agents.AI.Postgres.UnitTests/Microsoft.Agents.AI.Postgres.UnitTests.csproj" />
   </Folder>
   <Folder Name="/Tests/IntegrationTests/">
     <Project Path="tests/AgentConformance.IntegrationTests/AgentConformance.IntegrationTests.csproj" />

dotnet/samples/02-agents/AgentWithMemory/AgentWithMemory_Step10_MemoryUsingPostgres/AgentWithMemory_Step10_MemoryUsingPostgres.csproj

@@ -0,0 +1,20 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+    <TargetFrameworks>net10.0</TargetFrameworks>
+
+    <Nullable>enable</Nullable>
+    <ImplicitUsings>enable</ImplicitUsings>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Azure.Identity" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Foundry\Microsoft.Agents.AI.Foundry.csproj" />
+    <ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Postgres\Microsoft.Agents.AI.Postgres.csproj" />
+  </ItemGroup>
+
+</Project>