Repository navigation
.NET: Postgres Memory - #8664
Jared (JaredMSFT) wants to merge 7 commits into
Conversation
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Failure
agent-framework-automation
Bot
added
documentation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Critical correctness issues remain in concurrent turn processing and reranking fallback, alongside unresolved validation, cancellation, isolation, and telemetry defects.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Open (8)
What changed in this PR
Adds opt-in PostgreSQL-backed durable memory and workflow checkpoint storage for Python and .NET.
Changes:
- Adds typed memory extraction, summaries, hybrid retrieval, reconciliation, and optional reranking.
- Adds scoped checkpoint persistence and cross-process recovery samples.
- Adds tests, documentation, package wiring, and architectural decisions.
| File | Description |
|---|---|
python/uv.lock |
Updates locked dependencies. |
python/packages/postgres/tests/postgres/test_checkpoint_storage.py |
Tests checkpoint storage. |
python/packages/postgres/tests/postgres_memory/test_memory_integration.py |
Tests live memory integration. |
python/packages/postgres/tests/postgres_memory/test_memory_context_provider.py |
Tests context-provider behavior. |
python/packages/postgres/tests/conftest.py |
Configures shared tests. |
python/packages/postgres/samples/postgres_memory.py |
Demonstrates durable memory. |
python/packages/postgres/samples/postgres_checkpointing.py |
Demonstrates checkpoint recovery. |
python/packages/postgres/README.md |
Documents Python integration. |
python/packages/postgres/pyproject.toml |
Updates package and test configuration. |
python/packages/postgres/agent_framework_postgres/_memory_types.py |
Defines memory models and options. |
python/packages/postgres/agent_framework_postgres/_memory_context_provider.py |
Integrates memory with agent runs. |
python/packages/postgres/agent_framework_postgres/_feature_usage.py |
Defines feature telemetry. |
python/packages/postgres/agent_framework_postgres/_checkpoint_storage.py |
Implements checkpoint persistence. |
python/packages/postgres/agent_framework_postgres/__init__.py |
Exports PostgreSQL APIs. |
python/AGENTS.md |
Registers PostgreSQL documentation. |
dotnet/tests/Microsoft.Agents.AI.Postgres.UnitTests/PostgresMemoryStoreTests.cs |
Tests memory storage. |
dotnet/tests/Microsoft.Agents.AI.Postgres.UnitTests/PostgresMemoryStoreIntegrationTests.cs |
Tests live memory storage. |
dotnet/tests/Microsoft.Agents.AI.Postgres.UnitTests/PostgresMemoryContextProviderTests.cs |
Tests context-provider behavior. |
dotnet/tests/Microsoft.Agents.AI.Postgres.UnitTests/PostgresCheckpointStoreTests.cs |
Tests checkpoint storage. |
dotnet/tests/Microsoft.Agents.AI.Postgres.UnitTests/PostgresCheckpointStoreIntegrationTests.cs |
Tests live checkpoint storage. |
dotnet/tests/Microsoft.Agents.AI.Postgres.UnitTests/Microsoft.Agents.AI.Postgres.UnitTests.csproj |
Defines the test project. |
dotnet/tests/Microsoft.Agents.AI.Postgres.UnitTests/AzureRankLiveIntegrationTests.cs |
Tests live Azure reranking. |
dotnet/src/Microsoft.Agents.AI.Postgres/README.md |
Documents the .NET integration. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryType.cs |
Defines memory categories. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryScope.cs |
Defines isolation scopes. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryRecord.cs |
Defines stored memory records. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryPrompts.cs |
Builds processing prompts. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryPromptOptions.cs |
Configures prompts. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryJsonUtilities.cs |
Configures JSON serialization. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryContextProviderOptions.cs |
Defines provider options. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryContextProvider.cs |
Integrates memory with agent invocations. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryClientOptions.cs |
Defines memory-client options. |
dotnet/src/Microsoft.Agents.AI.Postgres/PostgresCheckpointStore.cs |
Implements checkpoint persistence. |
dotnet/src/Microsoft.Agents.AI.Postgres/Microsoft.Agents.AI.Postgres.csproj |
Defines the integration package. |
dotnet/src/Microsoft.Agents.AI.Postgres/IPostgresMemoryStore.cs |
Defines the storage contract. |
dotnet/src/Microsoft.Agents.AI.Postgres/IPostgresMemoryClient.cs |
Defines the memory-client contract. |
dotnet/src/Microsoft.Agents.AI.Postgres/FeatureIndex.cs |
Defines feature telemetry. |
dotnet/samples/03-workflows/README.md |
Lists the checkpoint sample. |
dotnet/samples/03-workflows/Checkpoint/PostgresCheckpointing/README.md |
Documents checkpoint recovery. |
dotnet/samples/03-workflows/Checkpoint/PostgresCheckpointing/Program.cs |
Implements the checkpoint sample. |
dotnet/samples/03-workflows/Checkpoint/PostgresCheckpointing/PostgresCheckpointing.csproj |
Defines the checkpoint sample project. |
dotnet/samples/02-agents/AgentWithMemory/README.md |
Lists the PostgreSQL memory sample. |
dotnet/samples/02-agents/AgentWithMemory/AgentWithMemory_Step10_MemoryUsingPostgres/README.md |
Documents the memory sample. |
dotnet/samples/02-agents/AgentWithMemory/AgentWithMemory_Step10_MemoryUsingPostgres/Program.cs |
Implements the memory sample. |
dotnet/samples/02-agents/AgentWithMemory/AgentWithMemory_Step10_MemoryUsingPostgres/AgentWithMemory_Step10_MemoryUsingPostgres.csproj |
Defines the memory sample project. |
dotnet/Directory.Packages.props |
Adds PostgreSQL dependency versions. |
dotnet/agent-framework-dotnet.slnx |
Registers projects and samples. |
docs/specs/feature-usage-bit-registry.md |
Registers feature telemetry. |
docs/decisions/0043-postgres-workflow-checkpoints.md |
Documents checkpoint architecture. |
docs/decisions/0042-postgres-context-memory.md |
Documents context-memory architecture. |
docs/decisions/0041-postgres-memory-provider.md |
Records the earlier provider design. |
.vscode/settings.json |
Updates editor settings. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Failure
- serialize turn ID allocation with transaction-scoped advisory locks - reject non-finite confidence and deduplication thresholds - mark retrieved memories as untrusted context to prevent instruction injection - limit Azure AI reranking fallback to integration failures - align azure_ai.rank queries with the deployed id, rank, and score schema - add regression coverage for concurrency, validation, and reranker projection
…t-framework into postgres-squashed
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Failure
| /// Before a run, the provider searches typed long-term memories and independently retrieves the | ||
| /// cross-thread user summary. Both are injected as untrusted user-role context. After a run, the | ||
| /// provider stores new turns; the client owns background extraction, summaries, reconciliation, | ||
| /// and PostgreSQL persistence. |
There was a problem hiding this comment.
Jared (@JaredMSFT), we actually already have what is essentially the same solution.
We have a postgres implementation for the core .net vector store abstractions.
At the same time, we also have a ChatHistoryMemoryProvider, which just like this provider stores / searches chat history as memories. This can work with any Microsoft.Extensions.VectorData (MEVD) implementation, including the postgres one. Check out this sample: https://github.com/microsoft/agent-framework/blob/main/dotnet/samples/02-agents/AgentWithMemory/AgentWithMemory_Step01_ChatHistoryMemory/Program.cs, where InMemoryVectorStore is just replaced with PostgresVectorStore.
The checkpoint store may still be valuable though.
|
I would also prefer to have this in separate PR's for dotnet and python and as separate PR's for the different components that are proposed, and as westey (@westey-m) mentions, we already have several pieces that can work with Postgres, would be good to first further align on that. Jared (@JaredMSFT) |
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
MAF Automated Review — Iteration 1
Result: Findings reported
Scope: full PR (7 commit(s)): 3d2d4feecfdf, c1c082726b7c, 6eee79a4cf51, d86facae2dea, 30ed07dbb8b0, b5002e4491cc, d03fb2dc4951
Model: gpt-5.6-sol
Overview
The PR adds opt-in PostgreSQL-backed memory and checkpoint providers with scoped SQL predicates, parameterized values, defensive identifier quoting, and substantial unit and integration coverage. The latest hardening commit correctly adds transaction-scoped advisory locking to the .NET turn path and fixes several earlier validation, reranking, prompt-framing, and ADR issues. Three residual defects remain: per-scope processing locks are retained for the client lifetime in both SDKs, Python still lacks the turn-order serialization added to .NET, and Python assigns the new feature to an already-allocated telemetry bit.
Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
3 verified findings remained after source verification (2 high, 1 medium) across 3 files. Details are attached to the affected lines below.
Affected areas: dotnet/src/Microsoft.Agents.AI.Postgres/PostgresMemoryClient.cs, python/packages/postgres/agent_framework_postgres/_feature_usage.py, python/packages/postgres/agent_framework_postgres/_memory_store.py
| private SemaphoreSlim GetProcessingLock(PostgresMemoryScope scope, bool includeThread) | ||
| { | ||
| var key = PostgresMemoryStore.ComputeScopeKey(scope, includeThread); | ||
| return this._processingLocks.GetOrAdd(key, static _ => new SemaphoreSlim(1, 1)); |
There was a problem hiding this comment.
With AutoProcess enabled by default, every distinct user/thread scope reaches this GetOrAdd, but entries are removed only when the entire client is disposed. The Python mirror at _memory_client.py:614 likewise never pops its entries. In a long-lived shared host, a caller that repeatedly creates new sessions can therefore grow a process-wide lock map without bound until memory pressure or OOM affects unrelated users. Please use a keyed-lock implementation that safely evicts an entry after its final holder/waiter, or otherwise bound and reclaim this state in both implementations.
| content, | ||
| PgVector(embedding), | ||
| ] | ||
| async with self._client.connection(vectors=embedding is not None) as connection: |
There was a problem hiding this comment.
This insert allocates its BIGSERIAL ID without the per-scope transaction/advisory lock added to the .NET implementation. Two concurrent inserts can allocate IDs in one order and commit in the reverse order; extraction may then advance fact_through_turn_id to the higher committed ID, causing the delayed lower-ID row to be permanently excluded by id > watermark. Please mirror the .NET transaction-scoped advisory lock here and add the corresponding cross-instance ordering test.
| class FeatureIndex(IntEnum): | ||
| """PostgreSQL-owned feature-usage indexes.""" | ||
|
|
||
| POSTGRES_MEMORY = 75 |
There was a problem hiding this comment.
Python bit 75 is already assigned to declarative.agent in the per-language registry (docs/specs/feature-usage-bit-registry.md:177). Every Postgres memory operation therefore emits the declarative-agent bit, making those two capabilities indistinguishable and falsely reporting declarative usage. Please allocate an unused Python index (the registry reserves 89-127) and add the corresponding Python registry row rather than copying the independent .NET index.
Motivation & Context
Applications using PostgreSQL need a way to retain agent memory across conversations and recover workflow state after a process restart without introducing a separate persistence service.
This change adds PostgreSQL-backed durable agent memory and workflow checkpoint storage for .NET and Python. It supports cross-session recall, long-running workflows, and human-in-the-loop workflows that must pause and resume in another process.
Memory retrieval uses pgvector and PostgreSQL full-text search. Workflow checkpoint storage works independently and does not require pgvector, embedding generation, model inference, or Azure AI reranking.
Description & Review Guide
What are the major changes?
azure_ai.rank()reranking. Reranking is disabled by default and uses the configurableCohere-rerank-v4.0-fastmodel name when enabled.What is the impact of these changes?
What do you want reviewers to focus on?
azure_ai.rank()result columns differ from those expected by the connectors; Python's exception wrapping prevents the intended reranking fallback; and concurrent .NET integration tests showed connection-open timeouts despite passing individually. Successful live Cohere v4 reranking has not yet been verified.Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.