Skip to content

Python: Keep null fields in negated Azure Cosmos DB eq filters - #8771

Merged
Eduard van Valkenburg (eavanvalkenburg) merged 2 commits into
microsoft:mainfrom
nniiovoo:nniiovoo/cosmos-not-eq-null
Sep 28, 2026
Merged

Eduard van Valkenburg (eavanvalkenburg) merged 2 commits into
microsoft:mainfrom
nniiovoo:nniiovoo/cosmos-not-eq-null

Conversation

@nniiovoo

Copy link
Copy Markdown
Contributor

Motivation & Context

NOT over an eq filter on CosmosCollection leaves out records whose field is null, while InMemoryCollection and the other connectors include them. This affects both get(filter=...) and search(filter=...). eq was translated to (IS_DEFINED(x) AND x = @p). In Cosmos SQL, null = @p is undefined, true AND undefined is undefined, and NOT undefined stays undefined, so those items aren't returned. Details and repro in the linked issue.

Description & Review Guide

  • What are the major changes? eq is now translated to (IS_DEFINED(x) AND NOT IS_NULL(x) AND x = @p). That's the same guard the in/not_in and ordered operators already use, and matches what the SQL Server connector does for eq. The three exact-SQL test expectations are updated. There's a new NOT-over-eq case on the nullable optional field, and an assertion in the live integration test.
  • What is the impact of these changes? Under NOT, null rows are now returned, matching the portable semantics. Top-level eq results are unchanged: Filter rejects eq with None, so null rows never matched before either.
  • What do you want reviewers to focus on? The eq branch in _translate_filter. The unit tests pin the generated SQL. Only the opt-in live test (AZURE_COSMOS_VECTOR_TESTS=1, not run in CI) checks returned rows, and I couldn't run it without a Cosmos account. The new assertion runs before the missing record is inserted, so the expected set is {one, three}.

Related Issue

Fixes #8770

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

CosmosCollection translated eq as (IS_DEFINED(x) AND x = @p). For a
null field, x = @p is undefined in Cosmos SQL, and NOT undefined stays
undefined, so NOT(eq) dropped rows whose field is null. Guard eq with
NOT IS_NULL(x), as the in/not_in and ordered operators already do.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The focused translation change correctly handles Cosmos three-valued logic and is adequately covered by unit and integration tests.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes Cosmos DB negated equality filters so records with null fields match portable connector semantics.

Changes:

  • Adds an explicit null guard to equality predicates.
  • Updates SQL translation tests and adds null-field regression coverage.
  • Verifies behavior against live Cosmos DB integration tests.
File Description
python/​packages/​azure-cosmos/​agent_framework_azure_cosmos/​_vector_store.py Adds null-safe equality translation.
python/​packages/​azure-cosmos/​tests/​azure_cosmos/​test_vector_store.py Updates SQL expectations and regression coverage.
python/​packages/​azure-cosmos/​tests/​azure_cosmos/​test_vector_store_integration.py Verifies negated equality includes null records.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

@nniiovoo

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree

Merged via the queue into microsoft:main with commit e9d9962 Sep 28, 2026
44 checks passed

This branch was previously deployed

1 inactive deployment
github-app-auth — 59dc5386 Deployed Sep 28, 2026 by eavanvalkenburg via add_label #23858
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Bug]: Cosmos vector store drops rows whose field is null from NOT(eq) filters

3 participants