Skip to content

Python: Keep null fields out of Azure DocumentDB membership filters - #8654

Merged
Eduard van Valkenburg (eavanvalkenburg) merged 1 commit into
microsoft:mainfrom
saquibjawedbit:fix-documentdb-in-null
Sep 23, 2026
Merged

Eduard van Valkenburg (eavanvalkenburg) merged 1 commit into
microsoft:mainfrom
saquibjawedbit:fix-documentdb-in-null

Conversation

@saquibjawedbit

Copy link
Copy Markdown
Contributor

Motivation & Context

Portable in/not_in filters never match a null field. That is how the core reference evaluator behaves (_evaluate_filter, used by InMemoryCollection), and the Postgres, Redis, MongoDB and Azure Cosmos DB connectors all follow it. The Azure DocumentDB connector kept None in the $in operands and added only an $exists guard, so Filter("category", "in", [None, "a"]) also returned documents whose category is null, and Filter("category", "in", [None]) returned every null record instead of none. The same filter therefore returned different records depending on the store.

Description & Review Guide

  • What are the major changes? _FilterCompiler._prepare_filter_condition drops None operands for in/not_in, as the MongoDB and Redis connectors already do. An in with only non-null operands is a plain $in, which cannot match null or missing fields. not_in keeps its existing $exists + $ne: null guards, so its output is unchanged apart from None no longer appearing in $nin. A regression test covers in and not_in with and without other operands.
  • What is the impact of these changes? DocumentDB membership filters now return the same records as the in-memory reference and the other connectors. Filters without None operands compile exactly as before.
  • What do you want reviewers to focus on? The in/not_in branch in _prepare_filter_condition. None is the only input _prepare_filter_operand maps to None, so filtering it before preparation removes every null operand.

Related Issue

Fixes #8653

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

Portable in/not_in membership never matches a null field, but the
DocumentDB filter compiler kept None in the $in operands behind an
$exists guard, so Filter("x", "in", [None, "a"]) also returned
documents whose field is null. Drop None operands like the MongoDB,
Redis and Postgres connectors do.

Fixes microsoft#8653

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Saquib Jawed (@saquibjawedbit) Reviewed the current head; no blocking findings.

Merged via the queue into microsoft:main with commit 23d5d25 Sep 23, 2026
49 checks passed

This branch was previously deployed

1 inactive deployment
github-app-auth — d25ec554 Deployed Sep 22, 2026 by saquibjawedbit via team_check #5090
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Bug]: Azure DocumentDB 'in' filter with a None operand matches null fields

3 participants