Skip to content

feat(firestore): add PyMongo duck-typing serialization support - #18406

Open
ohmayr wants to merge 21 commits into
bson-pr3-orderingfrom
bson-pr4-duck-typing
Open

ohmayr wants to merge 21 commits into
bson-pr3-orderingfrom
bson-pr4-duck-typing

Conversation

@ohmayr

@ohmayr ohmayr commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Adds duck-typing detection in encode_value() for native PyMongo / third-party BSON objects (bson.ObjectId, bson.Decimal128, bson.Regex, bson.Timestamp, bson.MinKey, bson.MaxKey).

Allows applications migrating from MongoDB that use native PyMongo BSON objects to write directly to Firestore without triggering TypeError serialization failures or requiring manual type conversions.

Note that this is optional work and can be skipped. We can discuss this offline.

Fixes b/562164315 🦕

@ohmayr
ohmayr added this pull request to stack #18386 September 16, 2026 22:57

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds duck-typing support for native PyMongo and third-party BSON objects (such as ObjectId, Decimal128, Regex, Timestamp, MinKey, and MaxKey) within the encode_value helper function, along with corresponding unit tests. The review feedback suggests optimizing this performance-critical code path by using an O(1) set lookup for class names to avoid unnecessary attribute checks and recommends running benchmarks to verify that these changes do not introduce performance regressions.

Comment on lines +218 to +241
if hasattr(value, "__class__"):
cls_name = value.__class__.__name__
if cls_name == "ObjectId" and hasattr(value, "binary"):
return encode_value({"__oid__": str(value).lower()})
if cls_name == "Decimal128" and hasattr(value, "to_decimal"):
return encode_value({"__decimal128__": str(value)})
if cls_name == "Regex" and hasattr(value, "pattern"):
opts = getattr(value, "flags", "") or getattr(value, "options", "")
return encode_value(
{"__regex__": {"pattern": value.pattern, "options": str(opts)}}
)
if cls_name == "Timestamp" and hasattr(value, "time") and hasattr(value, "inc"):
return encode_value(
{
"__request_timestamp__": {
"seconds": value.time,
"increment": value.inc,
}
}
)
if cls_name == "MinKey":
return encode_value({"__min__": None})
if cls_name == "MaxKey":
return encode_value({"__max__": None})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The encode_value function is a critical hot path called recursively for every field of every document during serialization. While the proposed change aims to optimize standard types using an O(1) set lookup, any changes to this performance-critical code path must be validated and benchmarked to ensure they do not degrade performance or eliminate fast-path optimizations (such as the overhead of double getattr calls). Please run benchmarks to verify the performance impact of this change.

    cls_name = getattr(getattr(value, "__class__", None), "__name__", None)
    if cls_name in {"ObjectId", "Decimal128", "Regex", "Timestamp", "MinKey", "MaxKey"}:
        if cls_name == "ObjectId" and hasattr(value, "binary"):
            return encode_value({"__oid__": str(value).lower()})
        elif cls_name == "Decimal128" and hasattr(value, "to_decimal"):
            return encode_value({"__decimal128__": str(value)})
        elif cls_name == "Regex" and hasattr(value, "pattern"):
            opts = getattr(value, "flags", "") or getattr(value, "options", "")
            return encode_value(
                {"__regex__": {"pattern": value.pattern, "options": str(opts)}}
            )
        elif cls_name == "Timestamp" and hasattr(value, "time") and hasattr(value, "inc"):
            return encode_value(
                {
                    "__request_timestamp__": {
                        "seconds": value.time,
                        "increment": value.inc,
                    }
                }
            )
        elif cls_name == "MinKey":
            return encode_value({"__min__": None})
        elif cls_name == "MaxKey":
            return encode_value({"__max__": None})
References
  1. For performance-critical code paths executed on every request, validate and benchmark any proposed readability simplifications to ensure they do not degrade performance or eliminate fast-path optimizations.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed: Moved the duck-typing inspection entirely out of the standard serialization hot path down to the fallback position right before TypeError is raised. Standard native Python types (int, float, str, dict, list, etc.) and native Firestore types bypass this check completely with zero overhead.

@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from e40a2d2 to 06445ee Compare September 16, 2026 23:04
@ohmayr
ohmayr marked this pull request as ready for review September 16, 2026 23:11
@ohmayr
ohmayr requested a review from a team as a code owner September 16, 2026 23:11
@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from 06445ee to 80c0c3f Compare September 16, 2026 23:19
@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from 80c0c3f to 740326c Compare September 18, 2026 20:06

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like we should add to this type here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Updated the encode_value docstring Args to include ~google.cloud.firestore_v1.bson._BSONType as well as supported BSON / PyMongo-compatible types.

Comment thread packages/google-cloud-firestore/google/cloud/firestore_v1/_helpers.py Outdated
@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from 740326c to 7038386 Compare September 18, 2026 22:11
@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from 7038386 to 337ec30 Compare September 18, 2026 22:52
@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from 337ec30 to bb5e132 Compare September 19, 2026 12:05
@ohmayr
ohmayr requested a review from a team as a code owner September 21, 2026 19:31
@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from bb5e132 to f5c66e0 Compare September 21, 2026 20:00
return bson.BSONDecimal128(value.to_decimal())
if cls_name == "Regex" and hasattr(value, "pattern"):
opts = getattr(value, "flags", "") or getattr(value, "options", "")
return bson.BSONRegex(value.pattern, opts)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can opts here be non-string? Gemini flagged this as a potential issue

(I know I started a thread suggesting to not support int, but I don't feel too strongly one way or the other if it would make things easier here. We just need to make sure we're communicating what flags we accept and what we reject clearly)

value encoded as a Firestore protobuf.

Raises:
TypeError: If the ``value`` is not one of the accepted types.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think some of the BSON constructors raise ValueError or others. We should probably either update this docstring to be exhaustive, or catch and wrap the errors raised in _try_duck_type_bson

return document.Value(string_value=value)

if isinstance(value, bytes):
return document.Value(bytes_value=value)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like PyMongo Binary types would be caught here before reaching _try_duck_type_bson. This means we would lose the subtype field

@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from f5c66e0 to 4a5b47f Compare September 22, 2026 23:18
@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from 4a5b47f to 03363e3 Compare September 22, 2026 23:22
@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from 03363e3 to 018974b Compare September 23, 2026 20:21
…lidation

- Restrict BSONRegex options parameter to string only, removing re.RegexFlag and int support.
- Add client-side validation for BSON regex option characters ("i", "m", "s", "x", "u", "a"), raising ValueError on invalid options.
- Remove obsolete re-flag test and clean up unused import in test_bson.py.
Remove client-side regex options character validation and whitelist in
BSONRegex, letting the Firestore backend validate regex option flags.
Preserves alphabetical sorting and deduplication per BSON specification
while avoiding unhandled ValueError exceptions during read deserialization
when documents contain standard BSON options like 'l' (locale).

Fixes: b/562163604
Add sync and async system test cases verifying that writing a BSONRegex
with unsupported options (such as 'l') fails backend validation with
google.api_core.exceptions.InvalidArgument and an informative error
message on an Enterprise database.

Fixes: b/562163604
- Convert to_decimal from property to method call to_decimal().
- Add __float__ and __int__ numeric protocol methods to BSONDecimal128.
- Improve __eq__ and __hash__ for IEEE 754 special values (NaN, -NaN, sNaN, inf, -inf) using Decimal.is_nan().
- Add comprehensive test cases in test_bson.py for float/int conversions and special decimal values.
- Safely convert sNaN and -sNaN to float("nan") / float("-nan") in __float__ without raising ValueError.

Towards #18395
…trings

Disallow passing float to BSONDecimal128 to prevent binary floating-point
precision loss and edge cases, aligning with PyMongo's Decimal128
behavior. Validate string and numeric inputs via decimal.Decimal in
__init__ to reject invalid numeric strings.
…ation

- Perform automatic BSON deserialization in decode_dict and DocumentSnapshot.to_dict using _BSONType._from_dict.
- Remove decode_bson configuration parameter across Client, AsyncClient, BaseClient, and DocumentSnapshot.
- Preserve precise return type annotations in decode_dict and restore docstring Raises section.

Towards #18402
…ecode_value

- Restore full Union return type with _BSONType on decode_value.
- Restore Returns and Raises docstring sections in decode_value matching base branch.
- Remove unused _BSON_DECODERS import from _helpers.py.
- Revert extraneous changes to pipeline_result.py.

Towards #18402
… types with _BSONType

- Annotate decode_dict with Union[dict, Vector, _BSONType].
- Update PipelineResult.data to return dict | Vector | _BSONType | None.
- Import _BSONType under TYPE_CHECKING in pipeline_result.py.

Towards #18402
…nsions for librarian

- Make client a required positional parameter in decode_value and decode_dict.
- Format comprehensions in _helpers.py as single lines to satisfy librarian generation check.

Towards #18402
Rename abstract base class _BSONType to BSONType and export it in
google.cloud.firestore_v1 and __all__. Update return type annotations
and docstrings on decode_value, decode_dict, and PipelineResult.data.
…risons

Use _BSON_KEY_TO_TYPE_ORDER dictionary lookup in order.py for O(1) wire
key resolution, decoupling bson.py from query ordering. Consolidate
cross-type numeric comparisons in compare_numbers with safe Decimal
handling and restore compare_doubles to standard float comparisons.
Rename ambiguous single-letter variable l to left_val to satisfy flake8
E741 and align expression formatting with ruff.
…pare_numbers

Separate BSON_TIMESTAMP from native TIMESTAMP to conform to the
cross-SDK 17-rank TypeOrder specification. Restore compare_timestamps
to native Firestore timestamps and introduce compare_bson_timestamps.
Extract module-level _to_number and _is_nan helpers, directly inspecting
protobuf fields to optimize numeric comparisons.
…ate to BSON types

Move duck-typing checks in encode_value() to a fallback position before
raising TypeError, preserving hot-path performance for standard types.
Delegate conversion of duck-typed objects to Firestore BSON classes
rather than manually creating wire dictionaries, and document supported
BSON types in the encode_value docstring.
… typing

Support PyMongo Binary subclasses without losing subtype tags by inspecting
subtype attribute in bytes check. Support integer regex bitmasks in duck-typed
Regex by mapping standard re flags to BSON options characters. Document
ValueError in encode_value.
@ohmayr
ohmayr force-pushed the bson-pr4-duck-typing branch from 018974b to 977f9c0 Compare September 24, 2026 00:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants