Conversation
There was a problem hiding this comment.
Code Review
This pull request adds duck-typing support for native PyMongo and third-party BSON objects (such as ObjectId, Decimal128, Regex, Timestamp, MinKey, and MaxKey) within the encode_value helper function, along with corresponding unit tests. The review feedback suggests optimizing this performance-critical code path by using an O(1) set lookup for class names to avoid unnecessary attribute checks and recommends running benchmarks to verify that these changes do not introduce performance regressions.
| if hasattr(value, "__class__"): | ||
| cls_name = value.__class__.__name__ | ||
| if cls_name == "ObjectId" and hasattr(value, "binary"): | ||
| return encode_value({"__oid__": str(value).lower()}) | ||
| if cls_name == "Decimal128" and hasattr(value, "to_decimal"): | ||
| return encode_value({"__decimal128__": str(value)}) | ||
| if cls_name == "Regex" and hasattr(value, "pattern"): | ||
| opts = getattr(value, "flags", "") or getattr(value, "options", "") | ||
| return encode_value( | ||
| {"__regex__": {"pattern": value.pattern, "options": str(opts)}} | ||
| ) | ||
| if cls_name == "Timestamp" and hasattr(value, "time") and hasattr(value, "inc"): | ||
| return encode_value( | ||
| { | ||
| "__request_timestamp__": { | ||
| "seconds": value.time, | ||
| "increment": value.inc, | ||
| } | ||
| } | ||
| ) | ||
| if cls_name == "MinKey": | ||
| return encode_value({"__min__": None}) | ||
| if cls_name == "MaxKey": | ||
| return encode_value({"__max__": None}) |
There was a problem hiding this comment.
The encode_value function is a critical hot path called recursively for every field of every document during serialization. While the proposed change aims to optimize standard types using an O(1) set lookup, any changes to this performance-critical code path must be validated and benchmarked to ensure they do not degrade performance or eliminate fast-path optimizations (such as the overhead of double getattr calls). Please run benchmarks to verify the performance impact of this change.
cls_name = getattr(getattr(value, "__class__", None), "__name__", None)
if cls_name in {"ObjectId", "Decimal128", "Regex", "Timestamp", "MinKey", "MaxKey"}:
if cls_name == "ObjectId" and hasattr(value, "binary"):
return encode_value({"__oid__": str(value).lower()})
elif cls_name == "Decimal128" and hasattr(value, "to_decimal"):
return encode_value({"__decimal128__": str(value)})
elif cls_name == "Regex" and hasattr(value, "pattern"):
opts = getattr(value, "flags", "") or getattr(value, "options", "")
return encode_value(
{"__regex__": {"pattern": value.pattern, "options": str(opts)}}
)
elif cls_name == "Timestamp" and hasattr(value, "time") and hasattr(value, "inc"):
return encode_value(
{
"__request_timestamp__": {
"seconds": value.time,
"increment": value.inc,
}
}
)
elif cls_name == "MinKey":
return encode_value({"__min__": None})
elif cls_name == "MaxKey":
return encode_value({"__max__": None})References
- For performance-critical code paths executed on every request, validate and benchmark any proposed readability simplifications to ensure they do not degrade performance or eliminate fast-path optimizations.
There was a problem hiding this comment.
Addressed: Moved the duck-typing inspection entirely out of the standard serialization hot path down to the fallback position right before TypeError is raised. Standard native Python types (int, float, str, dict, list, etc.) and native Firestore types bypass this check completely with zero overhead.
e40a2d2 to
06445ee
Compare
06445ee to
80c0c3f
Compare
80c0c3f to
740326c
Compare
There was a problem hiding this comment.
it looks like we should add to this type here
There was a problem hiding this comment.
Done! Updated the encode_value docstring Args to include ~google.cloud.firestore_v1.bson._BSONType as well as supported BSON / PyMongo-compatible types.
740326c to
7038386
Compare
7038386 to
337ec30
Compare
337ec30 to
bb5e132
Compare
bb5e132 to
f5c66e0
Compare
| return bson.BSONDecimal128(value.to_decimal()) | ||
| if cls_name == "Regex" and hasattr(value, "pattern"): | ||
| opts = getattr(value, "flags", "") or getattr(value, "options", "") | ||
| return bson.BSONRegex(value.pattern, opts) |
There was a problem hiding this comment.
Can opts here be non-string? Gemini flagged this as a potential issue
(I know I started a thread suggesting to not support int, but I don't feel too strongly one way or the other if it would make things easier here. We just need to make sure we're communicating what flags we accept and what we reject clearly)
| value encoded as a Firestore protobuf. | ||
|
|
||
| Raises: | ||
| TypeError: If the ``value`` is not one of the accepted types. |
There was a problem hiding this comment.
I think some of the BSON constructors raise ValueError or others. We should probably either update this docstring to be exhaustive, or catch and wrap the errors raised in _try_duck_type_bson
| return document.Value(string_value=value) | ||
|
|
||
| if isinstance(value, bytes): | ||
| return document.Value(bytes_value=value) |
There was a problem hiding this comment.
It looks like PyMongo Binary types would be caught here before reaching _try_duck_type_bson. This means we would lose the subtype field
f5c66e0 to
4a5b47f
Compare
4a5b47f to
03363e3
Compare
03363e3 to
018974b
Compare
…lidation
- Restrict BSONRegex options parameter to string only, removing re.RegexFlag and int support.
- Add client-side validation for BSON regex option characters ("i", "m", "s", "x", "u", "a"), raising ValueError on invalid options.
- Remove obsolete re-flag test and clean up unused import in test_bson.py.
Remove client-side regex options character validation and whitelist in BSONRegex, letting the Firestore backend validate regex option flags. Preserves alphabetical sorting and deduplication per BSON specification while avoiding unhandled ValueError exceptions during read deserialization when documents contain standard BSON options like 'l' (locale). Fixes: b/562163604
Add sync and async system test cases verifying that writing a BSONRegex with unsupported options (such as 'l') fails backend validation with google.api_core.exceptions.InvalidArgument and an informative error message on an Enterprise database. Fixes: b/562163604
- Convert to_decimal from property to method call to_decimal(). - Add __float__ and __int__ numeric protocol methods to BSONDecimal128. - Improve __eq__ and __hash__ for IEEE 754 special values (NaN, -NaN, sNaN, inf, -inf) using Decimal.is_nan(). - Add comprehensive test cases in test_bson.py for float/int conversions and special decimal values.
- Safely convert sNaN and -sNaN to float("nan") / float("-nan") in __float__ without raising ValueError.
Towards #18395
…trings Disallow passing float to BSONDecimal128 to prevent binary floating-point precision loss and edge cases, aligning with PyMongo's Decimal128 behavior. Validate string and numeric inputs via decimal.Decimal in __init__ to reject invalid numeric strings.
…ation - Perform automatic BSON deserialization in decode_dict and DocumentSnapshot.to_dict using _BSONType._from_dict. - Remove decode_bson configuration parameter across Client, AsyncClient, BaseClient, and DocumentSnapshot. - Preserve precise return type annotations in decode_dict and restore docstring Raises section. Towards #18402
…ecode_value - Restore full Union return type with _BSONType on decode_value. - Restore Returns and Raises docstring sections in decode_value matching base branch. - Remove unused _BSON_DECODERS import from _helpers.py. - Revert extraneous changes to pipeline_result.py. Towards #18402
… types with _BSONType - Annotate decode_dict with Union[dict, Vector, _BSONType]. - Update PipelineResult.data to return dict | Vector | _BSONType | None. - Import _BSONType under TYPE_CHECKING in pipeline_result.py. Towards #18402
…nsions for librarian - Make client a required positional parameter in decode_value and decode_dict. - Format comprehensions in _helpers.py as single lines to satisfy librarian generation check. Towards #18402
Rename abstract base class _BSONType to BSONType and export it in google.cloud.firestore_v1 and __all__. Update return type annotations and docstrings on decode_value, decode_dict, and PipelineResult.data.
…risons Use _BSON_KEY_TO_TYPE_ORDER dictionary lookup in order.py for O(1) wire key resolution, decoupling bson.py from query ordering. Consolidate cross-type numeric comparisons in compare_numbers with safe Decimal handling and restore compare_doubles to standard float comparisons.
Rename ambiguous single-letter variable l to left_val to satisfy flake8 E741 and align expression formatting with ruff.
…pare_numbers Separate BSON_TIMESTAMP from native TIMESTAMP to conform to the cross-SDK 17-rank TypeOrder specification. Restore compare_timestamps to native Firestore timestamps and introduce compare_bson_timestamps. Extract module-level _to_number and _is_nan helpers, directly inspecting protobuf fields to optimize numeric comparisons.
…ate to BSON types Move duck-typing checks in encode_value() to a fallback position before raising TypeError, preserving hot-path performance for standard types. Delegate conversion of duck-typed objects to Firestore BSON classes rather than manually creating wire dictionaries, and document supported BSON types in the encode_value docstring.
… typing Support PyMongo Binary subclasses without losing subtype tags by inspecting subtype attribute in bytes check. Support integer regex bitmasks in duck-typed Regex by mapping standard re flags to BSON options characters. Document ValueError in encode_value.
018974b to
977f9c0
Compare
Adds duck-typing detection in
encode_value()for native PyMongo / third-party BSON objects (bson.ObjectId,bson.Decimal128,bson.Regex,bson.Timestamp,bson.MinKey,bson.MaxKey).Allows applications migrating from MongoDB that use native PyMongo BSON objects to write directly to Firestore without triggering
TypeErrorserialization failures or requiring manual type conversions.Note that this is optional work and can be skipped. We can discuss this offline.
Fixes b/562164315 🦕