Repository navigation
[SPARK-62056][CONNECT][PYTHON] Take the arguments of the methods of the maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors - #873
Closed
HyukjinKwon wants to merge 1 commit into
Conversation
dongjoon-hyun
approved these changes
Nov 19, 2026
…he maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors
The methods of the containers of the drop-in's Rust protobuf runtime (pyspark.sql.connect.proto)
take their arguments as upb's do, with upb's errors, where they had PyO3's keyword arguments and
errors:
- Maps: MergeFrom, clear, GetEntryClass and get_or_create are upb's C methods of one argument or
of none, positional only, with CPython's errors naming upb's class, ScalarMapContainer or
MessageMapContainer ("ScalarMapContainer.MergeFrom() takes exactly one argument (0 given)").
setdefault takes no keyword arguments, refuses a map of messages before it counts its
arguments, and counts them as PyArg_UnpackTuple does. get parses its key and default as
PyArg_ParseTupleAndKeywords("O|O") does. pop, popitem, keys, values and items are
collections.abc's MutableMapping's and Mapping's, as upb's are, so their errors are Python's.
- Repeated containers: append, extend, remove and MergeFrom take one positional argument, insert
and pop parse theirs as PyArg_ParseTuple's "nO" and "|n" (an index of its __index__, with
"Python int too large to convert to C ssize_t"), and __deepcopy__ takes no keyword argument, as
upb's. Before, append(value=...), insert(key=..., value=...), pop(key=...), remove(elem=...)
and MergeFrom(other=...) were taken.
- A keyword argument of upb's METH_VARARGS methods (setdefault, insert, pop, __deepcopy__) is
CPython's error of the method of its class when it is called of its object, and of the method
only when CPython calls it as a bound method, as it calls each before Python 3.12 while a trace
or a profile function is set (coverage's): the runtime's error names the method only then too.
- docs/compatibility.md: the arguments of the containers' methods, and why a dunder method called
directly with the wrong arguments keeps CPython's error of a slot wrapper, and a keyword argument
of a bound method object of a METH_VARARGS method names the class; a reflowed paragraph.
Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
SPARK-62056-pbmsg-container-argument-errors
branch
from
November 19, 2026 04:06
5827755 to
a38cde9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A follow-up of #854 (SPARK-62010) and #860 (SPARK-62019). The methods of the containers of the drop-in's Rust protobuf runtime (
pyspark.sql.connect.proto,crates/pyspark-rs/src/pbmsg/containers.rs) take their arguments as upb's do (protobuf 7.36.2, the implementation that pyspark-client 4.3.0 uses), with upb's errors. Before, they had PyO3's arguments and errors. Each case was probed on both clients.Maps. upb's maps are of its C types
ScalarMapContainerandMessageMapContainer.MergeFrom,clear,GetEntryClassandget_or_createare C methods of one argument (METH_O) or of none (METH_NOARGS). They take positional arguments only, with CPython's errors naming upb's class:ScalarMapContainer.MergeFrom() takes exactly one argument (0 given);MessageMapContainer.clear() takes no keyword arguments.The runtime's errors named
MapContainer, andMergeFromtook an argument namedother. These now go through the helpersno_arguments,one_argumentandno_keywords. An argument error ofMergeFromcomes before the new version of the map, as upb's comes before its C function runs.setdefaultfollows upb's checks in order:NotImplementedError, whatever the arguments;PyArg_UnpackTupledoes (setdefault expected at least 1 argument, got 0).Before,
setdefault("k", default="d")set the key.getparses itskeyanddefaultasPyArg_ParseTupleAndKeywords(args, kwargs, "O|O", {"key", "default"})does. Its errors come in CPython's order (get_arguments):function takes at most 2 arguments (3 given), or... 2 keyword arguments ...when all are keywords;function missing required argument 'key' (pos 1);argument for function given by name ('key') and position (1);'x' is an invalid keyword argument for this function.pop,popitem,keys,valuesanditemsarecollections.abc'sMutableMappingandMappingmethods for upb's maps. They now call those functions (abc_method), so their results and their argument errors are Python's, in any version:MutableMapping.pop() missing 1 required positional argument: 'key'.updatealready followedMutableMapping.update([SPARK-62010][CONNECT][PYTHON] Make the maps of the Rust protobuf runtime merge their arguments, fail an iteration that they change, and order integer keys as upb's maps do #854).Repeated containers.
RepeatedScalarContainerandRepeatedCompositeContainertake positional arguments only, as upb's do.append,extend,removeandMergeFromtake exactly one argument (METH_O).insertandpopparse theirs asPyArg_ParseTuple's"nO"and"|n"do:function takes exactly 2 arguments (1 given);function takes at most 1 argument (2 given);__index__, withPython int too large to convert to C ssize_t, whichssize_t_ofgives throughPyNumber_IndexandPyLong_AsSsize_t. Before, PyO3's text said "C long".__deepcopy__takes any positional arguments and no keyword argument.TypeError: ... takes no keyword arguments:append(value=...),insert(key=..., value=...),pop(key=...),remove(elem=...),MergeFrom(other=...)and__deepcopy__(memo=...).extendandMergeFrommove intoextend_values, which both call.The wording of a keyword argument of upb's
METH_VARARGSmethods (no_keywords_of_positional).setdefault,insert,popand__deepcopy__areMETH_VARARGSmethods, and CPython words their keyword-argument error by the way the method is called:map.setdefault(key="k")), the error names the method of its class:ScalarMapContainer.setdefault() takes no keyword arguments;setdefault() takes no keyword arguments.trace_call_functioncalls every method of a native type as a bound method, so that a profiler sees it.coverage run, which the coverage job uses, sets a trace function, so there upb's error names the method only.sys.gettrace()orsys.getprofile()set. Elsewhere it names the class too.METH_OandMETH_NOARGSmethods (MergeFrom,append,clear, ...) name the class in every call, on upb and on the runtime alike.Docs (
docs/compatibility.md).setdefault,insert,popor__deepcopy__called through a bound method object (f = map.setdefault; f(key="k")): upb's error names the method only, and the runtime's, which cannot tell that call from one on the map, names the class too. Second, the one below. A dunder method of a map called directly with the wrong arguments (map.__eq__()) gets the error of CPython's slot wrapper of the native type,expected 1 argument, got 0. upb's__eq__of a map isMapping.__eq__, a Python function, whose error isMapping.__eq__() missing 1 required positional argument: 'other'.Why are the changes needed?
The containers took keyword arguments that upb refuses, so code that runs on the drop-in failed on pyspark-client. Their errors differed from upb's too. One example:
setdefault("k", default="d")set the key where upb raisesTypeError.Does this PR introduce any user-facing change?
Yes, the arguments and errors of these methods, as upb's:
options.MergeFrom()TypeError: ScalarMapContainer.MergeFrom() takes exactly one argument (0 given)TypeError: MapContainer.MergeFrom() missing 1 required positional argument: 'other'options.MergeFrom(other=[("a", "b")])TypeError: ScalarMapContainer.MergeFrom() takes no keyword argumentsoptions.setdefault("k", default="d")TypeError: ScalarMapContainer.setdefault() takes no keyword arguments'd'options.get("k", x=1)TypeError: 'x' is an invalid keyword argument for this functionTypeError: MapContainer.get() got an unexpected keyword argument 'x'options.pop()TypeError: MutableMapping.pop() missing 1 required positional argument: 'key'TypeError: MapContainer.pop() missing 1 required positional argument: 'key'probabilities.append(value=0.5)TypeError: RepeatedScalarContainer.append() takes no keyword argumentsprobabilities.insert(0)TypeError: function takes exactly 2 arguments (1 given)TypeError: RepeatedScalarContainer.insert() missing 1 required positional argument: 'value'elements.pop(2**70)OverflowError: Python int too large to convert to C ssize_tOverflowError: Python int too large to convert to C longHow was this patch tested?
A new test in
python/tests/test_proto_rust_offline.py,test_methods_of_containers_raise_upbs_errors_of_their_arguments. It fails on master and passes with this change. It compares against upb's classes:setdefault,insert,popand__deepcopy__, with no profile function and with one set, so that a plain run also covers the bound calls thatcoveragemakes.Each call compares its outcome and the bytes of the message after it. The test passes under plain pytest and under
coverage run --rcfile=.coveragerc, as coverage.yml runs it. Undercoverage run, the first push of the PR failed onsetdefault(key=k).Probes of both clients (130 cases of maps, 94 of repeated containers, and the keyword arguments of the four
METH_VARARGSmethods: plain, undersys.settrace, undersys.setprofile, and through a bound method object). All give upb's results except the documented differences: the names of the types in anAttributeError, a dunder method of a map called directly, and a bound method object called with a keyword argument and no trace function.Rust gates:
cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;cargo test -p pyspark-rs --lib, linked asrust.ymllinks it: 15 passed.Python gates:
checkandformat --check,test_ruff_excludes.pyandscripts/audit_no_stubs.sh;gen_type_stubs.py --checkandgen_fn_dispatch.py --check, and the guards of the test lists;test_type_stubs_offline.pywith mypy 2.4.1: 13 passed and 3 skipped.The offline suite of
coverage.yml: 6495 passed and 77 skipped, the skips of master.test_dropin_e2e.pyagainst a Spark 4.3.0 server: 333 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.