Skip to content

[SPARK-62056][CONNECT][PYTHON] Take the arguments of the methods of the maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors - #873

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62056-pbmsg-container-argument-errors
Closed

HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62056-pbmsg-container-argument-errors

Conversation

@HyukjinKwon

@HyukjinKwon HyukjinKwon commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

A follow-up of #854 (SPARK-62010) and #860 (SPARK-62019). The methods of the containers of the drop-in's Rust protobuf runtime (pyspark.sql.connect.proto, crates/pyspark-rs/src/pbmsg/containers.rs) take their arguments as upb's do (protobuf 7.36.2, the implementation that pyspark-client 4.3.0 uses), with upb's errors. Before, they had PyO3's arguments and errors. Each case was probed on both clients.

Maps. upb's maps are of its C types ScalarMapContainer and MessageMapContainer.

  • MergeFrom, clear, GetEntryClass and get_or_create are C methods of one argument (METH_O) or of none (METH_NOARGS). They take positional arguments only, with CPython's errors naming upb's class:

    • ScalarMapContainer.MergeFrom() takes exactly one argument (0 given);
    • MessageMapContainer.clear() takes no keyword arguments.

    The runtime's errors named MapContainer, and MergeFrom took an argument named other. These now go through the helpers no_arguments, one_argument and no_keywords. An argument error of MergeFrom comes before the new version of the map, as upb's comes before its C function runs.

  • setdefault follows upb's checks in order:

    1. it takes no keyword arguments;
    2. a map of messages raises its NotImplementedError, whatever the arguments;
    3. a map of scalars counts the arguments as PyArg_UnpackTuple does (setdefault expected at least 1 argument, got 0).

    Before, setdefault("k", default="d") set the key.

  • get parses its key and default as PyArg_ParseTupleAndKeywords(args, kwargs, "O|O", {"key", "default"}) does. Its errors come in CPython's order (get_arguments):

    • too many arguments: function takes at most 2 arguments (3 given), or ... 2 keyword arguments ... when all are keywords;
    • a missing key: function missing required argument 'key' (pos 1);
    • an argument given twice: argument for function given by name ('key') and position (1);
    • an unknown keyword: 'x' is an invalid keyword argument for this function.
  • pop, popitem, keys, values and items are collections.abc's MutableMapping and Mapping methods for upb's maps. They now call those functions (abc_method), so their results and their argument errors are Python's, in any version: MutableMapping.pop() missing 1 required positional argument: 'key'. update already followed MutableMapping.update ([SPARK-62010][CONNECT][PYTHON] Make the maps of the Rust protobuf runtime merge their arguments, fail an iteration that they change, and order integer keys as upb's maps do #854).

Repeated containers. RepeatedScalarContainer and RepeatedCompositeContainer take positional arguments only, as upb's do.

  • append, extend, remove and MergeFrom take exactly one argument (METH_O).
  • insert and pop parse theirs as PyArg_ParseTuple's "nO" and "|n" do:
    • function takes exactly 2 arguments (1 given);
    • function takes at most 1 argument (2 given);
    • an index of its __index__, with Python int too large to convert to C ssize_t, which ssize_t_of gives through PyNumber_Index and PyLong_AsSsize_t. Before, PyO3's text said "C long".
  • __deepcopy__ takes any positional arguments and no keyword argument.
  • Before, these all succeeded where upb raises TypeError: ... takes no keyword arguments: append(value=...), insert(key=..., value=...), pop(key=...), remove(elem=...), MergeFrom(other=...) and __deepcopy__(memo=...).
  • The bodies of extend and MergeFrom move into extend_values, which both call.

The wording of a keyword argument of upb's METH_VARARGS methods (no_keywords_of_positional).

  • upb's setdefault, insert, pop and __deepcopy__ are METH_VARARGS methods, and CPython words their keyword-argument error by the way the method is called:
    • called on its object (map.setdefault(key="k")), the error names the method of its class: ScalarMapContainer.setdefault() takes no keyword arguments;
    • called as a bound method object, it names the method only: setdefault() takes no keyword arguments.
  • Before Python 3.12, while a trace or a profile function is set, CPython's trace_call_function calls every method of a native type as a bound method, so that a profiler sees it. coverage run, which the coverage job uses, sets a trace function, so there upb's error names the method only.
  • The runtime's methods cannot tell the two calls apart. Their error now names the method only where CPython binds every call: before 3.12, with sys.gettrace() or sys.getprofile() set. Elsewhere it names the class too.
  • Probed with upb on Python 3.11, 3.12 and 3.13: 3.12 and later keep the class in the error under a trace or profile function.
  • The METH_O and METH_NOARGS methods (MergeFrom, append, clear, ...) name the class in every call, on upb and on the runtime alike.

Docs (docs/compatibility.md).

  • The difference list states that the containers' methods take upb's arguments and errors.
  • It also states two inherent exceptions. First, a keyword argument of setdefault, insert, pop or __deepcopy__ called through a bound method object (f = map.setdefault; f(key="k")): upb's error names the method only, and the runtime's, which cannot tell that call from one on the map, names the class too. Second, the one below. A dunder method of a map called directly with the wrong arguments (map.__eq__()) gets the error of CPython's slot wrapper of the native type, expected 1 argument, got 0. upb's __eq__ of a map is Mapping.__eq__, a Python function, whose error is Mapping.__eq__() missing 1 required positional argument: 'other'.
  • The paragraph that [SPARK-62010][CONNECT][PYTHON] Make the maps of the Rust protobuf runtime merge their arguments, fail an iteration that they change, and order integer keys as upb's maps do #854 left with the short line "either. A message of the" is reflowed.

Why are the changes needed?

The containers took keyword arguments that upb refuses, so code that runs on the drop-in failed on pyspark-client. Their errors differed from upb's too. One example: setdefault("k", default="d") set the key where upb raises TypeError.

Does this PR introduce any user-facing change?

Yes, the arguments and errors of these methods, as upb's:

pyspark-client 4.3.0, and now before
options.MergeFrom() TypeError: ScalarMapContainer.MergeFrom() takes exactly one argument (0 given) TypeError: MapContainer.MergeFrom() missing 1 required positional argument: 'other'
options.MergeFrom(other=[("a", "b")]) TypeError: ScalarMapContainer.MergeFrom() takes no keyword arguments merged
options.setdefault("k", default="d") TypeError: ScalarMapContainer.setdefault() takes no keyword arguments 'd'
options.get("k", x=1) TypeError: 'x' is an invalid keyword argument for this function TypeError: MapContainer.get() got an unexpected keyword argument 'x'
options.pop() TypeError: MutableMapping.pop() missing 1 required positional argument: 'key' TypeError: MapContainer.pop() missing 1 required positional argument: 'key'
probabilities.append(value=0.5) TypeError: RepeatedScalarContainer.append() takes no keyword arguments appended
probabilities.insert(0) TypeError: function takes exactly 2 arguments (1 given) TypeError: RepeatedScalarContainer.insert() missing 1 required positional argument: 'value'
elements.pop(2**70) OverflowError: Python int too large to convert to C ssize_t OverflowError: Python int too large to convert to C long

How was this patch tested?

  • A new test in python/tests/test_proto_rust_offline.py, test_methods_of_containers_raise_upbs_errors_of_their_arguments. It fails on master and passes with this change. It compares against upb's classes:

    • 42 calls of the methods of maps, on a map of scalars and on one of messages;
    • 33 calls of the methods of repeated fields, on a repeated scalar and on a repeated message field;
    • the keyword arguments of setdefault, insert, pop and __deepcopy__, with no profile function and with one set, so that a plain run also covers the bound calls that coverage makes.

    Each call compares its outcome and the bytes of the message after it. The test passes under plain pytest and under coverage run --rcfile=.coveragerc, as coverage.yml runs it. Under coverage run, the first push of the PR failed on setdefault(key=k).

  • Probes of both clients (130 cases of maps, 94 of repeated containers, and the keyword arguments of the four METH_VARARGS methods: plain, under sys.settrace, under sys.setprofile, and through a bound method object). All give upb's results except the documented differences: the names of the types in an AttributeError, a dunder method of a map called directly, and a bound method object called with a keyword argument and no trace function.

  • Rust gates:

    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;
    • cargo test -p pyspark-rs --lib, linked as rust.yml links it: 15 passed.
  • Python gates:

    • ruff 0.14.0 check and format --check, test_ruff_excludes.py and scripts/audit_no_stubs.sh;
    • gen_type_stubs.py --check and gen_fn_dispatch.py --check, and the guards of the test lists;
    • test_type_stubs_offline.py with mypy 2.4.1: 13 passed and 3 skipped.
  • The offline suite of coverage.yml: 6495 passed and 77 skipped, the skips of master.

  • test_dropin_e2e.py against a Spark 4.3.0 server: 333 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…he maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors

The methods of the containers of the drop-in's Rust protobuf runtime (pyspark.sql.connect.proto)
take their arguments as upb's do, with upb's errors, where they had PyO3's keyword arguments and
errors:

- Maps: MergeFrom, clear, GetEntryClass and get_or_create are upb's C methods of one argument or
  of none, positional only, with CPython's errors naming upb's class, ScalarMapContainer or
  MessageMapContainer ("ScalarMapContainer.MergeFrom() takes exactly one argument (0 given)").
  setdefault takes no keyword arguments, refuses a map of messages before it counts its
  arguments, and counts them as PyArg_UnpackTuple does. get parses its key and default as
  PyArg_ParseTupleAndKeywords("O|O") does. pop, popitem, keys, values and items are
  collections.abc's MutableMapping's and Mapping's, as upb's are, so their errors are Python's.
- Repeated containers: append, extend, remove and MergeFrom take one positional argument, insert
  and pop parse theirs as PyArg_ParseTuple's "nO" and "|n" (an index of its __index__, with
  "Python int too large to convert to C ssize_t"), and __deepcopy__ takes no keyword argument, as
  upb's. Before, append(value=...), insert(key=..., value=...), pop(key=...), remove(elem=...)
  and MergeFrom(other=...) were taken.
- A keyword argument of upb's METH_VARARGS methods (setdefault, insert, pop, __deepcopy__) is
  CPython's error of the method of its class when it is called of its object, and of the method
  only when CPython calls it as a bound method, as it calls each before Python 3.12 while a trace
  or a profile function is set (coverage's): the runtime's error names the method only then too.
- docs/compatibility.md: the arguments of the containers' methods, and why a dunder method called
  directly with the wrong arguments keeps CPython's error of a slot wrapper, and a keyword argument
  of a bound method object of a METH_VARARGS method names the class; a reflowed paragraph.

Co-authored-by: Isaac <no-reply@databricks.com>
@HyukjinKwon
HyukjinKwon force-pushed the SPARK-62056-pbmsg-container-argument-errors branch from 5827755 to a38cde9 Compare November 19, 2026 04:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants