Repository navigation
Conversation
Keep NumPy subclass storage and missing schema aliases on the existing row fallback before Arrow encoding. Cover transformed values, conversion errors, and lazy module hooks. Co-authored-by: Isaac <no-reply@databricks.com>
dbtsai
marked this pull request as ready for review
November 7, 2026 05:24
…e#280 Co-authored-by: Isaac <no-reply@databricks.com>
…declines Since SPARK-61079 such frames convert as PySpark's Arrow conversion does, which reads the configs and the raw storage of a NumPy subclass without calling its astype, rather than the row path. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Preserve the existing row conversion of customized pandas float64 input in the fast path added by #207 (SPARK-60727).
copy=Falsecan retain an ndarray subclass inside pandas' concrete NumPy wrapper. The column path skips that subclass'sastypehook, which can change values or suppress an existing conversion error. Require exact ndarray storage before using this path.__getattr__before row conversion.Add three mock-service regression cases for transformed values, conversion errors, and a missing alias with a raising lazy hook. Ordinary ndarray input keeps the fast path; subclasses such as
np.memmaptake the row path.Why are the changes needed?
These inputs are customized cases that #207 intended to keep on the existing row path. Both guard issues remain on current master. Official Spark 4.3 also bypasses the ndarray hook; this change preserves the Rust client's prior custom-input behavior rather than claiming an official parity fix.
Does this PR introduce any user-facing change?
It restores the prior values and errors for the affected customized input. Public signatures and inline transport policy are unchanged.
How was this patch tested?
git diff --checkpass.Draft: required validation remains pending. The current environment has no usable Cargo/rustc toolchain, matching native extension, or protoc. The new public-API test attempt is blocked during collection; Rust compile/fmt/Clippy, complete native suites, API/official parity, and live Spark 4.3 checks have not passed for this candidate. Historical maintainer results are not claimed as validation of this follow-up.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex; reviewed with installed local Claude Code.
This pull request and its description were written by Isaac.