Repository navigation
Conversation
dbtsai
marked this pull request as ready for review
November 19, 2026 05:47
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Part 1/6 of the planned split of #474, tracked by SPARK-61384.
Add an unpublished, opt-in
spark-connect-serverlibrary. Applications provide anEngineand a bound TCP listener; the frontend serves Spark Connect gRPC requests and delegates planning, execution, analysis and configuration to that engine.INTERNAL.allow_arrow_batch_chunkingreceive arbitrary byte chunks with chunk metadata.Bytes::sliceshares the original allocation; clients reassemble before Arrow decoding. Batches fitting the negotiated limit remain one chunk. Without opt-in, send the entire IPC batch unchanged.ServerOptions.max_arrow_chunk_sizewith the JVM default of 120,795,955 payload bytes (about 115.2 MiB; 90% of 128 MiB). Clients opting into chunking can request a payload from 1 KiB up to the configured maximum; other values use that maximum. Setting the server maximum to zero disables chunking. Producer batching belongs to the engine. Payload limits exclude protobuf overhead; schema and eager-command responses retain a separate 3 MiB envelope limit.serverfeature on the protocol crate, removing the manual RPC table and frontend build script. Share recursive schema transformations and test fixtures while keeping distinct lifecycle assertions. Accepted sockets use TCP_NODELAY.Why are the changes needed?
The combined server change in #474 is difficult to review as one patch. This establishes an independently buildable protocol/engine boundary targeting
master, without depending on later execution-adapter, JNI, licensing or distribution PRs.The framing follows JVM Spark v4.2.0. It removes the row splitter, trial/sizing encodes and oversized-IPC decode/re-encode path. Chunking works across compressed buffer boundaries and supports a row larger than an individual message payload.
Does this PR introduce any user-facing change?
Yes: an experimental library API for embedding applications. Existing client APIs and default workspace members are unchanged. The published protocol crate gains optional server bindings; enabling them does not enable tonic transport/router features. There is no execution backend, planner, standalone executable, Java/JNI integration or distribution artifact.
The supported subset delegates
ExecutePlan,AnalyzePlanandConfig, and implements operation/session release. Reattachment, artifacts,Interrupt, enriched errors, session cloning and status queries returnUNIMPLEMENTED.Use a loopback listener with trusted clients and engine outputs: this frontend provides no authentication, authorization or TLS. Engines must honor cancellation/cleanup contracts and supply valid IPC and truthful row counts. Batches in one result must use consistent emitted physical types/nullability; metadata may differ. Record-batch encoding materializes view arrays, while existing IPC keeps its physical types. Active sessions/operations have no admission limit or idle expiry. Shared chunks keep their complete backing allocation alive; larger batches can increase retained memory. Clients need receive limits allowing payload plus envelope overhead, or an explicit smaller preference.
How was this patch tested?
On macOS ARM64, clean candidate
56640d306ec510252d962b77396f7f64ac7484ae:serveroff/on on both toolchains. Frontend tests also check default Prost recursion behavior andprost/no-recursion-limit.wasm-udf.On a disposable merge with
masterataeabd2369dc748978339470eb09aec26e2ef2e1e, all 42 frontend tests passed on both Rust versions, all 814 client release unit tests passed, and client integration suites compiled with and withoutwasm-udf.A controlled local release benchmark compared the published frontend (
4a12199) with benchmark revisionc07e3c8: five alternating measurements after a full warmup, timing the complete ExecutePlan RPC through client receipt, chunk reassembly, Arrow decode and value/order checks. Inputs and the connection were prepared before timing. The final follow-up changes only documentation, comments and capitalization of an error unused by these fixtures; encoding and framing behavior are unchanged. These are injected-engine frontend measurements, not a JVM comparison. The 3 MiB columns below use an explicit client preference, not the server default.The single 80 MB record-batch fixture regressed. The new path encodes the entire batch before sending; the previous splitter sent smaller encoded batches. Pipelining and allocation behavior changed, but this benchmark does not isolate their contribution. Small-batch/wide-schema overhead improved after removing the second per-batch encode. Larger payloads also retain more memory; producer batch sizing remains an engine responsibility. Five local samples are indicative, not a throughput or production-latency guarantee. The pre-encoded IPC fixtures use 80 MB of deterministic high-entropy values. Producer IPC encoding and data generation are excluded; the repeatable fixture's owned-Vec clone, RPC, client decode and value checks are included equally for both revisions. Candidate byte equality is checked after timing and retains its received allocation; those checks can affect cache state. These comparisons cover the complete frontend changes and framing behavior, rather than isolating copy costs; they support no server CPU or peak-memory conclusion.
These pre-encoded IPC fixtures were slower in the candidate on loopback. The compressed fixture preserves 75,633,480 bytes exactly; the previous path decoded/re-encoded it as about 80 MB of uncompressed IPC. The new semantics avoid server decode/re-encode but move decompression to the client and change pipelining/memory behavior; this benchmark does not isolate the cause or predict a slower network. The change is a protocol/simplicity fix, not a uniform performance win.
Tests use deterministic injected engines; they establish frontend behavior, not real Spark query semantics. Hosted CI for this update is pending. No real-engine or JVM-versus-native performance claim is made for this frontend extraction.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (version unavailable in this session)
This pull request and its description were written by OpenAI Codex.