Skip to content

[SPARK-61384][CONNECT] Add an experimental Rust Spark Connect frontend (1/6) - #870

Open
dbtsai wants to merge 17 commits into
apache:masterfrom
dbtsai:codex/pr474-protocol-frontend
Open

dbtsai wants to merge 17 commits into
apache:masterfrom
dbtsai:codex/pr474-protocol-frontend

Conversation

@dbtsai

@dbtsai dbtsai commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Part 1/6 of the planned split of #474, tracked by SPARK-61384.

Add an unpublished, opt-in spark-connect-server library. Applications provide an Engine and a bound TCP listener; the frontend serves Spark Connect gRPC requests and delegates planning, execution, analysis and configuration to that engine.

  • Own protocol sessions, canonical UUIDs, operation IDs, framing, backpressure and cancellation on release, disconnect, shutdown or dropping the serving future. Asynchronous engine panics become INTERNAL.
  • Arrow responses follow JVM Spark v4.2.0 framing, with no default 3 MiB Arrow split. Encode each Arrow record batch once, or forward producer IPC unchanged, including compression. Only clients opting into allow_arrow_batch_chunking receive arbitrary byte chunks with chunk metadata. Bytes::slice shares the original allocation; clients reassemble before Arrow decoding. Batches fitting the negotiated limit remain one chunk. Without opt-in, send the entire IPC batch unchanged.
  • Expose ServerOptions.max_arrow_chunk_size with the JVM default of 120,795,955 payload bytes (about 115.2 MiB; 90% of 128 MiB). Clients opting into chunking can request a payload from 1 KiB up to the configured maximum; other values use that maximum. Setting the server maximum to zero disables chunking. Producer batching belongs to the engine. Payload limits exclude protobuf overhead; schema and eager-command responses retain a separate 3 MiB envelope limit.
  • Generate server bindings with a default-off server feature on the protocol crate, removing the manual RPC table and frontend build script. Share recursive schema transformations and test fixtures while keeping distinct lifecycle assertions. Accepted sockets use TCP_NODELAY.

Why are the changes needed?

The combined server change in #474 is difficult to review as one patch. This establishes an independently buildable protocol/engine boundary targeting master, without depending on later execution-adapter, JNI, licensing or distribution PRs.

The framing follows JVM Spark v4.2.0. It removes the row splitter, trial/sizing encodes and oversized-IPC decode/re-encode path. Chunking works across compressed buffer boundaries and supports a row larger than an individual message payload.

Does this PR introduce any user-facing change?

Yes: an experimental library API for embedding applications. Existing client APIs and default workspace members are unchanged. The published protocol crate gains optional server bindings; enabling them does not enable tonic transport/router features. There is no execution backend, planner, standalone executable, Java/JNI integration or distribution artifact.

The supported subset delegates ExecutePlan, AnalyzePlan and Config, and implements operation/session release. Reattachment, artifacts, Interrupt, enriched errors, session cloning and status queries return UNIMPLEMENTED.

Use a loopback listener with trusted clients and engine outputs: this frontend provides no authentication, authorization or TLS. Engines must honor cancellation/cleanup contracts and supply valid IPC and truthful row counts. Batches in one result must use consistent emitted physical types/nullability; metadata may differ. Record-batch encoding materializes view arrays, while existing IPC keeps its physical types. Active sessions/operations have no admission limit or idle expiry. Shared chunks keep their complete backing allocation alive; larger batches can increase retained memory. Clients need receive limits allowing payload plus envelope overhead, or an explicit smaller preference.

How was this patch tested?

On macOS ARM64, clean candidate 56640d306ec510252d962b77396f7f64ac7484ae:

  • All 42 frontend tests passed on Rust 1.100.0 and MSRV 1.88.0, including real gRPC framing/reassembly, exact plain/ZSTD/LZ4 IPC byte preservation, chunk preferences/configuration, oversized single rows, sliced buffers, schema consistency, canonical UUIDs, panic translation and lifecycle cancellation.
  • Protocol crate checks passed independently with server off/on on both toolchains. Frontend tests also check default Prost recursion behavior and prost/no-recursion-limit.
  • Workspace formatting and strict Clippy, strict MSRV frontend Clippy, strict frontend rustdoc and generated license freshness passed. Proto documentation builds with inherited generated-comment warnings; a published-revision control reproduced those warnings.
  • All 809 client release unit tests and 18 core session/artifact integration tests passed; client integration suites compiled with and without wasm-udf.

On a disposable merge with master at aeabd2369dc748978339470eb09aec26e2ef2e1e, all 42 frontend tests passed on both Rust versions, all 814 client release unit tests passed, and client integration suites compiled with and without wasm-udf.

A controlled local release benchmark compared the published frontend (4a12199) with benchmark revision c07e3c8: five alternating measurements after a full warmup, timing the complete ExecutePlan RPC through client receipt, chunk reassembly, Arrow decode and value/order checks. Inputs and the connection were prepared before timing. The final follow-up changes only documentation, comments and capitalization of an error unused by these fixtures; encoding and framing behavior are unchanged. These are injected-engine frontend measurements, not a JVM comparison. The 3 MiB columns below use an explicit client preference, not the server default.

Arrow record-batch fixture Published median Candidate, JVM chunk default Candidate, 3 MiB client preference
20,000 batches × 100 rows 586.53 ms 295.13 ms 294.95 ms
50 batches × 2,000 columns × 50 rows 104.89 ms 44.80 ms 44.57 ms
One 80 MB batch 26.46 ms 44.84 ms 35.68 ms
20 slices × 1,000 rows retaining 80 MB 0.83 ms 0.50 ms 0.50 ms

The single 80 MB record-batch fixture regressed. The new path encodes the entire batch before sending; the previous splitter sent smaller encoded batches. Pipelining and allocation behavior changed, but this benchmark does not isolate their contribution. Small-batch/wide-schema overhead improved after removing the second per-batch encode. Larger payloads also retain more memory; producer batch sizing remains an engine responsibility. Five local samples are indicative, not a throughput or production-latency guarantee. The pre-encoded IPC fixtures use 80 MB of deterministic high-entropy values. Producer IPC encoding and data generation are excluded; the repeatable fixture's owned-Vec clone, RPC, client decode and value checks are included equally for both revisions. Candidate byte equality is checked after timing and retains its received allocation; those checks can affect cache state. These comparisons cover the complete frontend changes and framing behavior, rather than isolating copy costs; they support no server CPU or peak-memory conclusion.

Pre-encoded fixture Published median Candidate, JVM chunk default Candidate, 3 MiB client preference
Uncompressed IPC (80.00 MB) 26.60 ms 43.11 ms 36.84 ms
ZSTD IPC (75.63 MB from 80 MB values) 104.41 ms 155.73 ms 112.74 ms

These pre-encoded IPC fixtures were slower in the candidate on loopback. The compressed fixture preserves 75,633,480 bytes exactly; the previous path decoded/re-encoded it as about 80 MB of uncompressed IPC. The new semantics avoid server decode/re-encode but move decompression to the client and change pipelining/memory behavior; this benchmark does not isolate the cause or predict a slower network. The change is a protocol/simplicity fix, not a uniform performance win.

Tests use deterministic injected engines; they establish frontend behavior, not real Spark query semantics. Hosted CI for this update is pending. No real-engine or JVM-versus-native performance claim is made for this frontend extraction.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (version unavailable in this session)

This pull request and its description were written by OpenAI Codex.

@dbtsai
dbtsai marked this pull request as ready for review November 19, 2026 05:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant