Skip to content

[SPARK-61384][CONNECT] Add embedded Rust Spark Connect JVM server - #474

Open
dbtsai wants to merge 18 commits into
apache:masterfrom
dbtsai:codex/jvm-spark-connect-server-wip
Open

dbtsai wants to merge 18 commits into
apache:masterfrom
dbtsai:codex/jvm-spark-connect-server-wip

Conversation

@dbtsai

@dbtsai dbtsai commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Review split and dependency

This PR is rebased on the standalone Rust frontend in #870, at 56640d306ec510252d962b77396f7f64ac7484ae. Review the remaining native layer with the parent-to-head comparison. Both PRs currently target Apache master, so GitHub's Files changed tab includes #870 until it lands. This PR depends on #870 and preserves its frontend implementation unchanged.

The planned follow-up split remains: Java Spark adapter and executor IPC contract; JNI integration and embedded launcher; native dependency licensing; platform JAR packaging and loading; and combined/universal distributions. Lifecycle and cancellation checks stay with the implementation they protect. Performance optimization and producer/off-heap API redesign remain separate follow-up work.

What changes were proposed in this pull request?

Add an experimental Spark JVM execution engine for #870's Rust Spark Connect frontend. Rust owns the gRPC listener, protocol sessions, operations and response streams. A Java adapter invokes Spark planning and execution APIs in the same JVM through JNI. The server and JNI crates are opt-in and unpublished.

Executors produce complete Arrow IPC batches using Spark's existing ArrowConverters.toBatchWithSchemaIterator. The driver fetches and materializes one partition at a time. JNI copies each Java byte[] directly into a Rust-owned Vec; the intermediate C allocation and full-payload copy have been removed. Moving that allocation into protobuf Bytes and slicing negotiated transport chunks do not copy the payload. Standard Prost encoding still copies it into the outgoing gRPC buffer. The Java producer API is unchanged.

The inherited frontend follows JVM result framing: with client opt-in, it slices serialized IPC bytes and sends chunk indexes/counts for client reassembly. Without opt-in, it sends the whole IPC batch. Arrow responses have no unconditional 3 MiB split or decode/re-encode path. The frontend defaults to about 115.2 MiB maximum chunk payloads, accepts smaller valid client preferences, and permits disabling chunking with a zero configured maximum. These are transport settings: this adapter retains its separate approximate 2 MiB producer target and post-conversion 64 MiB JNI IPC safety bound. The embedded Java launcher currently uses the frontend defaults. The frontend's separate schema/eager-command response bound is 3 MiB.

Deferred handle cleanup retains ownership and admission permits until teardown succeeds. Cancellation failures retain ownership for retry; config reads apply the host redaction pattern; and the launcher detects unexpected listener termination. Native libraries can be packaged in Linux/macOS x86-64/ARM64 platform JARs, a macOS universal JAR, or an all-platform JAR alongside the Java adapter. The loader checks artifact/ABI versions and checksums, and generated dependency licenses accompany native artifacts. CI builds platform artifacts and exercises a pinned Spark assembly.

Why are the changes needed?

This prototype explores reusing the Rust protocol and transport stack as an embedded frontend while Spark remains the execution engine. It provides an implementation for discussing that boundary and its lifecycle.

JIRA: https://issues.apache.org/jira/browse/SPARK-61384

Does this PR introduce any user-facing change?

Yes. Developers can explicitly build and launch the embedded prototype. Existing default workspace members and client APIs are unchanged.

The prototype supports trusted local workloads and a subset of batch queries, result types and RPCs. The named-table restriction applies to protobuf READ relations; SQL and named-table resolution follow the host session semantics, including access to files and reflective builtins. Unsupported operations return UNIMPLEMENTED. Authentication, authorization, TLS, full protocol parity, Windows and native library unloading are outside its scope. Session/task limits and the JNI payload check do not provide complete memory or CPU quotas.

The Java adapter must be available to executors. Partition fetching is sequential, and each fetched partition is materialized on the driver; large partitions can exhaust memory or hit spark.driver.maxResultSize. Arrow conversion inherits Spark's allocator and task cleanup behavior. Nondefault IPC compression interoperability has not been validated for this native adapter. The README documents supported operations and lifecycle contracts.

The adapter depends on internal Spark APIs. Local integration validation uses Spark 4.2.0-SNAPSHOT revision 250ccca3f2b5f043e20b415adf17bda5d202c60f, Scala 2.13 and JDK 21 on macOS ARM64. The unmodified pinned assembly is used for real JNI smoke; broader acceptance uses a separately repaired Arrow-input runtime. These checks do not establish compatibility with every Spark 4.2 build or Spark 4.3/4.4 assemblies.

How was this patch tested?

Fresh local checks for the rebased native layer passed on macOS ARM64 with JDK 21:

  • Rust 1.100 and MSRV 1.88: 52 JNI tests, 42 frontend tests and strict native Clippy. Workspace Clippy and formatting passed on Rust 1.100.
  • 809 client release unit tests; all client integration tests compiled with and without wasm-udf.
  • C bridge tests passed normally and with AddressSanitizer/UndefinedBehaviorSanitizer.
  • 35 license-tool tests, generated native/Python license freshness, 26 acceptance-runner guards and 33 packaging tests passed.
  • Java adapter/test compilation and real JNI start/query/stop passed under -Xcheck:jni against the unmodified pinned Spark assembly. New checks reassemble negotiated 1 KiB chunks and verify row values/order/offsets, then verify a complete IPC batch above 3 MiB is delivered without client chunking opt-in.
  • Live engine, IPC, separate-executor/Kryo, driver result-size, cancellation, lifecycle and cleanup-failure checks passed with the separately repaired Arrow-input runtime.
  • Platform JAR loader checks and actual native loading passed locally. Packaged acceptance passed 40 recorded checks (20 functional scenarios and 20 session-cleanup confirmations) plus the Rust client example; shutdown, listener closure, JNI diagnostics and extracted-library cleanup were checked.

Hosted CI has not yet validated this rebased head. Earlier multi-platform/universal-JAR CI results belong to the original implementation and are not current-head results. This local run does not validate nondefault IPC codecs, other platforms or other Spark versions.

Historical performance measurement (November 17, 2026; native source 23b2eb6)

These measurements predate this rebase and #870's frontend framing; they do not measure the rebased revision.

Four matched process pairs compared the OSS JVM Connect server with this embedded Rust implementation on the same Linux ARM64 host and unmodified Spark assembly (250ccca3f2b5f043e20b415adf17bda5d202c60f). Pair order was AB, BA, BA, AB, with fresh server and official Python client processes. Both used local[2], a 1 GiB heap, two input/shuffle partitions, UTC, 10,000 Arrow rows per batch and IPC codec none. The runtime was JDK 21.0.13, PySpark client 4.2.0, PyArrow 26.0.0 and gRPC 1.84.0. Server CPUs were 2-5 and client launch CPUs 6-7; affinity was nonexclusive.

Latency columns are geometric means of four fork medians. Changes are the geometric mean of the four matched native/JVM ratios; positive values mean higher native latency.

Workload OSS JVM (ms) Native (ms) Native latency change
Config Get 0.837 0.429 -48.8%
Range schema analysis 2.121 2.327 +9.7%
Tiny SQL and collect 33.202 47.829 +44.1%
Range/filter aggregate and collect 49.855 78.712 +57.9%
Join aggregate and collect 72.780 92.920 +27.7%
Raw Arrow complete receipt 246.370 382.529 +55.3%
Raw Arrow receipt and table decode 260.639 384.016 +47.3%
Public DataFrame.toArrow() 428.790 547.088 +27.6%
Concurrent Config per-call latency 0.641 0.615 -4.1%

Public toArrow() was slower in all four native pairs, ranging from 24.0% to 32.6% higher latency. It includes official-client response handling, IPC decode, table materialization and schema cast. Raw receipt drains all protobuf responses through completion, with IPC decode outside timing. Both servers returned 8 million int64 rows, 800 IPC batches and 65,248,000 IPC bytes per raw Arrow action. These separately sampled windows cannot be subtracted to estimate decode cost.

Concurrent Config was slower in Rust in three of four pairs despite its lower geometric mean of fork medians; whole-round throughput was 23.3% lower. These results establish neither a general native speedup nor improved concurrent capacity.

All 13,152 action records passed their recorded checks, including 9,464 measured records and 3,688 warmups. Explicit session/client/server cleanup and listener closure passed. The server README records the complete workload table, sampling settings and limits. The benchmark uses local executors and loopback transport on one host; within-process calls are correlated. It establishes no significance, distributed-network, wide-row, compressed-IPC or retained-memory result. The current implementation is not presented as an Arrow performance improvement over JVM Connect.

A separate November 18 native-before/after experiment used native revision 23b2eb6538b355987341b385cdfccc3222669e4b on both sides and tested an unmerged Spark producer output-sizing patch against a matched recompilation control. Its two matched pairs in AB, BA order retained 1,500 actions (1,040 measured, 460 warmups): raw Arrow receipt latency was 9.43% lower, while public toArrow() latency was 1.31% higher. Tiny SQL varied widely (pair ratios 0.9253 and 1.8582). That run had no JVM reference, demonstrated no consistent public-client gain and did not measure GC. It cannot be compared directly with the November 17 figures. The Spark patch is not included in this PR. Performance optimization and producer/off-heap API redesign remain follow-up work.

Was this patch authored or co-authored using generative AI tooling?

This pull request and its description were written by OpenAI Codex.

@dbtsai dbtsai changed the title [WIP][SPARK-61384][CONNECT] Add embedded Rust Spark Connect JVM server [SPARK-61384][CONNECT] Add embedded Rust Spark Connect JVM server Nov 13, 2026
@dbtsai
dbtsai force-pushed the codex/jvm-spark-connect-server-wip branch from 580577c to 821c9a2 Compare November 14, 2026 08:46
@dbtsai
dbtsai marked this pull request as ready for review November 15, 2026 02:36
@dbtsai
dbtsai requested review from HyukjinKwon, viirya and zhengruifeng and removed request for HyukjinKwon November 15, 2026 02:36

@HyukjinKwon HyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran an AI-assisted review (Claude Code /code-review, max effort) of aa4e45c. Its verification pass checked each finding against the code, but the scenarios were not reproduced by running them. It found 15 correctness issues: 11 are inline comments, and 4 are below, for a file GitHub shows no diff for.

The first three inline comments look security-relevant:

  • The UDF guard can be bypassed through typed aggregates.
  • Config reads return values that Spark Connect redacts.
  • The SQL relation can read local files and call Java methods, despite the named-table-only READ restriction.

crates/spark-connect-server/jni/src/engine.rs

GitHub renders no diff for this new file, so these can't be inline comments.

engine.rs line 668

A failed cleanup-lane spawn is cached for the life of the process. If the first std::thread::Builder::spawn for a lane fails transiently (EAGAIN under a thread limit or memory pressure), cleanup_sender's OnceLock stores the Err(String) permanently. From then on, every Handle::drop and Task::cancel takes Err(_) => cleanup(item) (line 742) and runs the JNI free or cancel inline on the dropping thread.

In execute_plan, that thread is one of the runtime's two async workers (worker_threads(2)). The worker then attaches to the JVM and blocks in Query.close, Session.close or Task.cancel, stalling gRPC I/O for the whole server. Retrying the spawn, or not caching the error, would avoid this.

engine.rs line 1345

silent_config_set stops at the first per-key error it doesn't list. Only InvalidArgument, NotFound, AlreadyExists and OutOfRange are turned into warnings.

Take a silent Set of [a, b, c] where RuntimeConfig.set(b) throws something else, such as an internal-error SparkThrowable or an IllegalStateException from a setter. applyConfig/engineError map it to INTERNAL, Err(error) => return Err(error) fails the RPC, and c is never applied. Spark's SparkConnectConfigHandler.handleSet instead adds a warning for any Throwable and goes on to the next key.

engine.rs line 696

A panic on a cleanup lane drops its queued items, and wait_drained then never returns. eprintln! panics on a stderr write error such as EPIPE (std ignores only EBADF). That happens, for example, when spark-submit's output is piped to a reader that has exited.

If a Java close fails in that state, the eprintln! in cleanup_with panics and the lane thread unwinds. Its mpsc receiver then drops the queued Cleanup::Free items without running them. counts.handles never reaches 0, so the listener thread's wait_drained() never returns, and every NativeEngineServer.close() reports 'JNI stop is incomplete' until the JVM exits.

Writing to stderr with writeln! and ignoring the error, or wrapping each item in catch_unwind, would keep the lane alive.

engine.rs line 420

A failed detach drops the only pointer to the attachment. Suppose spark_connect_engine_thread_detach returns -1 during on_thread_stop or thread-local teardown. C keeps the EngineThread and its context reference, as bridge.h says: 'A failed detach retains the guard so its owner can retry'. But WorkerAttachment::drop discards the pointer and never decrements worker_attachments. As a result:

  • every later stop_server retry fails verify_workers_detached ('1 JNI worker attachments remain after runtime stop');
  • free_checked refuses the factory, because context->references != 1;
  • release_factory can't be reached until the JVM exits.

Native license hashes

The native license assets bind whole-file SHA-256 hashes of Cargo.lock, the root Cargo.toml, LICENSE, NOTICE and both generator scripts (scripts/gen_native_binary_licenses.py:49-61). So any unrelated lockfile change fails native CI and packaging until about 4k generated lines are regenerated. Hashing only what the licenses depend on, such as the resolved dependency list, would avoid that churn.

}
}
String type = message.getDescriptorForType().getFullName();
if (type.endsWith("CommonInlineUserDefinedFunction") || type.endsWith("CommonInlineUserDefinedTableFunction")) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The UDF guard can be bypassed through typed aggregate expressions. validateRelations rejects UDFs only when the protobuf descriptor name ends in CommonInlineUserDefinedFunction or CommonInlineUserDefinedTableFunction. A ScalarScalaUDF nested in Expression.typed_aggregate_expression, inside an allowed AGGREGATE or PROJECT relation, matches neither name and passes.

That is what the Scala client sends for ds.select(aggregator.toColumn). The call path is SparkConnectPlanner.transformTypedAggregateExpression -> unpackScalaUDF -> UdfSerialization.deserialize(payload), which runs ObjectInputStream over client-supplied bytes with no class allowlist.

The guard says user-defined functions are unsupported in this prototype. Should it reject ScalarScalaUDF and TypedAggregateExpression explicitly, or allowlist expression types instead of denylisting names?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. The recursive guard now rejects both TypedAggregateExpression and ScalarScalaUDF, including when nested in an otherwise supported relation. Regression tests use empty/default protobufs and a nested Project and verify rejection before planning.

List<Pair> output = new ArrayList<>();
switch (opcode) {
case 1:
for (Pair pair : input) output.add(new Pair(pair.key, spark.conf().get(pair.key)));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config reads return secrets that Spark Connect redacts. Get, GetOption, GetWithDefault and GetAll (opcodes 1-4) return raw values with no spark.redaction.regex filtering. For example, with spark.hadoop.fs.s3a.secret.key=XYZ set:

  • spark.conf.get("spark.hadoop.fs.s3a.secret.key") returns XYZ.
  • GetAll with an empty or spark.hadoop. prefix lists it.

Spark 4.3's SparkConnectConfigHandler handles these keys differently (isRedacted, SparkConnectConfigHandler.scala:109-232):

  • Get fails with SQL_CONF_NOT_FOUND.
  • GetOption and GetWithDefault return None or the default.
  • GetAll drops the entry.

No other layer redacts them either: not the Java adapter, bridge.c, jni/src/engine.rs or service.rs.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. Config reads now use the hosting SparkContext's cached redaction pattern and check both the full key and value before stripping a GetAll prefix. Get returns the same error code and message as the same unset key, GetOption omits it, GetWithDefault returns the caller's default, and GetAll omits the entry. Tests cover all four reads and ensure a session-level pattern override cannot weaken the host pattern. This matches the newer Spark 4.3 handler; the pinned 4.2 snapshot used by the smoke tests did not yet redact these reads.

if (!RELATIONS.contains(kind)) {
throw new EngineException(12, "unsupported JVM engine relation: " + kind);
}
if (relation.hasRead() && !relation.getRead().hasNamedTable()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SQL relation bypasses the named-table-only READ restriction. The restriction is checked only on Relation.read. But SQL is in RELATIONS (line 112), and rejectCommands lets a SELECT through, so:

  • spark.sql("SELECT * FROM parquet.`/home/u/private.parquet`") reads a local file through ResolveSQLOnFile. So do json., csv. and text. paths.
  • SELECT reflect('java.lang.System', 'getenv', 'X') calls static Java methods.

The README says READ is restricted to named tables, and lists spark.read.parquet() as unsupported. If that restriction is meant to be a boundary, the SQL path probably needs spark.sql.runSQLOnFiles=false and to reject reflect/java_method. Otherwise the README should say the restriction isn't enforced for SQL.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified in dce536e8d00a747b8f23df0e663fbcbca962c3cc. The named-table restriction applies to the protobuf Relation.read form. SQL and named-table resolution use the hosting Spark session's semantics and can access files or reflective builtins. The README and security documentation now state that explicitly. This prototype accepts trusted local queries and does not provide an access-isolation boundary.

Comment thread Cargo.toml
"crates/pyspark-rs",
"examples",
"crates/spark-connect-server",
"crates/spark-connect-server/jni",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the JNI crate in the workspace, every --workspace command needs a JDK and a C compiler. jni/build.rs panics without JAVA_HOME, and on Windows ('unsupported JNI target'). So on machines that only have Rust and the wasm target, these now fail:

  • cargo test --workspace;
  • the pre-push cargo clippy --workspace --all-targets -- -D warnings that AGENTS.md requires;
  • rust-analyzer's build-script runs.

That breaks this file's stated goal that cargo test --workspace stays green without extra toolchains, and lint.yml had to add a JDK. The --exclude spark-connect-jni workaround is documented, but a cfg or feature gate would keep the default commands working. A build.rs that silently skipped the shim would only move the failure to link time.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The toolchain requirement is intentional for --workspace, and is documented immediately below this comment: workspace commands include the JNI host and require macOS/Linux, a C compiler and a JDK. The earlier “without it” refers specifically to the optional wasm target. Default workspace members remain unchanged and exclude the server/JNI crates. I kept this policy rather than adding a separate JNI feature gate; client-only workspace commands can use --exclude spark-connect-jni.

if (error->data == NULL) set_error(error, "native Spark Connect operation failed");
jstring message = utf8_string(env, error->data, error->len, NULL);
jclass type = (*env)->FindClass(env, "java/lang/IllegalStateException");
if (!(*env)->ExceptionCheck(env) && message != NULL && type != NULL) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

throw_error can return without a pending Java exception. If building the message fails, the message != NULL guard skips Throw. That happens when utf8_string returns NULL after exception_error clears its OutOfMemoryError.

For example, nativeStop can fail with 'JNI stop is incomplete' after the 10 s deadline while the heap is low. It then returns normally, and NativeEngineServer.close() sets handle = 0 even though Rust still holds the server and factory, so the close can never be retried. nativeAddress likewise returns NULL with no exception, and address() throws a NullPointerException.

Throwing a preallocated or fixed-message error when the message can't be built would keep these failures visible.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. throw_error preserves an existing pending exception and uses a fixed-message ThrowNew fallback if constructing the detailed message fails. Allocation failures in the fallback also leave their exception pending. The C mock tests exercise these paths, including message-allocation failures, and pass normally and under ASan/UBSan.

if (bytes.length > MAX_IPC_BYTES) throw new EngineException(8, "config fields exceed 64 MiB");
try (DataInputStream input = new DataInputStream(new ByteArrayInputStream(bytes))) {
int count = input.readInt();
if (count < 0 || count > MAX_CONFIG_FIELDS || count > input.available() / 8) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config requests with more than 10,000 keys are rejected. decodePairs rejects more than MAX_CONFIG_FIELDS (10000) pairs with EngineException(3, "invalid config field count"), which the client sees as INVALID_ARGUMENT. Rust's encode_pairs forwards any count, and Spark Connect accepts the same request, which is far below the 64 MiB request limit. GetAll output isn't capped either, so a client can't send back the key list it just received.

The limit should either match on both sides and be documented in the README, or be raised or removed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. The arbitrary 10,000-pair cap is removed. The 64 MiB encoded-payload bound and count-versus-remaining-bytes checks remain. Regression tests accept 10,001 well-formed pairs and reject malformed counts.

JavaScope scope;
if (engine_scope(task, &scope, NULL) != 0) return;
(*scope.env)->CallVoidMethod(scope.env, task->object, task->api->cancel);
(void) engine_exception(scope.env, task->api, NULL);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spark_connect_engine_cancel silently drops cancel failures. It returns void, so a failed scope entry and an exception from Java Task.cancel() are both discarded. Rust's one-shot Task::cancel then treats the cancel as delivered while the job group keeps running.

For example, if the host stopped the shared SparkContext before the server drained, cancelJobGroupAndFutureJobs throws an IllegalStateException from assertNotStopped:

  • It is stored in cancellationFailure and cleared by (void) engine_exception(...).
  • Every later Task.close() rethrows it.
  • The Rust free stays in failed_cleanup, and stop_server reports incomplete cleanup indefinitely.

Returning a status, as the other entry points do, would let Rust handle the failure.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. Cancellation now returns status and an error buffer across the C ABI. Both Rust callers retain failed cancellation ownership and admission permits for an explicit close retry instead of marking delivery successful. C and Rust tests cover scope-entry and Java failures. A partially failed Java cancellation remains sticky: retrying delivery does not claim to undo a terminal host failure.

return;
}
let data = libc::malloc(bytes.len()).cast::<u8>();
if data.is_null() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

write_buffer reports success with an empty buffer when malloc fails. It sets *out to the default and returns. spark_connect_address then maps Ok(()) to 0, and bridge.c builds an empty byte[]. So NativeEngineServer.address() returns "", and EmbeddedSparkEngine prints 'listening on ' instead of raising an error. Returning an error from write_buffer when allocation fails would surface it like the other failures.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. write_buffer now returns a Result, and the address entry point propagates allocation failure instead of returning success with an empty address. An injected-allocation test exercises the helper's failure path.

JAVAC="${JAVA_HOME:+$JAVA_HOME/bin/}javac"
JAR="${JAVA_HOME:+$JAVA_HOME/bin/}jar"
BUILD_DIR="$PROJECT_DIR/.build/jvm"
rm -rf -- "$BUILD_DIR/classes" "$BUILD_DIR/test-classes"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concurrent script runs race on the shared build output. build.sh always deletes and rewrites .build/jvm/classes and the bridge jar in place, and run-engine.sh, test-engine.sh and test-jni-smoke.sh all call it.

Say run-engine.sh starts while test-engine.sh is compiling. The second build's rm -rf -- "$BUILD_DIR/classes" deletes classes the first javac is compiling against. A spark-submit that starts in that window can find the jar missing or truncated, which shows up as a spurious ClassNotFound or corrupt-zip error.

A lock (for example flock), or a per-run output directory with an atomic rename, would avoid this.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. Each Java build now uses a private staging directory and atomically publishes complete, self-contained JARs after compilation succeeds. Adapter packaging validates the exact temporary copy before replacing the published JAR, and writes checksum files atomically. Three concurrent builds passed while a reader successfully opened the published JAR 718 times with no errors; regressions cover source replacement during copying and interrupted JAR/checksum writes.

System.out.println("Embedded Spark Connect JVM PID " + ProcessHandle.current().pid()
+ "; Java Connect listener absent");
launched.set(true);
stopped.await();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main is never told when the native listener exits on its own. It blocks on a latch that only the shutdown path counts down. Suppose the tonic server future returns an error, or block_on panics ('Rust listener panicked' in start_host, jni/src/lib.rs). The listener thread runs shutdown and wait_drained, then sends to done, which only stop_server reads. main stays in stopped.await(), holding the SparkSession with nothing bound to the port, until an external signal arrives.

This is rare, since tonic 0.14 ignores accept errors, but when it happens the process looks healthy while serving nothing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. The native host exposes listener-running state independently of cleanup drain, and the launcher polls it so an unexpected listener exit triggers teardown and an error. A close retry is allowed only after confirmed drain; incomplete or unknown cleanup retains the native handle and prevents stopping shared Spark resources. A Rust test exercises the real listener exiting while retained cleanup blocks drain; stock JNI smoke calls both health methods with a live handle. Java lifecycle tests cover the failure and cleanup ordering.

@dbtsai

dbtsai commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

Addressed the four Rust lifecycle/config findings from this review in dce536e8d00a747b8f23df0e663fbcbca962c3cc:

  • Cleanup lanes cache only successful senders and retry failed/disconnected lanes. A failed spawn retains ownership and permits instead of running blocking JNI cleanup on the async caller.
  • Silent Config Set receives typed per-key setter warnings from Java after successful rollback, including INTERNAL setter failures, and continues with later keys. Cancellation, interruption, admission, JNI and rollback failures still fail the operation.
  • Cleanup records retained ownership before diagnostics and uses fallible stderr writes. A broken-writer test covers the reported logging failure.
  • Detach gets one bounded retry on its original OS thread. A persistent failure retains the C guard/context and attachment count and reports failed shutdown. A later close on another thread cannot safely perform that detach; the docs now state this limit.

I kept the native-license source fingerprints. Packaging can validate assets without running Cargo, so a hash of the generated dependency inventory alone would not detect a stale inventory after dependency or feature changes. An unrelated manifest-comment probe required regeneration but changed only one fingerprint line; the generated LICENSE and NOTICE text stayed unchanged. A semantic dependency projection could reduce that conservative invalidation in a follow-up.

The 49 JNI tests, 29 frontend tests and strict native Clippy passed on Rust 1.100 and MSRV 1.88. Workspace Clippy, 691 client unit tests, C tests with ASan/UBSan, Java lifecycle/config tests, concurrent builds, packaging and license checks also passed. Real JNI start/query/stop passed against the unmodified pinned Spark 4.2 snapshot. The full packaged PySpark/raw-RPC and Rust-client acceptance run also passed using the separately repaired Arrow-input runtime described in the README. Cross-platform hosted checks for this head remain pending.

@HyukjinKwon HyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

The earlier review comments are addressed at this head. The UDF guard now rejects typed aggregates and ScalarScalaUDF. Config reads apply the host redaction pattern. The SQL access limit is documented. The tombstone, throw_error, cancel-status, write_buffer, build-staging and listener-exit fixes are in place, and the four engine.rs lifecycle and config findings are fixed. This review found only non-blocking issues.

Two validators added by this PR have no test that would fail if they regressed: the named-table-only READ check in the Java adapter, and the IPC branch of the frontend's per-batch schema check, which is the branch every JNI result takes. The UDTF clause of the same relation guard needs no separate test: a UDTF can appear in a plan only as a relation kind, and the relation allowlist already rejects it before that clause runs.

The rest are text corrections. The server README says each batch fetch is a separate SQL execution, but the code starts one per partition fetch. Three comments describe something other than the code: the cancellation flag's row-iteration rationale, the u32 pair count in the config ABI, and the AnalysisGate barrier's scope. The tombstone comment sits on ConnectService instead of SessionRegistry.entries. The new AGENTS.md pre-push rule copies the native-license input list instead of pointing at the inventory the generator records.

Findings

8 total: 0 P0, 0 P1, 0 P2, 8 P3.

Nit (P3)

  • No test covers the named-table-only READ guard — crates/spark-connect-server/jvm/src/main/java/org/apache/spark/sql/connect/nativebridge/engine/SparkJvmEngine.java:924 — see inline.
  • The IPC arm of the per-batch schema check has no test — crates/spark-connect-server/src/service.rs:621 — see inline.
  • README says each batch fetch is a separate SQL execution; it is each partition fetch — crates/spark-connect-server/README.md:289 — see inline.
  • The Task.cancelled comment cites a row loop that no longer exists — crates/spark-connect-server/jvm/src/main/java/org/apache/spark/sql/connect/nativebridge/engine/SparkJvmEngine.java:260 — see inline.
  • Session.config Javadoc calls the pair count u32; decodePairs takes a non-negative i32 — crates/spark-connect-server/jvm/src/main/java/org/apache/spark/sql/connect/nativebridge/engine/SparkJvmEngine.java:509 — see inline.
  • Tombstone comment is attached to ConnectService, not to SessionRegistry.entries — crates/spark-connect-server/src/service.rs:157 — see inline.
  • AnalysisGate Javadoc overstates which threads block — crates/spark-connect-server/jvm/src/test/java/org/apache/spark/sql/connect/nativebridge/engine/EngineSmokeTest.java:743 — see inline.
  • AGENTS.md hardcodes the native-license input list that the generator computes — AGENTS.md:274 — see inline.

Files not given in full to design and detailed review

This change is too large to give design and detailed review every file's diff. They judged each group of mechanical edits from its shape and samples, and read the remaining files only where a judgment called for it; every file was still scanned individually.

  • crates/spark-connect-server/**/*.java: 10 of 12 files not given in full.
  • crates/spark-connect-server/**/*.rs: 6 of 11 files not given in full.

if (!RELATIONS.contains(kind)) {
throw new EngineException(12, "unsupported JVM engine relation: " + kind);
}
if (relation.hasRead() && !relation.getRead().hasNamedTable()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): Nothing exercises this check. No JVM test or acceptance script builds a protobuf Read relation in either form, so deleting this condition would leave every suite green. A DataSource-form read such as spark.read.parquet(path) would then reach SparkConnectPlanner instead of failing with UNIMPLEMENTED as the README says. Could EngineSmokeTest send a DataSource Read through matchingMetadataFailure (prepare and every Analyze mode, expecting code 12), plus a named-table Read of a temp view as the positive case?

Verification:

  • Regression: A DataSource-form Read relation is rejected with UNIMPLEMENTED before planning, in prepare and in every Analyze mode.
  • Behavior: A named-table Read of an existing temp view is accepted and returns its rows.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a DataSource READ regression through prepare and all Analyze modes, asserting UNIMPLEMENTED and the named-table guard message, plus a named-table temp-view query that decodes and checks its row value and batch counts. The live JVM suite passes, and removing the guard in an isolated control makes the new test fail at the intended assertion.

Fixed in ac55ecd.

let batch = batch?;
let batch_schema = match &batch {
ResultBatch::Arrow(batch) => batch.schema(),
ResultBatch::Ipc { data, .. } => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): The JNI engine only ever produces ResultBatch::Ipc, so this is the branch real results go through, but no test reaches it. The schema-mismatch cases in tests/engine_injection.rs return ResultBatch::Arrow, and the IPC unit tests in this file call send_batch directly, which skips this check. Could the injection fixture gain an IPC mode with one batch whose schema differs from the declared one and one with invalid bytes, each expecting INTERNAL after the schema header and before any Arrow batch?

Verification:

  • Regression: An engine that emits an IPC batch whose schema differs from its declared schema makes ExecutePlan fail with INTERNAL without forwarding that batch.
  • Regression: An engine that emits invalid IPC bytes makes ExecutePlan fail with INTERNAL without forwarding them.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added real IPC fixture modes for a schema mismatch and malformed bytes. The gRPC test checks the schema header followed immediately by INTERNAL, before any Arrow batch or completion. It passes normally; isolated controls that disable schema checking or bypass IPC parsing each fail at the expected assertion.

Fixed in ac55ecd.

Comment thread crates/spark-connect-server/README.md Outdated
an admission timeout or another operation-level failure. All keys share the
original RPC's admission deadline; a long list does not reset that budget.

Each batch fetch creates a separate Spark SQL execution ID. The Spark UI and

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): SQLExecution.withNewExecutionId wraps loadNextPartition, which nextIpcBatch calls only once the previous partition's buffered batches are used up, and which collects all batches of one partition at once. So the Spark UI shows one execution per partition fetch, not per batch. When maxRecordsPerBatch splits a partition into many batches, this sentence overstates the count. The shuffle-cleanup reasoning below is still right if it also says "partition fetch".

Verification:

  • Inspection: The README's execution-ID and shuffle-cleanup statements refer to partition fetches, consistent with loadNextPartition.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected this to one execution per partition fetch, with explicit wording for multi-partition queries and zero-partition initialization. The shuffle-cleanup explanation now also refers to partition fetches.

Fixed in ac55ecd.

public final class Task implements AutoCloseable {
private final String jobGroup = "spark-connect-engine-" + UUID.randomUUID();
private final String description;
// Row iteration reads cancellation frequently; compound task transitions remain locked.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): No driver-side row loop exists in the IPC design anymore: ArrowPartitionConverter converts rows on executors and never reads this flag. The flag is read by check() per partition, per batch and per task step, and is set by cancel() on other threads. Could the comment name those readers instead?

Verification:

  • Inspection: The Task.cancelled comment describes per-partition and per-batch checks and cross-thread cancellation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the comment to describe partition, batch and task-step checks observing cross-thread cancellation.

Fixed in ac55ecd.

}

/**
* Internal typed fields: BE u32 pair count, then UTF-8 key length/data and nullable value

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): decodePairs reads this count with readInt() and rejects count < 0, so a value of 0x80000000 or more, which is valid as u32, fails with INVALID_ARGUMENT (configFieldCounts asserts exactly that). The 64 MiB cap keeps such counts unreachable, but the ABI comment should say a non-negative big-endian 32-bit signed count.

Verification:

  • Inspection: The Javadoc no longer says u32 and describes a non-negative signed 32-bit big-endian count.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected the ABI Javadoc to a non-negative big-endian signed 32-bit count. The Rust encoder now also uses checked i32 conversion; native tests and strict Clippy pass on Rust 1.100 and 1.88.

Fixed in ac55ecd.

}
}

// None is a recently released-session tombstone, retained by SessionRegistry.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): ConnectService holds no Option, so the None here refers to nothing a reader can see. It describes the Option<Arc<Session>> value of SessionRegistry.entries at line 131, so that field is the place for this comment.

Verification:

  • Inspection: The comment sits on the Option-valued entries field it describes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the tombstone comment to SessionRegistry.entries, next to its Option-valued map.

Fixed in ac55ecd.

}
}

/** Test-only rule barrier: the first analyzer invocation on each thread waits inside Spark. */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): enter() returns immediately for any thread not in readers, so only registered reader threads block, on their first invocation. The concurrent Config read and the mutation worker in metadataConcurrentReadersAndWriters pass straight through, and that test depends on it. The comment should say so.

Verification:

  • Inspection: The comment describes blocking as limited to registered reader threads on their first invocation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected the comment to say each registered reader waits on its first analyzer invocation.

Fixed in ac55ecd.

Comment thread AGENTS.md Outdated
cargo clippy --workspace --all-targets -- -D warnings # BLOCKING CI gate (needs the wasm32-unknown-unknown target)
# CI's lint job pins Rust 1.100.0 and also lints datafusion,polars and the excluded crates
```
Native JAR license assets bind whole-file hashes. After any change to `Cargo.lock`,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): This list matches today's inputs, but the generator computes the real set: build_outputs adds the Cargo.toml of every workspace package in the JNI cargo-tree closure and records the result in dependencies.json inputs. Once the closure gains another workspace crate, an agent following this rule would edit that crate's manifest without regenerating, and the --check step and packaging would then fail as stale. Could the rule point at dependencies.json inputs, or tell agents to run python3 scripts/gen_native_binary_licenses.py --check and regenerate when it reports stale?

Verification:

  • Inspection: The pre-push instruction no longer enumerates the fingerprinted files and instead refers to the recorded inventory or the generator's check mode.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the rule to reference the recorded dependencies.json inputs and require --check before pushing, with regeneration when assets are stale. Updated the same guidance in the server README, CI design and contributing docs. The merged target’s refreshed inventory passes the generator check, all 35 license-tool tests and packaging checks.

Fixed in ac55ecd.

@dbtsai

dbtsai commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

Latest source and benchmark update (2026-11-17)

The latest fixes are already pushed as ac55ecd: upstream integration/license inventory repair and all eight maintainer fixes. All eight feedback replies are posted. In the saved 2026-11-17 15:01 UTC snapshot, all 26 non-skipped CI checks passed, including all eight native workflow checks; docs deployment was skipped.

No performance candidate passed the complete acceptance criteria or was pushed. The measurements below describe local, unmerged experiments, not the performance of ac55ecd. The earlier published-head dce536e8 comparison remains historical: raw complete receipt was 226.521 ms native versus 170.223 ms JVM (raw generated-stub timer; not comparable with the toArrow() timings below). That benchmark was not rerun on ac55ecd.

The partition-overlap experiment fetches at most one following result partition while the current one streams, retaining up to two partition results; this PR fetches partitions sequentially. An unmodified official PySpark 4.2 DataFrame.toArrow() comparison produced:

Configuration OSS JVM median, pair 1 / 2 (ms) Experimental native median, pair 1 / 2 (ms) Paired native/JVM latency GM
NONE, 10,000 rows/batch 249.339 / 231.556 224.416 / 222.027 0.928981
ZSTD level 3, 100,000 rows/batch 169.933 / 176.116 163.858 / 168.991 0.961895

The observed action times were 7.1% and 3.8% lower, respectively. These are auxiliary observations. The primary sequence did not screen this candidate: an identical-binary Config Get control compared the same binary against itself and failed the original 0.75–1.25 guard with ratio 1.727346; the control budget was exhausted. This shows instability in the metadata measurement and does not clear the candidate for acceptance. The uncompressed official-client runs recorded more full-compaction and evacuation-failure pause events at a 1 GiB heap: whole-log full-compaction pause events were native 12/12 versus JVM 1/1, and evacuation-failure pause events were 62/62 versus 19/19. Client-window total logged pause fractions were native 4.42%/4.53% versus JVM 4.12%/3.52% (pair 1 / pair 2); these windows also include warmups, validation and cleanup. Both compressed forks for both servers recorded zero of those events. Event counts are logged pauses, not a count of all unique GC cycles or a latency decomposition.

Method: Apple M3 Max, 16 cores/64 GiB, macOS ARM64; JDK 21; the same stock Spark assembly at 250ccca3f2b5f043e20b415adf17bda5d202c60f; local[2]; 1 GiB driver heap; spark.range(0, 8_000_000, 1, 2); Python 3.12.14, PySpark 4.2.0, PyArrow 26.0.0 and gRPC 1.84.0. There were two AB/BA process pairs per configuration, eight fresh JVMs total, and three warmups plus 100 measured calls per JVM: 824 public actions including warmups. Ratios are geometric means of paired process medians. Identical GC logging and hosting-JVM RSS sampling were enabled for both servers. Timers surrounded only public toArrow(), including its normal response handling, IPC decoding and table assembly/cast. Schema/null/count/sum/extrema validation and explicit cleanup were outside timing; aggregate checks do not prove exact row order or uniqueness. Asynchronous response-release acknowledgements and wire retry counts were not observed.

Separate follow-ups on the same unmerged native experiment used raw generated gRPC stubs. Complete receipt drains through ResultComplete; receipt plus table also opens IPC streams and assembles a PyArrow table. These are separate executions with different boundaries from public toArrow():

Before → after Paired latency ratio: complete receipt / receipt plus table Observation
NONE → ZSTD, both 10K rows/batch 1.111824 / 1.425116 IPC bytes 65,248,000 → 8,457,816 (about 87% smaller), but about 11% / 43% slower locally
NONE/10K → ZSTD/100K 0.947430 / 1.240503 About 5.3% lower receipt and 24.1% higher receipt plus table latency; codec and batch-size effects are combined

A separate shared Spark block-fetch copy experiment (NONE, 10K rows/batch) completed 412 official toArrow() actions. Its patched/stock action-time ratio was 0.999923 (pairs 0.990636/1.009297), showing no meaningful end-to-end gain in this run. Both arms used the unmerged native experiment; only the Spark core block-fetch copy differed. It remains unpublished, and its normal upstream module build was blocked before compilation by the SBT launcher download.

All timings are from a shared local desktop and a narrow, highly compressible int64 workload. There is no production, distributed-network, significance or compression-default claim. Raw-stub and official-client timings are not interchangeable. The unmerged implementations cannot be reproduced from the PR head alone.

Executor-produced Arrow IPC is preserved. Ordinary batches pass through without decompression/recompression; oversized responses may still need decoding and splitting. The producer/off-heap API change remains deferred.

Benchmark summary prepared by OpenAI Codex.

@dbtsai

dbtsai commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

@HyukjinKwon @zhengruifeng , I planed to break this down into smaller pieces, separate the dependencies from independently mergeable changes, and prepare a concrete split for review.

@viirya

viirya commented Nov 17, 2026

Copy link
Copy Markdown
Member

Thanks for putting this together, and for offering to split it. Before going into the code, I'd like to settle the direction, since that decides how it should be split.

What problem does this solve, and for whom? The description presents this as a prototype for exploring the embedding boundary. It doesn't name a user problem it fixes. Spark is still the engine and the JVM is still the host; Rust only replaces the gRPC/session/operation layer of the existing JVM Connect server, which isn't usually the bottleneck. Could you add the concrete use case and goal?

The measured benefit doesn't justify the size yet. In the PR's own benchmark at dce536e8:

  • Config Get improves from 0.36 ms to 0.115 ms, but it's sub-millisecond either way.
  • Query workloads are roughly at parity (0.84–1.07).
  • Arrow result delivery, the hot path where a native frontend would be expected to help, is 33–40% slower, partly because partitions are fetched one at a time.

Only the unmerged overlap experiment reaches 0.93–0.96. That experiment didn't pass its own acceptance criteria and showed more full-GC pauses. So the result so far is a server with a smaller feature set than the JVM one (subset of RPCs, no auth/TLS/streaming) that is slower on the main data path.

Does this belong in this repository? This repo is the Rust Connect client and the PySpark drop-in. The JVM engine adapter depends on Spark internals (SessionHolder, SparkConnectPlanner, ArrowConverters). It was validated against a 4.2.0 snapshot, while the repo tracks the 4.3.0 protocol, and full acceptance needed a locally patched ArrowConverters. Code that has to change in lockstep with Spark internals seems a better fit for apache/spark's connect server module, or a separate repository, than for the client repo. It also adds a second implementation of server-side security behavior (config redaction, UDF guards, READ restrictions) that has to follow the JVM server. The first review here already found three bypasses of that kind.

Cost to the client project. native-server.yml triggers on crates/spark-connect/**, Cargo.toml and Cargo.lock, so most client PRs would run the four-platform native build and the pinned Spark assembly job. The native license fingerprint hashes all of Cargo.lock, so an unrelated lockfile change fails native CI until about 4k generated lines are regenerated.

Most of the complexity comes from the JNI boundary itself. Cleanup lanes, same-thread detach retries, tombstones, ownership retained after failed cleanup, and the "restart the JVM to recover" failure modes all exist because Rust owns Spark resources across JNI. Most of the correctness findings in the first review were in this layer.

Possible paths, depending on the goal:

  • If the goal is to discuss the boundary, a design doc or SPIP on dev@ with this PR kept as a draft or reference branch seems better than merging to master.
  • If the goal is to reuse the Rust protocol stack for other engines, the frontend crate with the Engine trait and mock-engine protocol tests (a few thousand lines, without JNI, C, Java, packaging or license tooling) could be reviewed on its own. The JVM engine could then live where Spark's internals do. Have you also looked at existing Rust Spark Connect servers such as Sail, and how this would relate to them?
  • If the goal is Connect server performance, it may be more direct to profile and improve SparkConnectService in Spark itself. The JVM server is currently ahead on Arrow delivery.

Once the direction is agreed, the split can follow from it, and the CI trigger scope and license fingerprint can be narrowed as part of that.


private static byte[] createSchemaIpc(StructType schema, String timeZone, boolean largeVarTypes) {
return checkedIpc(ArrowConverters$.MODULE$.createEmptyArrowBatch(
schema, timeZone, false, largeVarTypes));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve duplicate nested field names in returned schemas

For SELECT named_struct('a', 1, 'a', 2) AS s, passing false for errorOnDuplicatedFieldNames makes the pinned Spark Arrow converter rename the nested fields to a_0 and a_1. The JNI adapter then derives the schema returned to Connect clients from this Arrow schema, so both schema analysis and execution advertise names different from Catalyst's original schema. For example, selecting s.a_0 using the reported schema fails to resolve against the original plan.

validateResultType currently accepts this shape. Could we transport the original Spark schema separately, or explicitly reject duplicate nested names before conversion? A schema regression test using this query would cover the gap.

// conservatively restore every attempted key; unvisited keys never need rollback.
if (current != null && Objects.equals(value, current.get(key))) continue;
try {
if (value == null) spark.conf().unset(key);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Account for config dependencies during rollback

Starting with spark.sql.legacy.setCommandRejectsSparkCoreConfs=false, a nonsilent Set batch can change spark.driver.maxResultSize to 2g, set that legacy flag to true, and then fail on an invalid spark.sql.shuffle.partitions value. This loop restores keys in their original attempt order, so restoring the core setting calls RuntimeConfig.set/unset while the flag still prohibits core-config changes. The loop subsequently restores the flag but never retries the earlier key, leaving the core setting changed after the RPC fails.

This violates the full rollback behavior asserted by the new config tests. Could rollback restore the saved state independently of the batch's temporary setter restrictions, or restore the controlling settings first, with a regression test for this dependent-setting sequence?

Stack the native JNI host, Java adapter, platform packaging and integration coverage on the standalone frontend in PR apache#870. Preserve executor-produced IPC and exercise negotiated byte chunking through real JNI.

Written by OpenAI Codex.
@dbtsai
dbtsai force-pushed the codex/jvm-spark-connect-server-wip branch from f656e63 to 0f6a512 Compare November 19, 2026 06:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants