Repository navigation
Conversation
580577c to
821c9a2
Compare
HyukjinKwon
left a comment
There was a problem hiding this comment.
I ran an AI-assisted review (Claude Code /code-review, max effort) of aa4e45c. Its verification pass checked each finding against the code, but the scenarios were not reproduced by running them. It found 15 correctness issues: 11 are inline comments, and 4 are below, for a file GitHub shows no diff for.
The first three inline comments look security-relevant:
- The UDF guard can be bypassed through typed aggregates.
- Config reads return values that Spark Connect redacts.
- The SQL relation can read local files and call Java methods, despite the named-table-only READ restriction.
crates/spark-connect-server/jni/src/engine.rs
GitHub renders no diff for this new file, so these can't be inline comments.
engine.rs line 668
A failed cleanup-lane spawn is cached for the life of the process. If the first std::thread::Builder::spawn for a lane fails transiently (EAGAIN under a thread limit or memory pressure), cleanup_sender's OnceLock stores the Err(String) permanently. From then on, every Handle::drop and Task::cancel takes Err(_) => cleanup(item) (line 742) and runs the JNI free or cancel inline on the dropping thread.
In execute_plan, that thread is one of the runtime's two async workers (worker_threads(2)). The worker then attaches to the JVM and blocks in Query.close, Session.close or Task.cancel, stalling gRPC I/O for the whole server. Retrying the spawn, or not caching the error, would avoid this.
engine.rs line 1345
silent_config_set stops at the first per-key error it doesn't list. Only InvalidArgument, NotFound, AlreadyExists and OutOfRange are turned into warnings.
Take a silent Set of [a, b, c] where RuntimeConfig.set(b) throws something else, such as an internal-error SparkThrowable or an IllegalStateException from a setter. applyConfig/engineError map it to INTERNAL, Err(error) => return Err(error) fails the RPC, and c is never applied. Spark's SparkConnectConfigHandler.handleSet instead adds a warning for any Throwable and goes on to the next key.
engine.rs line 696
A panic on a cleanup lane drops its queued items, and wait_drained then never returns. eprintln! panics on a stderr write error such as EPIPE (std ignores only EBADF). That happens, for example, when spark-submit's output is piped to a reader that has exited.
If a Java close fails in that state, the eprintln! in cleanup_with panics and the lane thread unwinds. Its mpsc receiver then drops the queued Cleanup::Free items without running them. counts.handles never reaches 0, so the listener thread's wait_drained() never returns, and every NativeEngineServer.close() reports 'JNI stop is incomplete' until the JVM exits.
Writing to stderr with writeln! and ignoring the error, or wrapping each item in catch_unwind, would keep the lane alive.
engine.rs line 420
A failed detach drops the only pointer to the attachment. Suppose spark_connect_engine_thread_detach returns -1 during on_thread_stop or thread-local teardown. C keeps the EngineThread and its context reference, as bridge.h says: 'A failed detach retains the guard so its owner can retry'. But WorkerAttachment::drop discards the pointer and never decrements worker_attachments. As a result:
- every later
stop_serverretry failsverify_workers_detached('1 JNI worker attachments remain after runtime stop'); free_checkedrefuses the factory, becausecontext->references != 1;release_factorycan't be reached until the JVM exits.
Native license hashes
The native license assets bind whole-file SHA-256 hashes of Cargo.lock, the root Cargo.toml, LICENSE, NOTICE and both generator scripts (scripts/gen_native_binary_licenses.py:49-61). So any unrelated lockfile change fails native CI and packaging until about 4k generated lines are regenerated. Hashing only what the licenses depend on, such as the resolved dependency list, would avoid that churn.
| } | ||
| } | ||
| String type = message.getDescriptorForType().getFullName(); | ||
| if (type.endsWith("CommonInlineUserDefinedFunction") || type.endsWith("CommonInlineUserDefinedTableFunction")) { |
There was a problem hiding this comment.
The UDF guard can be bypassed through typed aggregate expressions. validateRelations rejects UDFs only when the protobuf descriptor name ends in CommonInlineUserDefinedFunction or CommonInlineUserDefinedTableFunction. A ScalarScalaUDF nested in Expression.typed_aggregate_expression, inside an allowed AGGREGATE or PROJECT relation, matches neither name and passes.
That is what the Scala client sends for ds.select(aggregator.toColumn). The call path is SparkConnectPlanner.transformTypedAggregateExpression -> unpackScalaUDF -> UdfSerialization.deserialize(payload), which runs ObjectInputStream over client-supplied bytes with no class allowlist.
The guard says user-defined functions are unsupported in this prototype. Should it reject ScalarScalaUDF and TypedAggregateExpression explicitly, or allowlist expression types instead of denylisting names?
There was a problem hiding this comment.
Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. The recursive guard now rejects both TypedAggregateExpression and ScalarScalaUDF, including when nested in an otherwise supported relation. Regression tests use empty/default protobufs and a nested Project and verify rejection before planning.
| List<Pair> output = new ArrayList<>(); | ||
| switch (opcode) { | ||
| case 1: | ||
| for (Pair pair : input) output.add(new Pair(pair.key, spark.conf().get(pair.key))); |
There was a problem hiding this comment.
Config reads return secrets that Spark Connect redacts. Get, GetOption, GetWithDefault and GetAll (opcodes 1-4) return raw values with no spark.redaction.regex filtering. For example, with spark.hadoop.fs.s3a.secret.key=XYZ set:
spark.conf.get("spark.hadoop.fs.s3a.secret.key")returnsXYZ.- GetAll with an empty or
spark.hadoop.prefix lists it.
Spark 4.3's SparkConnectConfigHandler handles these keys differently (isRedacted, SparkConnectConfigHandler.scala:109-232):
- Get fails with SQL_CONF_NOT_FOUND.
- GetOption and GetWithDefault return None or the default.
- GetAll drops the entry.
No other layer redacts them either: not the Java adapter, bridge.c, jni/src/engine.rs or service.rs.
There was a problem hiding this comment.
Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. Config reads now use the hosting SparkContext's cached redaction pattern and check both the full key and value before stripping a GetAll prefix. Get returns the same error code and message as the same unset key, GetOption omits it, GetWithDefault returns the caller's default, and GetAll omits the entry. Tests cover all four reads and ensure a session-level pattern override cannot weaken the host pattern. This matches the newer Spark 4.3 handler; the pinned 4.2 snapshot used by the smoke tests did not yet redact these reads.
| if (!RELATIONS.contains(kind)) { | ||
| throw new EngineException(12, "unsupported JVM engine relation: " + kind); | ||
| } | ||
| if (relation.hasRead() && !relation.getRead().hasNamedTable()) { |
There was a problem hiding this comment.
The SQL relation bypasses the named-table-only READ restriction. The restriction is checked only on Relation.read. But SQL is in RELATIONS (line 112), and rejectCommands lets a SELECT through, so:
spark.sql("SELECT * FROM parquet.`/home/u/private.parquet`")reads a local file throughResolveSQLOnFile. So dojson.,csv.andtext.paths.SELECT reflect('java.lang.System', 'getenv', 'X')calls static Java methods.
The README says READ is restricted to named tables, and lists spark.read.parquet() as unsupported. If that restriction is meant to be a boundary, the SQL path probably needs spark.sql.runSQLOnFiles=false and to reject reflect/java_method. Otherwise the README should say the restriction isn't enforced for SQL.
There was a problem hiding this comment.
Clarified in dce536e8d00a747b8f23df0e663fbcbca962c3cc. The named-table restriction applies to the protobuf Relation.read form. SQL and named-table resolution use the hosting Spark session's semantics and can access files or reflective builtins. The README and security documentation now state that explicitly. This prototype accepts trusted local queries and does not provide an access-isolation boundary.
| "crates/pyspark-rs", | ||
| "examples", | ||
| "crates/spark-connect-server", | ||
| "crates/spark-connect-server/jni", |
There was a problem hiding this comment.
With the JNI crate in the workspace, every --workspace command needs a JDK and a C compiler. jni/build.rs panics without JAVA_HOME, and on Windows ('unsupported JNI target'). So on machines that only have Rust and the wasm target, these now fail:
cargo test --workspace;- the pre-push
cargo clippy --workspace --all-targets -- -D warningsthat AGENTS.md requires; - rust-analyzer's build-script runs.
That breaks this file's stated goal that cargo test --workspace stays green without extra toolchains, and lint.yml had to add a JDK. The --exclude spark-connect-jni workaround is documented, but a cfg or feature gate would keep the default commands working. A build.rs that silently skipped the shim would only move the failure to link time.
There was a problem hiding this comment.
The toolchain requirement is intentional for --workspace, and is documented immediately below this comment: workspace commands include the JNI host and require macOS/Linux, a C compiler and a JDK. The earlier “without it” refers specifically to the optional wasm target. Default workspace members remain unchanged and exclude the server/JNI crates. I kept this policy rather than adding a separate JNI feature gate; client-only workspace commands can use --exclude spark-connect-jni.
| if (error->data == NULL) set_error(error, "native Spark Connect operation failed"); | ||
| jstring message = utf8_string(env, error->data, error->len, NULL); | ||
| jclass type = (*env)->FindClass(env, "java/lang/IllegalStateException"); | ||
| if (!(*env)->ExceptionCheck(env) && message != NULL && type != NULL) { |
There was a problem hiding this comment.
throw_error can return without a pending Java exception. If building the message fails, the message != NULL guard skips Throw. That happens when utf8_string returns NULL after exception_error clears its OutOfMemoryError.
For example, nativeStop can fail with 'JNI stop is incomplete' after the 10 s deadline while the heap is low. It then returns normally, and NativeEngineServer.close() sets handle = 0 even though Rust still holds the server and factory, so the close can never be retried. nativeAddress likewise returns NULL with no exception, and address() throws a NullPointerException.
Throwing a preallocated or fixed-message error when the message can't be built would keep these failures visible.
There was a problem hiding this comment.
Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. throw_error preserves an existing pending exception and uses a fixed-message ThrowNew fallback if constructing the detailed message fails. Allocation failures in the fallback also leave their exception pending. The C mock tests exercise these paths, including message-allocation failures, and pass normally and under ASan/UBSan.
| if (bytes.length > MAX_IPC_BYTES) throw new EngineException(8, "config fields exceed 64 MiB"); | ||
| try (DataInputStream input = new DataInputStream(new ByteArrayInputStream(bytes))) { | ||
| int count = input.readInt(); | ||
| if (count < 0 || count > MAX_CONFIG_FIELDS || count > input.available() / 8) { |
There was a problem hiding this comment.
Config requests with more than 10,000 keys are rejected. decodePairs rejects more than MAX_CONFIG_FIELDS (10000) pairs with EngineException(3, "invalid config field count"), which the client sees as INVALID_ARGUMENT. Rust's encode_pairs forwards any count, and Spark Connect accepts the same request, which is far below the 64 MiB request limit. GetAll output isn't capped either, so a client can't send back the key list it just received.
The limit should either match on both sides and be documented in the README, or be raised or removed.
There was a problem hiding this comment.
Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. The arbitrary 10,000-pair cap is removed. The 64 MiB encoded-payload bound and count-versus-remaining-bytes checks remain. Regression tests accept 10,001 well-formed pairs and reject malformed counts.
| JavaScope scope; | ||
| if (engine_scope(task, &scope, NULL) != 0) return; | ||
| (*scope.env)->CallVoidMethod(scope.env, task->object, task->api->cancel); | ||
| (void) engine_exception(scope.env, task->api, NULL); |
There was a problem hiding this comment.
spark_connect_engine_cancel silently drops cancel failures. It returns void, so a failed scope entry and an exception from Java Task.cancel() are both discarded. Rust's one-shot Task::cancel then treats the cancel as delivered while the job group keeps running.
For example, if the host stopped the shared SparkContext before the server drained, cancelJobGroupAndFutureJobs throws an IllegalStateException from assertNotStopped:
- It is stored in
cancellationFailureand cleared by(void) engine_exception(...). - Every later
Task.close()rethrows it. - The Rust free stays in
failed_cleanup, andstop_serverreports incomplete cleanup indefinitely.
Returning a status, as the other entry points do, would let Rust handle the failure.
There was a problem hiding this comment.
Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. Cancellation now returns status and an error buffer across the C ABI. Both Rust callers retain failed cancellation ownership and admission permits for an explicit close retry instead of marking delivery successful. C and Rust tests cover scope-entry and Java failures. A partially failed Java cancellation remains sticky: retrying delivery does not claim to undo a terminal host failure.
| return; | ||
| } | ||
| let data = libc::malloc(bytes.len()).cast::<u8>(); | ||
| if data.is_null() { |
There was a problem hiding this comment.
write_buffer reports success with an empty buffer when malloc fails. It sets *out to the default and returns. spark_connect_address then maps Ok(()) to 0, and bridge.c builds an empty byte[]. So NativeEngineServer.address() returns "", and EmbeddedSparkEngine prints 'listening on ' instead of raising an error. Returning an error from write_buffer when allocation fails would surface it like the other failures.
There was a problem hiding this comment.
Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. write_buffer now returns a Result, and the address entry point propagates allocation failure instead of returning success with an empty address. An injected-allocation test exercises the helper's failure path.
| JAVAC="${JAVA_HOME:+$JAVA_HOME/bin/}javac" | ||
| JAR="${JAVA_HOME:+$JAVA_HOME/bin/}jar" | ||
| BUILD_DIR="$PROJECT_DIR/.build/jvm" | ||
| rm -rf -- "$BUILD_DIR/classes" "$BUILD_DIR/test-classes" |
There was a problem hiding this comment.
Concurrent script runs race on the shared build output. build.sh always deletes and rewrites .build/jvm/classes and the bridge jar in place, and run-engine.sh, test-engine.sh and test-jni-smoke.sh all call it.
Say run-engine.sh starts while test-engine.sh is compiling. The second build's rm -rf -- "$BUILD_DIR/classes" deletes classes the first javac is compiling against. A spark-submit that starts in that window can find the jar missing or truncated, which shows up as a spurious ClassNotFound or corrupt-zip error.
A lock (for example flock), or a per-run output directory with an atomic rename, would avoid this.
There was a problem hiding this comment.
Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. Each Java build now uses a private staging directory and atomically publishes complete, self-contained JARs after compilation succeeds. Adapter packaging validates the exact temporary copy before replacing the published JAR, and writes checksum files atomically. Three concurrent builds passed while a reader successfully opened the published JAR 718 times with no errors; regressions cover source replacement during copying and interrupted JAR/checksum writes.
| System.out.println("Embedded Spark Connect JVM PID " + ProcessHandle.current().pid() | ||
| + "; Java Connect listener absent"); | ||
| launched.set(true); | ||
| stopped.await(); |
There was a problem hiding this comment.
main is never told when the native listener exits on its own. It blocks on a latch that only the shutdown path counts down. Suppose the tonic server future returns an error, or block_on panics ('Rust listener panicked' in start_host, jni/src/lib.rs). The listener thread runs shutdown and wait_drained, then sends to done, which only stop_server reads. main stays in stopped.await(), holding the SparkSession with nothing bound to the port, until an external signal arrives.
This is rare, since tonic 0.14 ignores accept errors, but when it happens the process looks healthy while serving nothing.
There was a problem hiding this comment.
Fixed in dce536e8d00a747b8f23df0e663fbcbca962c3cc. The native host exposes listener-running state independently of cleanup drain, and the launcher polls it so an unexpected listener exit triggers teardown and an error. A close retry is allowed only after confirmed drain; incomplete or unknown cleanup retains the native handle and prevents stopping shared Spark resources. A Rust test exercises the real listener exiting while retained cleanup blocks drain; stock JNI smoke calls both health methods with a live handle. Java lifecycle tests cover the failure and cleanup ordering.
|
Addressed the four Rust lifecycle/config findings from this review in dce536e8d00a747b8f23df0e663fbcbca962c3cc:
I kept the native-license source fingerprints. Packaging can validate assets without running Cargo, so a hash of the generated dependency inventory alone would not detect a stale inventory after dependency or feature changes. An unrelated manifest-comment probe required regeneration but changed only one fingerprint line; the generated LICENSE and NOTICE text stayed unchanged. A semantic dependency projection could reduce that conservative invalidation in a follow-up. The 49 JNI tests, 29 frontend tests and strict native Clippy passed on Rust 1.100 and MSRV 1.88. Workspace Clippy, 691 client unit tests, C tests with ASan/UBSan, Java lifecycle/config tests, concurrent builds, packaging and license checks also passed. Real JNI start/query/stop passed against the unmodified pinned Spark 4.2 snapshot. The full packaged PySpark/raw-RPC and Rust-client acceptance run also passed using the separately repaired Arrow-input runtime described in the README. Cross-platform hosted checks for this head remain pending. |
HyukjinKwon
left a comment
There was a problem hiding this comment.
Review summary
The earlier review comments are addressed at this head. The UDF guard now rejects typed aggregates and ScalarScalaUDF. Config reads apply the host redaction pattern. The SQL access limit is documented. The tombstone, throw_error, cancel-status, write_buffer, build-staging and listener-exit fixes are in place, and the four engine.rs lifecycle and config findings are fixed. This review found only non-blocking issues.
Two validators added by this PR have no test that would fail if they regressed: the named-table-only READ check in the Java adapter, and the IPC branch of the frontend's per-batch schema check, which is the branch every JNI result takes. The UDTF clause of the same relation guard needs no separate test: a UDTF can appear in a plan only as a relation kind, and the relation allowlist already rejects it before that clause runs.
The rest are text corrections. The server README says each batch fetch is a separate SQL execution, but the code starts one per partition fetch. Three comments describe something other than the code: the cancellation flag's row-iteration rationale, the u32 pair count in the config ABI, and the AnalysisGate barrier's scope. The tombstone comment sits on ConnectService instead of SessionRegistry.entries. The new AGENTS.md pre-push rule copies the native-license input list instead of pointing at the inventory the generator records.
Findings
8 total: 0 P0, 0 P1, 0 P2, 8 P3.
Nit (P3)
- No test covers the named-table-only READ guard —
crates/spark-connect-server/jvm/src/main/java/org/apache/spark/sql/connect/nativebridge/engine/SparkJvmEngine.java:924— see inline. - The IPC arm of the per-batch schema check has no test —
crates/spark-connect-server/src/service.rs:621— see inline. - README says each batch fetch is a separate SQL execution; it is each partition fetch —
crates/spark-connect-server/README.md:289— see inline. - The Task.cancelled comment cites a row loop that no longer exists —
crates/spark-connect-server/jvm/src/main/java/org/apache/spark/sql/connect/nativebridge/engine/SparkJvmEngine.java:260— see inline. - Session.config Javadoc calls the pair count u32; decodePairs takes a non-negative i32 —
crates/spark-connect-server/jvm/src/main/java/org/apache/spark/sql/connect/nativebridge/engine/SparkJvmEngine.java:509— see inline. - Tombstone comment is attached to ConnectService, not to SessionRegistry.entries —
crates/spark-connect-server/src/service.rs:157— see inline. - AnalysisGate Javadoc overstates which threads block —
crates/spark-connect-server/jvm/src/test/java/org/apache/spark/sql/connect/nativebridge/engine/EngineSmokeTest.java:743— see inline. - AGENTS.md hardcodes the native-license input list that the generator computes —
AGENTS.md:274— see inline.
Files not given in full to design and detailed review
This change is too large to give design and detailed review every file's diff. They judged each group of mechanical edits from its shape and samples, and read the remaining files only where a judgment called for it; every file was still scanned individually.
crates/spark-connect-server/**/*.java: 10 of 12 files not given in full.crates/spark-connect-server/**/*.rs: 6 of 11 files not given in full.
| if (!RELATIONS.contains(kind)) { | ||
| throw new EngineException(12, "unsupported JVM engine relation: " + kind); | ||
| } | ||
| if (relation.hasRead() && !relation.getRead().hasNamedTable()) { |
There was a problem hiding this comment.
Nit (P3): Nothing exercises this check. No JVM test or acceptance script builds a protobuf Read relation in either form, so deleting this condition would leave every suite green. A DataSource-form read such as spark.read.parquet(path) would then reach SparkConnectPlanner instead of failing with UNIMPLEMENTED as the README says. Could EngineSmokeTest send a DataSource Read through matchingMetadataFailure (prepare and every Analyze mode, expecting code 12), plus a named-table Read of a temp view as the positive case?
Verification:
- Regression: A DataSource-form Read relation is rejected with UNIMPLEMENTED before planning, in prepare and in every Analyze mode.
- Behavior: A named-table Read of an existing temp view is accepted and returns its rows.
There was a problem hiding this comment.
Added a DataSource READ regression through prepare and all Analyze modes, asserting UNIMPLEMENTED and the named-table guard message, plus a named-table temp-view query that decodes and checks its row value and batch counts. The live JVM suite passes, and removing the guard in an isolated control makes the new test fail at the intended assertion.
Fixed in ac55ecd.
| let batch = batch?; | ||
| let batch_schema = match &batch { | ||
| ResultBatch::Arrow(batch) => batch.schema(), | ||
| ResultBatch::Ipc { data, .. } => { |
There was a problem hiding this comment.
Nit (P3): The JNI engine only ever produces ResultBatch::Ipc, so this is the branch real results go through, but no test reaches it. The schema-mismatch cases in tests/engine_injection.rs return ResultBatch::Arrow, and the IPC unit tests in this file call send_batch directly, which skips this check. Could the injection fixture gain an IPC mode with one batch whose schema differs from the declared one and one with invalid bytes, each expecting INTERNAL after the schema header and before any Arrow batch?
Verification:
- Regression: An engine that emits an IPC batch whose schema differs from its declared schema makes ExecutePlan fail with INTERNAL without forwarding that batch.
- Regression: An engine that emits invalid IPC bytes makes ExecutePlan fail with INTERNAL without forwarding them.
There was a problem hiding this comment.
Added real IPC fixture modes for a schema mismatch and malformed bytes. The gRPC test checks the schema header followed immediately by INTERNAL, before any Arrow batch or completion. It passes normally; isolated controls that disable schema checking or bypass IPC parsing each fail at the expected assertion.
Fixed in ac55ecd.
| an admission timeout or another operation-level failure. All keys share the | ||
| original RPC's admission deadline; a long list does not reset that budget. | ||
|
|
||
| Each batch fetch creates a separate Spark SQL execution ID. The Spark UI and |
There was a problem hiding this comment.
Nit (P3): SQLExecution.withNewExecutionId wraps loadNextPartition, which nextIpcBatch calls only once the previous partition's buffered batches are used up, and which collects all batches of one partition at once. So the Spark UI shows one execution per partition fetch, not per batch. When maxRecordsPerBatch splits a partition into many batches, this sentence overstates the count. The shuffle-cleanup reasoning below is still right if it also says "partition fetch".
Verification:
- Inspection: The README's execution-ID and shuffle-cleanup statements refer to partition fetches, consistent with loadNextPartition.
There was a problem hiding this comment.
Corrected this to one execution per partition fetch, with explicit wording for multi-partition queries and zero-partition initialization. The shuffle-cleanup explanation now also refers to partition fetches.
Fixed in ac55ecd.
| public final class Task implements AutoCloseable { | ||
| private final String jobGroup = "spark-connect-engine-" + UUID.randomUUID(); | ||
| private final String description; | ||
| // Row iteration reads cancellation frequently; compound task transitions remain locked. |
There was a problem hiding this comment.
Nit (P3): No driver-side row loop exists in the IPC design anymore: ArrowPartitionConverter converts rows on executors and never reads this flag. The flag is read by check() per partition, per batch and per task step, and is set by cancel() on other threads. Could the comment name those readers instead?
Verification:
- Inspection: The Task.cancelled comment describes per-partition and per-batch checks and cross-thread cancellation.
There was a problem hiding this comment.
Updated the comment to describe partition, batch and task-step checks observing cross-thread cancellation.
Fixed in ac55ecd.
| } | ||
|
|
||
| /** | ||
| * Internal typed fields: BE u32 pair count, then UTF-8 key length/data and nullable value |
There was a problem hiding this comment.
Nit (P3): decodePairs reads this count with readInt() and rejects count < 0, so a value of 0x80000000 or more, which is valid as u32, fails with INVALID_ARGUMENT (configFieldCounts asserts exactly that). The 64 MiB cap keeps such counts unreachable, but the ABI comment should say a non-negative big-endian 32-bit signed count.
Verification:
- Inspection: The Javadoc no longer says u32 and describes a non-negative signed 32-bit big-endian count.
There was a problem hiding this comment.
Corrected the ABI Javadoc to a non-negative big-endian signed 32-bit count. The Rust encoder now also uses checked i32 conversion; native tests and strict Clippy pass on Rust 1.100 and 1.88.
Fixed in ac55ecd.
| } | ||
| } | ||
|
|
||
| // None is a recently released-session tombstone, retained by SessionRegistry. |
There was a problem hiding this comment.
Nit (P3): ConnectService holds no Option, so the None here refers to nothing a reader can see. It describes the Option<Arc<Session>> value of SessionRegistry.entries at line 131, so that field is the place for this comment.
Verification:
- Inspection: The comment sits on the Option-valued entries field it describes.
There was a problem hiding this comment.
Moved the tombstone comment to SessionRegistry.entries, next to its Option-valued map.
Fixed in ac55ecd.
| } | ||
| } | ||
|
|
||
| /** Test-only rule barrier: the first analyzer invocation on each thread waits inside Spark. */ |
There was a problem hiding this comment.
Nit (P3): enter() returns immediately for any thread not in readers, so only registered reader threads block, on their first invocation. The concurrent Config read and the mutation worker in metadataConcurrentReadersAndWriters pass straight through, and that test depends on it. The comment should say so.
Verification:
- Inspection: The comment describes blocking as limited to registered reader threads on their first invocation.
There was a problem hiding this comment.
Corrected the comment to say each registered reader waits on its first analyzer invocation.
Fixed in ac55ecd.
| cargo clippy --workspace --all-targets -- -D warnings # BLOCKING CI gate (needs the wasm32-unknown-unknown target) | ||
| # CI's lint job pins Rust 1.100.0 and also lints datafusion,polars and the excluded crates | ||
| ``` | ||
| Native JAR license assets bind whole-file hashes. After any change to `Cargo.lock`, |
There was a problem hiding this comment.
Nit (P3): This list matches today's inputs, but the generator computes the real set: build_outputs adds the Cargo.toml of every workspace package in the JNI cargo-tree closure and records the result in dependencies.json inputs. Once the closure gains another workspace crate, an agent following this rule would edit that crate's manifest without regenerating, and the --check step and packaging would then fail as stale. Could the rule point at dependencies.json inputs, or tell agents to run python3 scripts/gen_native_binary_licenses.py --check and regenerate when it reports stale?
Verification:
- Inspection: The pre-push instruction no longer enumerates the fingerprinted files and instead refers to the recorded inventory or the generator's check mode.
There was a problem hiding this comment.
Changed the rule to reference the recorded dependencies.json inputs and require --check before pushing, with regeneration when assets are stale. Updated the same guidance in the server README, CI design and contributing docs. The merged target’s refreshed inventory passes the generator check, all 35 license-tool tests and packaging checks.
Fixed in ac55ecd.
|
Latest source and benchmark update (2026-11-17) The latest fixes are already pushed as No performance candidate passed the complete acceptance criteria or was pushed. The measurements below describe local, unmerged experiments, not the performance of The partition-overlap experiment fetches at most one following result partition while the current one streams, retaining up to two partition results; this PR fetches partitions sequentially. An unmodified official PySpark 4.2
The observed action times were 7.1% and 3.8% lower, respectively. These are auxiliary observations. The primary sequence did not screen this candidate: an identical-binary Config Get control compared the same binary against itself and failed the original 0.75–1.25 guard with ratio 1.727346; the control budget was exhausted. This shows instability in the metadata measurement and does not clear the candidate for acceptance. The uncompressed official-client runs recorded more full-compaction and evacuation-failure pause events at a 1 GiB heap: whole-log full-compaction pause events were native 12/12 versus JVM 1/1, and evacuation-failure pause events were 62/62 versus 19/19. Client-window total logged pause fractions were native 4.42%/4.53% versus JVM 4.12%/3.52% (pair 1 / pair 2); these windows also include warmups, validation and cleanup. Both compressed forks for both servers recorded zero of those events. Event counts are logged pauses, not a count of all unique GC cycles or a latency decomposition. Method: Apple M3 Max, 16 cores/64 GiB, macOS ARM64; JDK 21; the same stock Spark assembly at Separate follow-ups on the same unmerged native experiment used raw generated gRPC stubs. Complete receipt drains through
A separate shared Spark block-fetch copy experiment (NONE, 10K rows/batch) completed 412 official All timings are from a shared local desktop and a narrow, highly compressible int64 workload. There is no production, distributed-network, significance or compression-default claim. Raw-stub and official-client timings are not interchangeable. The unmerged implementations cannot be reproduced from the PR head alone. Executor-produced Arrow IPC is preserved. Ordinary batches pass through without decompression/recompression; oversized responses may still need decoding and splitting. The producer/off-heap API change remains deferred. Benchmark summary prepared by OpenAI Codex. |
|
@HyukjinKwon @zhengruifeng , I planed to break this down into smaller pieces, separate the dependencies from independently mergeable changes, and prepare a concrete split for review. |
|
Thanks for putting this together, and for offering to split it. Before going into the code, I'd like to settle the direction, since that decides how it should be split. What problem does this solve, and for whom? The description presents this as a prototype for exploring the embedding boundary. It doesn't name a user problem it fixes. Spark is still the engine and the JVM is still the host; Rust only replaces the gRPC/session/operation layer of the existing JVM Connect server, which isn't usually the bottleneck. Could you add the concrete use case and goal? The measured benefit doesn't justify the size yet. In the PR's own benchmark at
Only the unmerged overlap experiment reaches 0.93–0.96. That experiment didn't pass its own acceptance criteria and showed more full-GC pauses. So the result so far is a server with a smaller feature set than the JVM one (subset of RPCs, no auth/TLS/streaming) that is slower on the main data path. Does this belong in this repository? This repo is the Rust Connect client and the PySpark drop-in. The JVM engine adapter depends on Spark internals ( Cost to the client project. Most of the complexity comes from the JNI boundary itself. Cleanup lanes, same-thread detach retries, tombstones, ownership retained after failed cleanup, and the "restart the JVM to recover" failure modes all exist because Rust owns Spark resources across JNI. Most of the correctness findings in the first review were in this layer. Possible paths, depending on the goal:
Once the direction is agreed, the split can follow from it, and the CI trigger scope and license fingerprint can be narrowed as part of that. |
|
|
||
| private static byte[] createSchemaIpc(StructType schema, String timeZone, boolean largeVarTypes) { | ||
| return checkedIpc(ArrowConverters$.MODULE$.createEmptyArrowBatch( | ||
| schema, timeZone, false, largeVarTypes)); |
There was a problem hiding this comment.
[P2] Preserve duplicate nested field names in returned schemas
For SELECT named_struct('a', 1, 'a', 2) AS s, passing false for errorOnDuplicatedFieldNames makes the pinned Spark Arrow converter rename the nested fields to a_0 and a_1. The JNI adapter then derives the schema returned to Connect clients from this Arrow schema, so both schema analysis and execution advertise names different from Catalyst's original schema. For example, selecting s.a_0 using the reported schema fails to resolve against the original plan.
validateResultType currently accepts this shape. Could we transport the original Spark schema separately, or explicitly reject duplicate nested names before conversion? A schema regression test using this query would cover the gap.
| // conservatively restore every attempted key; unvisited keys never need rollback. | ||
| if (current != null && Objects.equals(value, current.get(key))) continue; | ||
| try { | ||
| if (value == null) spark.conf().unset(key); |
There was a problem hiding this comment.
[P2] Account for config dependencies during rollback
Starting with spark.sql.legacy.setCommandRejectsSparkCoreConfs=false, a nonsilent Set batch can change spark.driver.maxResultSize to 2g, set that legacy flag to true, and then fail on an invalid spark.sql.shuffle.partitions value. This loop restores keys in their original attempt order, so restoring the core setting calls RuntimeConfig.set/unset while the flag still prohibits core-config changes. The loop subsequently restores the flag but never retries the earlier key, leaving the core setting changed after the RPC fails.
This violates the full rollback behavior asserted by the new config tests. Could rollback restore the saved state independently of the batch's temporary setter restrictions, or restore the controlling settings first, with a regression test for this dependent-setting sequence?
Stack the native JNI host, Java adapter, platform packaging and integration coverage on the standalone frontend in PR apache#870. Preserve executor-produced IPC and exercise negotiated byte chunking through real JNI. Written by OpenAI Codex.
f656e63 to
0f6a512
Compare
Review split and dependency
This PR is rebased on the standalone Rust frontend in #870, at
56640d306ec510252d962b77396f7f64ac7484ae. Review the remaining native layer with the parent-to-head comparison. Both PRs currently target Apachemaster, so GitHub's Files changed tab includes #870 until it lands. This PR depends on #870 and preserves its frontend implementation unchanged.The planned follow-up split remains: Java Spark adapter and executor IPC contract; JNI integration and embedded launcher; native dependency licensing; platform JAR packaging and loading; and combined/universal distributions. Lifecycle and cancellation checks stay with the implementation they protect. Performance optimization and producer/off-heap API redesign remain separate follow-up work.
What changes were proposed in this pull request?
Add an experimental Spark JVM execution engine for #870's Rust Spark Connect frontend. Rust owns the gRPC listener, protocol sessions, operations and response streams. A Java adapter invokes Spark planning and execution APIs in the same JVM through JNI. The server and JNI crates are opt-in and unpublished.
Executors produce complete Arrow IPC batches using Spark's existing
ArrowConverters.toBatchWithSchemaIterator. The driver fetches and materializes one partition at a time. JNI copies each Javabyte[]directly into a Rust-ownedVec; the intermediate C allocation and full-payload copy have been removed. Moving that allocation into protobufBytesand slicing negotiated transport chunks do not copy the payload. Standard Prost encoding still copies it into the outgoing gRPC buffer. The Java producer API is unchanged.The inherited frontend follows JVM result framing: with client opt-in, it slices serialized IPC bytes and sends chunk indexes/counts for client reassembly. Without opt-in, it sends the whole IPC batch. Arrow responses have no unconditional 3 MiB split or decode/re-encode path. The frontend defaults to about 115.2 MiB maximum chunk payloads, accepts smaller valid client preferences, and permits disabling chunking with a zero configured maximum. These are transport settings: this adapter retains its separate approximate 2 MiB producer target and post-conversion 64 MiB JNI IPC safety bound. The embedded Java launcher currently uses the frontend defaults. The frontend's separate schema/eager-command response bound is 3 MiB.
Deferred handle cleanup retains ownership and admission permits until teardown succeeds. Cancellation failures retain ownership for retry; config reads apply the host redaction pattern; and the launcher detects unexpected listener termination. Native libraries can be packaged in Linux/macOS x86-64/ARM64 platform JARs, a macOS universal JAR, or an all-platform JAR alongside the Java adapter. The loader checks artifact/ABI versions and checksums, and generated dependency licenses accompany native artifacts. CI builds platform artifacts and exercises a pinned Spark assembly.
Why are the changes needed?
This prototype explores reusing the Rust protocol and transport stack as an embedded frontend while Spark remains the execution engine. It provides an implementation for discussing that boundary and its lifecycle.
JIRA: https://issues.apache.org/jira/browse/SPARK-61384
Does this PR introduce any user-facing change?
Yes. Developers can explicitly build and launch the embedded prototype. Existing default workspace members and client APIs are unchanged.
The prototype supports trusted local workloads and a subset of batch queries, result types and RPCs. The named-table restriction applies to protobuf READ relations; SQL and named-table resolution follow the host session semantics, including access to files and reflective builtins. Unsupported operations return
UNIMPLEMENTED. Authentication, authorization, TLS, full protocol parity, Windows and native library unloading are outside its scope. Session/task limits and the JNI payload check do not provide complete memory or CPU quotas.The Java adapter must be available to executors. Partition fetching is sequential, and each fetched partition is materialized on the driver; large partitions can exhaust memory or hit
spark.driver.maxResultSize. Arrow conversion inherits Spark's allocator and task cleanup behavior. Nondefault IPC compression interoperability has not been validated for this native adapter. The README documents supported operations and lifecycle contracts.The adapter depends on internal Spark APIs. Local integration validation uses Spark
4.2.0-SNAPSHOTrevision250ccca3f2b5f043e20b415adf17bda5d202c60f, Scala 2.13 and JDK 21 on macOS ARM64. The unmodified pinned assembly is used for real JNI smoke; broader acceptance uses a separately repaired Arrow-input runtime. These checks do not establish compatibility with every Spark 4.2 build or Spark 4.3/4.4 assemblies.How was this patch tested?
Fresh local checks for the rebased native layer passed on macOS ARM64 with JDK 21:
wasm-udf.-Xcheck:jniagainst the unmodified pinned Spark assembly. New checks reassemble negotiated 1 KiB chunks and verify row values/order/offsets, then verify a complete IPC batch above 3 MiB is delivered without client chunking opt-in.Hosted CI has not yet validated this rebased head. Earlier multi-platform/universal-JAR CI results belong to the original implementation and are not current-head results. This local run does not validate nondefault IPC codecs, other platforms or other Spark versions.
Historical performance measurement (November 17, 2026; native source
23b2eb6)These measurements predate this rebase and #870's frontend framing; they do not measure the rebased revision.
Four matched process pairs compared the OSS JVM Connect server with this embedded Rust implementation on the same Linux ARM64 host and unmodified Spark assembly (
250ccca3f2b5f043e20b415adf17bda5d202c60f). Pair order was AB, BA, BA, AB, with fresh server and official Python client processes. Both usedlocal[2], a 1 GiB heap, two input/shuffle partitions, UTC, 10,000 Arrow rows per batch and IPC codecnone. The runtime was JDK 21.0.13, PySpark client 4.2.0, PyArrow 26.0.0 and gRPC 1.84.0. Server CPUs were 2-5 and client launch CPUs 6-7; affinity was nonexclusive.Latency columns are geometric means of four fork medians. Changes are the geometric mean of the four matched native/JVM ratios; positive values mean higher native latency.
DataFrame.toArrow()Public
toArrow()was slower in all four native pairs, ranging from 24.0% to 32.6% higher latency. It includes official-client response handling, IPC decode, table materialization and schema cast. Raw receipt drains all protobuf responses through completion, with IPC decode outside timing. Both servers returned 8 million int64 rows, 800 IPC batches and 65,248,000 IPC bytes per raw Arrow action. These separately sampled windows cannot be subtracted to estimate decode cost.Concurrent Config was slower in Rust in three of four pairs despite its lower geometric mean of fork medians; whole-round throughput was 23.3% lower. These results establish neither a general native speedup nor improved concurrent capacity.
All 13,152 action records passed their recorded checks, including 9,464 measured records and 3,688 warmups. Explicit session/client/server cleanup and listener closure passed. The server README records the complete workload table, sampling settings and limits. The benchmark uses local executors and loopback transport on one host; within-process calls are correlated. It establishes no significance, distributed-network, wide-row, compressed-IPC or retained-memory result. The current implementation is not presented as an Arrow performance improvement over JVM Connect.
A separate November 18 native-before/after experiment used native revision
23b2eb6538b355987341b385cdfccc3222669e4bon both sides and tested an unmerged Spark producer output-sizing patch against a matched recompilation control. Its two matched pairs in AB, BA order retained 1,500 actions (1,040 measured, 460 warmups): raw Arrow receipt latency was 9.43% lower, while publictoArrow()latency was 1.31% higher. Tiny SQL varied widely (pair ratios 0.9253 and 1.8582). That run had no JVM reference, demonstrated no consistent public-client gain and did not measure GC. It cannot be compared directly with the November 17 figures. The Spark patch is not included in this PR. Performance optimization and producer/off-heap API redesign remain follow-up work.Was this patch authored or co-authored using generative AI tooling?
This pull request and its description were written by OpenAI Codex.