Repository navigation
Conversation
rich7420
added
run-all-spark-profiles
github-actions
Bot
added
enhancement
rich7420
force-pushed
the
feat/2837-native-map-explode
branch
from
September 28, 2026 14:50
76768f2 to
9601e81
Compare
Cover JVM-fed maps, nullable Boolean batch boundaries, all-NULL/empty outer batches, native map producers, and stored floating-point keys. Require native generators and matching Spark results and schemas. Benchmark map variants and nested map/array inputs with sinks that consume keys and positions. Remove the unrelated Parquet field-ID assertion change so it can be reviewed independently.
Keep input arrays as temporary placeholders until generated columns replace them. Build take indices only when a carried column consumes them. Cover recursive unnest with and without carried columns and both drop and outer null handling. Preserve the existing bounded map and array execution path.
rich7420
marked this pull request as ready for review
October 3, 2026 16:36
rich7420
added
run-spark-4.1-tests
Port map planning to the extracted ExplodeBuilder while preserving once-per-batch evaluation and returning an error for malformed map entry types.
Split individual List rows before materialization and cap repetition so carried buffers and child offsets stay within Arrow's offset limits. Preserve shared entry buffers, positions and outer semantics.
Fill non-null primitive runs directly for expansions of at least five and build short-string offsets with one Vec extension per run. Retain Arrow take for smaller expansions and nullable primitives.
Native generators materialize carried columns between expansions and were slower than Spark's pipeline in the nested benchmark. Keep maps followed by another generator on Spark across projections and filters, while retaining native single maps and chains separated by an exchange.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
CI mirror of upstream apache/datafusion-comet#6107, addressing apache#2837. This fork PR does not close or merge the upstream issue.
Rationale for this change
Map inputs to
explodeandposexplodecurrently fall back to Spark.What changes are included in this PR?
Support maps in all four explode variants by sharing their entry buffers with the existing unnest path. Preserve field metadata, entry order and outer NULL rows. Add regression tests and update support documentation.
How are these changes tested?
Local verification passed 18 Rust tests and 52 targeted tests on each of Spark 3.5.9 and 4.1.3, covering native execution, NULL/empty maps, nested inputs, sliced arrays and batch boundaries.
Upstream CI passed at
9601e81ba, including Required Checks. The corresponding fork run is not fully green: Spark 3.4 scans faileda file without ids next to a file with ids is checked on its ownin ParquetReadSuite (544 passed, one failed in that job). Its missing-field-ID error assertion failed. The relationship to this map-explode change has not been established; do not treat the run as a complete cross-profile pass until the failure is classified and resolved.