[spark] Add paimon spark4.2 module by LuciferYang · Pull Request #9265 · apache/paimon

LuciferYang · 2026-08-17T05:48:09Z

Purpose

Raises the spark4 profile baseline from 4.1.2 to 4.2.0, adds a paimon-spark-4.2 module, and keeps Spark 4.0 and 4.1 working under the new baseline. Same shape as #7648, which did this for 4.1. Related to #8901.

paimon-spark-common and paimon-spark4-common are compiled once, against the newest supported Spark, and the resulting classfiles ship to every 4.x runtime. Raising the baseline therefore changes bytecode that 4.0 and 4.1 have to load, and Spark 4.2 made several source-compatible but binary-incompatible changes: CatalogManager became an interface (a 4.2-built call site emits invokeinterface and dies with IncompatibleClassChangeError on 4.0/4.1); case classes gained fields (CatalogStorageFormat, AppendData, DataSourceV2ScanRelation), so positional patterns and named-argument copy calls no longer link across versions; RewriteRowLevelCommand's DELTA_OPERATIONS_WITH_* constants were renamed; V2WriteCommand gained a WriteWithSchemaEvolution supertype; and DESCRIBE ... PARTITION moved out of DescribeRelation into its own DescribeTablePartition plan (SPARK-39660).

Four mechanisms are used, in order of preference:

Version-neutral construction: match by type with named accessors instead of positional patterns, and build placeholders through factory methods (CatalogStorageFormat.empty) rather than arity-sensitive constructors.
SparkShim methods where only the arity differs, so each per-version module supplies its own call.
SparkVersionCompat reflective accessors where the signature is incompatible. Reflection is immune to the class/interface flip, since only invoke opcodes carry that distinction.
Same-FQCN forks in paimon-spark-4.0 / -4.1 where a supertype or a parameter type differs and no accessor can paper over it. Shade writes the module's own classes first, so the fork wins.

paimon-spark-ut-4.0 and -4.1 recompile the shared test sources against their own baseline and produce test-jars only. They set skipTests as a module property rather than as scalatest plugin configuration: surefire inherits the root pom's default-test and integration-tests executions, and both plugins bind that parameter to ${skipTests}. Without it, mvn install over the reactor without -DskipTests runs the shared Java tests a second time there and fails on TestLoggerExtension with NoClassDefFoundError: org/slf4j/spi/LoggingEventBuilder, since neither module carries the SLF4J 2.x pin that paimon-spark-4.0 and -4.1 have. publish-snapshot-spark4.yml deploys them next to paimon-spark-ut, because the published 4.0 and 4.1 poms now declare those test-jars at test scope. The release lane publishes none of the three, which is how paimon-spark-ut was already handled.

That also retires StreamTestCheckAnswerWithTimeoutStub in paimon-spark-4.0. Its eight empty classes stood in for StreamTest inner classes that exist only from 4.1 on, which the baseline-compiled test classfiles reached through mix-in forwarders and a 4.0 runtime could not resolve. Compiled against 4.0.3 those references are never emitted: the paimon-spark-ut-4.0 test-jar and the baseline one carry the same 589 classes, 15 of which name those symbols in the baseline build and none in the 4.0 build.

One behaviour fix is included because the baseline bump caused it. qualifyIdentifier has to carry the catalog so Spark 4.2's SimpleFunctionRegistryBase.normalizeFuncName sees a 3-part identifier, but the same identifier reached the expression builder, which renamed the default output column of an unaliased v1 function call from db.udf(...) to catalog.db.udf(...) on every version from 3.4 up. The builder name now drops the catalog; the registry key keeps it.

On the test side, paimon-spark-4.2 picks up the three suites its siblings already have (KeyPropertyCaseResolutionTest, GeospatialTypeSQLTest, GeospatialTypeTest) and gets its own hive-site.xml, so its Hive suites bind port 9093 instead of picking up 9083 from the shared ut test-jar. SparkInternalRowVariantTest lived only in paimon-spark-4.0, even though #9493 added toSparkVariant to the 4.1 shim in the same commit; it now runs in all three 4.x modules, with a second case for the typed getVariant getter that the generic get(pos, VariantType) path never reaches.

Porting those suites turned up two tests that could not fail. POINT_WKB was 20 bytes where a little-endian POINT(1 2) needs 21, so the 4.1 fixture had been asserting a round-trip over malformed WKB; 4.2 parses it and rejects it. And Spark4InternalRow.getBinaryView dispatches on the Paimon field type, which nothing could observe while both columns used OGC:CRS84 and the same bytes, so the geometry column now uses the projected EPSG:3857.

Tests

All 25 CI lanes passed on 0094db2, including Java / Spark 4 / Scala 2.13 (1h4m) and both Spark 3 lanes. The branch has since been merged with master (9140cb3, which already contains #9794) and CI is re-running.

On the merge head, locally on JDK 17: paimon-spark-4.2 46 suites / 1302 tests, paimon-spark-4.1 46 / 1302, paimon-spark-4.0 44 / 1219 with 22 version-gated cancellations. No failures, no aborted suites.

Locally on JDK 17 unless noted, over a chain rebuilt with clean install:

Full module suites: paimon-spark-4.2 46 suites / 1302 tests, paimon-spark-4.1 46 / 1302, paimon-spark-4.0 44 / 1219 with 22 version-gated cancellations. No failures.
paimon-spark-ut 124 suites / 955 tests plus 74 JUnit tests. Its only failures are the 11 in LuminaVectorIndexTest, which cannot run on this machine at all: lumina-jni ships linux/amd64 natives only.
mvn -Papache-release,spark4,flink1 clean install over paimon-spark-ut-4.0 and -ut-4.1, matching what the snapshot workflow runs: success.
mvn -Pspark3 clean compile over paimon-spark-common + paimon-spark3-common on JDK 11: success.
spotless:check clean on the eight modules the spark4 profile builds here, and on paimon-spark3-common + paimon-spark-common under -Pflink1,spark3,scala-2.12.
mvn -pl paimon-spark-ut-4.0,paimon-spark-ut-4.1 -Pflink1,spark4 install without -DskipTests: success, six Tests are skipped. lines, 588 test classes still compiled in each. Before the property was added the same command exited 1 with the NoClassDefFoundError above, which is how that defect was found.
python3 -m unittest discover -s tools/ci: 28 tests, including the spark4 module list, which now also fails when a version is added rather than only when one is dropped.

Three of the fixes were checked by breaking the code they cover:

swapping the two branches of getBinaryView fails GeospatialTypeTest with [ST_INVALID_CRS_VALUE] ... 'EPSG:3857', thrown from GeographyType$.toSrid on the first getBinaryView(0);
swapping value() and metadata() in toSparkVariant fails the generic case, and doing the same in Spark4InternalRow.getVariant fails only the typed case, in both the 4.1 fork and paimon-spark4-common;
reverting the column-name fix turns the new PaimonV1FunctionTestBase case red with ArraySeq("paimon.test.udf_add2(3, 4)") did not equal List("test.udf_add2(3, 4)").

That last case asserts the column name rather than the row values, which is why the existing cases could not catch the regression: they compare rows with checkAnswer and spell out AS wherever a name is involved.

The 4.0 stub removal was verified after clearing target/test-classes, since the compiled stubs survive a source deletion and a green run over stale output proves nothing.

Three of the fixes were checked by breaking the code they cover:

swapping the two branches of getBinaryView fails GeospatialTypeTest with [ST_INVALID_CRS_VALUE] ... 'EPSG:3857', thrown from GeographyType$.toSrid on the first getBinaryView(0);
swapping value() and metadata() in toSparkVariant fails the generic case, and doing the same in Spark4InternalRow.getVariant fails only the typed case, in both the 4.1 fork and paimon-spark4-common;
reverting the column-name fix turns the new PaimonV1FunctionTestBase case red with ArraySeq("paimon.test.udf_add2(3, 4)") did not equal List("test.udf_add2(3, 4)").

That last case asserts the column name rather than the row values, which is why the existing cases could not catch the regression: they compare rows with checkAnswer and spell out AS wherever a name is involved.

The 4.0 stub removal was verified after clearing target/test-classes, since the compiled stubs survive a source deletion and a green run over stale output proves nothing.

LuciferYang · 2026-08-17T06:07:33Z

Let me resolve the conflicts.

Raises the `spark4` profile baseline from 4.1.2 to 4.2.0, adds a `paimon-spark-4.2` module, and keeps Spark 4.0 and 4.1 working under the new baseline. Same shape as apache#7648, which did this for 4.1. ## Why the bump breaks the older modules `paimon-spark-common` and `paimon-spark4-common` are compiled once, against the newest supported Spark, and the resulting classfiles ship to every 4.x runtime. Raising the baseline therefore changes bytecode that 4.0 and 4.1 have to load, and Spark 4.2 made several source-compatible but binary-incompatible changes: - `CatalogManager` became an interface, so a 4.2-built call site emits `invokeinterface` and dies with `IncompatibleClassChangeError` on 4.0/4.1. - Case classes gained fields (`CatalogStorageFormat`, `AppendData`, `DataSourceV2ScanRelation`), so positional patterns and named-argument `copy` calls no longer compile or link across versions. - `RewriteRowLevelCommand`'s `DELTA_OPERATIONS_WITH_*` constants were renamed, and `V2WriteCommand` gained a `WriteWithSchemaEvolution` supertype. - `DESCRIBE ... PARTITION` moved out of `DescribeRelation` into its own `DescribeTablePartition` plan (SPARK-39660). - SPARK-57058 folded the geo value classes into `BinaryView`: `SpecializedGetters` lost `getGeometry` / `getGeography` in favour of `getBinaryView`, `GeometryVal` / `GeographyVal` were removed, and `STUtils.stAsBinary` / `stSetSrid` were split into `stGeomAsBinary` / `stGeogAsBinary` and `stGeomSetSrid` / `stGeogSetSrid`. ## Four mechanisms, in order of preference 1. **Version-neutral construction.** Match by type with named accessors instead of positional patterns; build placeholders through factory methods (`CatalogStorageFormat.empty`) rather than arity-sensitive constructors. 2. **`SparkShim` methods** where only the arity differs, so each per-version module supplies its own call. 3. **`SparkVersionCompat`** reflective accessors where the *signature* is incompatible. Reflection is immune to the class/interface flip: only invoke opcodes carry that distinction. 4. **Same-FQCN forks** in `paimon-spark-4.0` / `-4.1` where a supertype or a parameter type differs and no accessor can paper over it. Shade writes the module's own classes before the ones it pulls in from `paimon-spark4-common`, so the fork wins. The geospatial support added by apache#9251 needs mechanism 4. `paimon-spark4-common`'s `Spark4ArrayData`, `Spark4InternalRow` and `Spark4Shim` now implement the 4.2 shape (`getBinaryView`, `stGeomAsBinary` / `stGeogAsBinary`, `stGeogSetSrid`), and `paimon-spark-4.1` forks all three to keep the pre-4.2 pair of overrides, which 4.1 still declares abstract. `paimon-spark-4.0` already forked the two data classes for the same reason -- 4.0 has no geo types at all. `paimon-spark-ut-4.0` and `-4.1` recompile the shared test sources against their own baseline; they produce test-jars only and are deliberately left out of publish and release, unlike `paimon-spark-ut`. ## Also fixed here `qualifyIdentifier` has to carry the catalog so Spark 4.2's `SimpleFunctionRegistryBase.normalizeFuncName` sees a 3-part identifier. The same identifier reached the expression builder, which renamed the default output column of an unaliased v1 function call from `db.udf(...)` to `catalog.db.udf(...)` on every version from 3.4 up. The builder name now drops the catalog; the registry key keeps it. The new test asserts the column name -- the existing cases all compare rows with `checkAnswer` and spell out `AS` wherever a name is involved, so none of them could see it. ## Verification `mvn -Pspark4 clean install` over `paimon-spark-common`, `paimon-spark4-common`, the three ut modules and `paimon-spark-4.0` / `-4.1` / `-4.2`: success. `spotless:check` clean on all six touched modules. Targeted suites: `PaimonV1FunctionTest` 13/13 on 4.2, 4.1 and 4.0; `DescribeTableTest` 4/4 on 4.2, 4.1 and 4.0; `SparkVersionCompatTest` 14/14. For the geospatial path, `GeospatialTypeSQLTest` 2/2 and `GeospatialTypeTest` 2/2 on 4.1 (the version that exercises the forked pre-4.2 overrides) and `GeospatialUnsupportedTest` 1/1 on 4.0. The column-name fix was verified in both directions -- reverting it turns the new case red with `ArraySeq("paimon.test.udf_add2(3, 4)") did not equal List("test.udf_add2(3, 4)")`. The full per-module suites have not been run in this branch yet.

All three come from Spark 4.2's storage-partitioned-join rework (SPARK-55535). `EnsureRequirements` now wraps a bucketed scan in `GroupPartitionsExec`, which casts `child.outputPartitioning` to `Partitioning with Expression`. Our AQE prep rule runs after it and disabled the bucketing underneath, which degrades that to `UnknownPartitioning`, so the cast threw at execution time. The traversal now stops at such a node and leaves its whole subtree alone: `EnsureRequirements` wraps whatever satisfied the distribution, which can be a join or an aggregate with several scans below it, and disabling only some of them would instead trip `PartitioningCollection`'s equal-`numPartitions` requirement. `DataSourceRDDPartition`'s payload changed from `Seq[InputPartition]` to `Option[InputPartition]`. `PaimonSparkMicroBatchMetadata` reads that field reflectively and rejected the new shape, and the blanket `catch NonFatal` around it reported that as "metadata absent" instead of failing. It now accepts every shape the field has had: a bare `InputPartition` on 3.2, a `Seq` from 3.3 to 4.1, and an `Option` since 4.2. The accessor is also resolved once per RDD rather than once per partition, since a miss costs a thrown `NoSuchMethodException`. `CREATE TABLE LIKE` with a four-part name is parsed by Spark itself on 4.2 and by Paimon's extension parser below it, so the middle parts arrive split differently. The test now asserts the invariant that holds on every version, which is that the parts survive in order, rather than pinning one version's split.

…LE assertions Spark 4.2 (SPARK-55372) makes `SHOW CREATE TABLE` print the collation on every string-ish column even when the column has none, so that replaying the emitted DDL cannot silently pick up a table- or schema-level `DEFAULT COLLATION` instead. A column declared `STRING` now renders as `STRING COLLATE UTF8_BINARY`, and so do VARCHAR, CHAR and the string leaves of ARRAY / MAP / STRUCT. This is not Paimon-specific: Spark's own golden file expects the same output for a parquet table, and Paimon's string columns reach it through the ordinary `StringType` case object, which Spark reads as "no explicit collation". The 31 failing assertions were all comparing against the pre-4.2 rendering. These tests are compiled once and run against every supported Spark version, so they now route `SHOW CREATE TABLE` through a helper that removes the marker rather than branching on the version. Only ` COLLATE UTF8_BINARY` is removed, so a column carrying a real collation such as `STRING COLLATE UTF8_LCASE` still appears and an assertion cannot be fooled into accepting the wrong collation.

LuciferYang · 2026-08-18T07:48:30Z

cc @melin FYI

JingsongLi · 2026-08-20T02:21:05Z

cc @Zouxxyy to take a review.

LuciferYang · 2026-08-20T04:05:13Z

Thank you @JingsongLi

JingsongLi · 2026-08-30T02:02:54Z

cc @Zouxxyy

JingsongLi

The Spark 4.2 support has concrete user-facing value (#8901), and the compatibility shims/forks address real binary differences in the shared Spark modules. I reviewed the main compatibility paths, old-version test-jar wiring and release inclusion; I did not find an evidence-backed blocking defect in that scope.

The existing Spark 4.x CI run (https://github.com/apache/paimon/actions/runs/32106655843/job/95617365797) actually executes SQL/storage tests across 4.0, 4.1 and 4.2, including reads, writes and schema evolution. Please resolve the current conflicts and rerun that matrix on the resulting head, including dynamic partition overwrite, MERGE, SQL functions and DESCRIBE PARTITION. I did not independently build the full Spark matrix or exhaustively audit every copied class; this is not a merge approval.

…trix

Resolve doc conflicts by taking master's restructured Compatibility Matrix and Spark quick-start, and carry the Spark 4.2 entries into the relocated spark/installation.mdx (version list, download tables, build command) and the ecosystem Spark row.

paimon-spark-4.2, paimon-spark-ut-4.0 and paimon-spark-ut-4.1 are new in this PR, so master's 2.2-SNAPSHOT version bump could not reach them and the merge left their paimon-spark parent pinned at 2.1-SNAPSHOT. The reactor then failed to resolve paimon-spark:pom:2.1-SNAPSHOT and could not read these three projects. Align them with the current 2.2-SNAPSHOT. Co-Authored-By: Claude Code <noreply@anthropic.com>

…on strip cannot tell apart

# Conflicts: # .github/workflows/utitcase-spark-4.x.yml

The old utitcase-spark-4.x.yml this branch edited was deleted upstream; the module list now lives in run-java-tests.sh.

createFormatTableBatchWrite took a fourth overwriteDynamic argument that no constructor accepts, and the interface does not declare: the shared builder already folds that mode into the table options before calling the shim. toSparkVariant was never implemented, so the class was abstract.

Tests 4.2 was missing, plus one that every 4.x module was missing: - KeyPropertyCaseResolutionTest, GeospatialTypeSQLTest, GeospatialTypeTest - SparkInternalRowVariantTest in 4.1 and 4.2, with a typed-access case that covers each module's own Spark4InternalRow.getVariant Test effectiveness: - pin the 4.2 geometry/geography dispatch by giving the geometry column a projected CRS; with both columns on OGC:CRS84 the two branches were indistinguishable - fix the 20-byte POINT_WKB fixture in 4.1 and 4.2 (POINT(1 2) needs 21) Build and infrastructure: - paimon-spark-4.2 gets its own hive-site.xml (port 9093) - drop the StreamTest stub from paimon-spark-4.0: paimon-spark-ut-4.0 compiles the shared test sources against 4.0.3, so the 4.1-only symbols it stood in for are never referenced - deploy paimon-spark-ut-4.0 / -4.1 from publish-snapshot-spark4.yml - expect 4.2 in the spark4 CI self-test and make the spark3 half symmetric Comments and docs that no longer matched the code: - the 4.0 forks of Spark4ArrayData / Spark4InternalRow described the 4.1 baseline; the slf4j pin comment claimed all of Spark 4.x needs SLF4J 2.x - geospatial version range in both the Spark and Iceberg data-type pages

…perty Only scalatest was skipped, so surefire kept the root pom's default-test and integration-tests executions and ran the shared Java tests a second time in paimon-spark-ut-4.0 and -ut-4.1. It failed there rather than merely duplicating work: neither module carries the SLF4J 2.x pin that paimon-spark-4.0 and -4.1 have, so TestLoggerExtension dies with NoClassDefFoundError on org/slf4j/spi/LoggingEventBuilder and `mvn install` without -DskipTests exits 1. Both plugins bind their skipTests parameter to ${skipTests}, so a module property covers them both. maven.test.skip would also skip test compilation and leave the test-jar these modules exist to produce empty.

LuciferYang · 2026-09-14T02:55:25Z

A rebase will be required after merging #9794.

# Conflicts: # .github/workflows/publish-snapshot-spark4.yml # paimon-spark/paimon-spark-4.0/pom.xml # paimon-spark/paimon-spark-ut-4.0/pom.xml # pom.xml

LuciferYang · 2026-09-14T06:35:27Z

Can we merge this first, and I'll address any follow-up issues in subsequent fixes? @JingsongLi

LuciferYang mentioned this pull request Aug 17, 2026

[Feature] spark 4.2 #8901

Open

2 tasks

LuciferYang force-pushed the spark-4.2-pr1 branch from 91640c1 to f37fb20 Compare August 17, 2026 06:49

LuciferYang added 2 commits August 17, 2026 21:33

JingsongLi reviewed Sep 10, 2026

View reviewed changes

LuciferYang and others added 9 commits September 10, 2026 15:58

Merge branch 'master' into spark-4.2-pr1 to rerun the Spark 4.x CI ma…

e4c3c30

…trix

Merge remote-tracking branch 'upstream/master' into HEAD

7df5522

docs: say what the bucketed-scan forks differ by and what the collati…

3aac2fb

…on strip cannot tell apart

Merge remote-tracking branch 'upstream/master' into spark-4.2-pr1

e9115b9

# Conflicts: # .github/workflows/utitcase-spark-4.x.yml

ci: run the 4.2 suite in the reusable Java workflow

e15c4dd

The old utitcase-spark-4.x.yml this branch edited was deleted upstream; the module list now lives in run-java-tests.sh.

LuciferYang marked this pull request as draft September 13, 2026 17:37

LuciferYang marked this pull request as ready for review September 13, 2026 17:37

LuciferYang mentioned this pull request Sep 13, 2026

[spark] paimon-spark-4.0 loads shared test bytecode compiled against a newer Spark #9793

Closed

2 tasks

Merge remote-tracking branch 'upstream/master' into spark-4.2-pr1

9140cb3

# Conflicts: # .github/workflows/publish-snapshot-spark4.yml # paimon-spark/paimon-spark-4.0/pom.xml # paimon-spark/paimon-spark-ut-4.0/pom.xml # pom.xml

Navigation Menu

Uh oh!

[spark] Add paimon spark4.2 module - #9265

[spark] Add paimon spark4.2 module#9265
LuciferYang wants to merge 14 commits into
apache:masterfrom
LuciferYang:spark-4.2-pr1

LuciferYang commented Aug 17, 2026 •

edited

Loading

Uh oh!

LuciferYang commented Aug 17, 2026

Uh oh!

LuciferYang commented Aug 18, 2026

Uh oh!

JingsongLi commented Aug 20, 2026

Uh oh!

LuciferYang commented Aug 20, 2026

Uh oh!

JingsongLi commented Aug 30, 2026

Uh oh!

JingsongLi left a comment

Uh oh!

LuciferYang commented Sep 14, 2026

Uh oh!

LuciferYang commented Sep 14, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

LuciferYang commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

Uh oh!

LuciferYang commented Aug 17, 2026

Uh oh!

LuciferYang commented Aug 18, 2026

Uh oh!

JingsongLi commented Aug 20, 2026

Uh oh!

LuciferYang commented Aug 20, 2026

Uh oh!

JingsongLi commented Aug 30, 2026

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Sep 14, 2026

Uh oh!

LuciferYang commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LuciferYang commented Aug 17, 2026 •

edited

Loading

LuciferYang commented Sep 14, 2026 •

edited

Loading