[build] Relocate shaded jackson and other internal deps in fluss-fs-s3

[Jackeyzhe]

Purpose

Linked issue: close #4072

The fluss-fs-s3 uber-jar bundles hadoop-common and aws-java-sdk-*
dependencies without any shade relocations, leaking 1,114 unshaded
jackson class files (jackson-databind, jackson-core,
jackson-annotations, jackson-dataformat-cbor) at com/fasterxml/*
paths. These can shadow a downstream application's own jackson-core
(e.g. 2.16+) and cause runtime failures such as NoSuchMethodError.

Brief change log

fluss-filesystems/fluss-fs-s3/pom.xml:

• Added shade relocations for the transitively-bundled third-party
  packages, namespaced under org.apache.fluss.fs.shaded.s3.*
  (mirroring the pattern already used by fluss-fs-hadoop-shaded):
  • com.fasterxml → org.apache.fluss.fs.shaded.s3.com.fasterxml
  • org.codehaus → org.apache.fluss.fs.shaded.s3.org.codehaus
  • com.ctc → org.apache.fluss.fs.shaded.s3.com.ctc
  • com.google.re2j → org.apache.fluss.fs.shaded.s3.com.google.re2j
  • org.apache.htrace → org.apache.fluss.fs.shaded.s3.org.apache.htrace
• Added a <filter> excluding META-INF/versions/**/com/fasterxml/**:
  the Maven Shade Plugin relocates base-path classes but not
  Multi-Release JAR entries (same issue class as [fluss-client][ fluss-flink-*] Uber-jars leak com.fasterxml.jackson.core alongside the shaded copy #3553, which affected
  fluss-shaded-jackson). The relocated base-path implementation is
  sufficient for all Java versions.

Tests

Build-only change (pom.xml), verified by inspecting the produced jar:

• Before fix: unzip -l fluss-fs-s3-*.jar | grep -c " com/fasterxml"
  → 1,114 unshaded entries
• After fix: 0 unshaded com/fasterxml entries (base path and MRJ);
  1,093 jackson classes correctly relocated to
  org/apache/fluss/fs/shaded/s3/com/fasterxml/*
• Bytecode references inside the bundled AWS SDK / Hadoop classes are
  rewritten to the shaded path (verified via javap on sampled classes)
• Intentional jaxb-api MRJ entries under META-INF/versions/** are
  preserved untouched
• ./mvnw spotless:check passes

API and Format

No API or storage format changes. This only affects the internal shading
layout of the fluss-fs-s3 uber-jar; all relocated classes move under
the org.apache.fluss.fs.shaded.s3 namespace.

Documentation

No new feature introduced; no documentation changes required.

[litiliu]

would it be better to shaded the whole com.google(including re2j and common)?

[Jackeyzhe]

Good catch — agreed, widened to the whole com.google namespace
(a6215ed). It turns out the jar was still leaking 1,974 unshaded
com.google.common (Guava) entries plus j2objc / thirdparty, which
is the same shadowing bug class as the jackson leak in #4072.

Verified: 0 unshaded com/google entries (base path and MRJ), all
constant-pool references rewritten to the shaded path, and hadoop's
own org.apache.hadoop.thirdparty.com.google namespace left untouched.

While verifying I also noticed fluss-fs-hdfs has the same problem
at a larger scale (1,073 unshaded com/fasterxml + 2,731 com/google
in its uber-jar) — I'll file a separate issue for that so this PR
stays scoped to fluss-fs-s3.

[Jackeyzhe]

Filed #4116 for the fluss-filesystems shading leak (all 7 plugins verified leaked, gs worst)
so #4073 stays scoped to s3 — thanks again for the catch, @litiliu.

[binary-signal]

@Jackeyzhe thank you for addressing this issue, I've left some comments

[binary-signal]

I applied this PR locally on top of main (4e372d5c7) and can confirm it does what it says: fluss-fs-s3 goes from 1050 unshaded com/fasterxml classes to 0, with 1974 guava, 48 re2j, 205 ctc and 125 codehaus classes correctly relocated under org.apache.fluss.fs.shaded.s3.

One thing worth addressing before merge, though.

com.google is broader than what the jar actually bundles

The shade plugin rewrites every reference matching a <relocation> pattern, including references to classes the jar does not contain. fluss-fs-s3 bundles only four com.google.* packages:

package	classes bundled
com.google.common	1974
com.google.re2j	48
com.google.j2objc	14
com.google.thirdparty	5

But the bundled Hadoop and AWS SDK code references com.google.protobuf and com.google.gson, neither of which is packaged (both are <exclusion>s in this pom). After this PR, those references are rewritten to org.apache.fluss.fs.shaded.s3.com.google.protobuf.* — about 40 distinct classes, e.g.:

org/apache/fluss/fs/shaded/s3/com/google/protobuf/Message                (11 refs)
org/apache/fluss/fs/shaded/s3/com/google/protobuf/Descriptors            (5 refs)
org/apache/fluss/fs/shaded/s3/com/google/protobuf/GeneratedMessage       (5 refs)
org/apache/fluss/fs/shaded/s3/com/google/gson/stream/JsonReader          (1 ref)

Before relocation, com.google.protobuf.Message is a soft dependency: absent from the uber-jar, but resolvable if the surrounding classpath or plugin parent classloader provides it. After relocation the reference points at a coordinate nothing will ever provide, because no artifact anywhere publishes org.apache.fluss.fs.shaded.s3.com.google.protobuf. A previously-satisfiable link becomes a guaranteed NoClassDefFoundError on any code path that touches it.

This is not hypothetical. I hit exactly this mechanism via the sibling org.apache.commons relocation (added in #3960, already on main) while extending this PR's approach to the other filesystem plugins:

java.lang.NoClassDefFoundError: org/apache/fluss/shaded/org/apache/commons/cli/ParseException
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1713)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1374)
    ...
    at org.apache.fluss.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBehaviorTest.java:62)

commons-cli was on the classpath. The reference had simply been rewritten to a name that does not exist.

To make this checkable , #4144 adds tools/ci/check_shaded_jars.py, whose --dangling mode reports exactly this: relocated class references a jar does not contain. Running it on fluss-fs-s3 built from this PR lists 92 such classes, 83 of them real classes rather than annotations.

Suggested change

Name the packages that are actually bundled rather than the com.google prefix:

<relocation>
    <pattern>com.google.common</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.common</shadedPattern>
</relocation>
<relocation>
    <pattern>com.google.j2objc</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.j2objc</shadedPattern>
</relocation>
<relocation>
    <pattern>com.google.thirdparty</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.thirdparty</shadedPattern>
</relocation>
<relocation>
    <pattern>com.google.re2j</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.re2j</shadedPattern>
</relocation>

This keeps the full guava fix (the actual win here — 1974 classes) and drops the dangling protobuf/gson links. It also matches what fluss-fs-hadoop-shaded already does, which the PR description cites as the pattern being mirrored: that module relocates com.google.re2j specifically, not com.google.

The same applies to the existing org.apache.commons relocation from #3960, which leaves dangling references to commons-cli, commons-codec, commons-math3, commons-net, commons-jxpath, commons-jexl2 and commons-vfs2 in every filesystem plugin. That is pre-existing rather than something this PR introduces; it is written up in #4143 along with the rest of the shading leaks.

#4144 applies the narrow-pattern approach across the modules #4143 covers, if a worked example is useful — every relocation there names the packages actually bundled, and io.netty.internal.tcnative and org.apache.arrow are excluded for the JNI reason above. It deliberately leaves fluss-fs-s3 to this PR.

Two notes for anyone verifying locally

• clean is required. An incremental build reuses already-relocated classes in target/classes, so a narrowed pattern appears to have no effect until you run mvn clean. This cost me a couple of confusing debug cycles.
• mvn test -pl fluss-filesystems -amd fails on HadoopUtilsTest independently of any of this: in a reactor containing fluss-fs-hadoop-shaded, dependent modules get its unshaded target/classes because shade runs at package. Use verify or install instead. Reproduces on unmodified main.

The MRJ exclusion in this PR is also worth keeping even alongside #3884's global one — #3884 puts it in the root pom under <filters combine.children="append">, so it already cascades here, but the local copy keeps this PR correct on its own.