Skip to content

[build] Relocate third-party classes leaking from uber-jars #4144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
binary-signal wants to merge 10 commits into
base: main
Choose a base branch
from
+1,140 −1
Open

[build] Relocate third-party classes leaking from uber-jars #4144

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5997a02
[build] Add jar shading leak checker
binary-signal Aug 28, 2026
c22976f
[docs] Document the shading rules and the jar leak checker
binary-signal Aug 28, 2026
768847c
[build] Widen leak checker to all shaded libraries, add audit mode
binary-signal Aug 28, 2026
9068ba2
[build] Relocate bundled third-party packages in the remaining fs plu…
binary-signal Aug 28, 2026
7d22f11
[build] Relocate guava and commons in fluss-fs-hadoop-shaded
binary-signal Aug 28, 2026
e3e6e07
[build] Make the leak checker's own globbing sufficient
binary-signal Aug 28, 2026
1b16ba4
[build] Fix leak checker false positive, sample labelling and --json
binary-signal Aug 28, 2026
e701adb
[client] Relocate commons-lang3 and commons-math3 in the client uber-jar
binary-signal Aug 28, 2026
876292a
[lake] Relocate bundled commons and jackson in the iceberg and lance …
binary-signal Aug 28, 2026
cf7f00e
[fs] Relocate netty in the azure, cos, obs and oss plugins
binary-signal Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions fluss-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,28 @@
<include>*:*</include>
</includes>
</artifactSet>
<relocations>
<!-- commons-lang3 and commons-math3 are compile dependencies of
fluss-common, so they land in this uber-jar at their original
package. On an engine classpath they shadow the engine's own
copies; #3960 fixed exactly that for the filesystem plugins
after Spark hit NoSuchMethodError on a shadowed commons class.

Named individually rather than as org.apache.commons: the
shade plugin rewrites every reference matching a pattern,
including ones to classes this jar does not bundle. Only
lang3 and math3 are bundled; commons-codec and
commons-logging are referenced but come from elsewhere, and
relocating them would leave links that resolve nowhere. -->
<relocation>
<pattern>org.apache.commons.lang3</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.lang3</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.math3</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.math3</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
Expand Down
49 changes: 49 additions & 0 deletions fluss-filesystems/fluss-fs-azure/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,10 +237,59 @@
</filter>
</filters>
<relocations>
<!-- relocate the third-party packages bundled from
Hadoop and the cloud SDK so they cannot shadow a
downstream application's own copies, mirroring
fluss-fs-hadoop-shaded and fluss-fs-s3 -->
<relocation>
<pattern>com.google</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.azure.com.google
</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.htrace</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.azure.org.apache.htrace
</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.azure.com.fasterxml
</shadedPattern>
</relocation>
<relocation>
<pattern>org.codehaus</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.azure.org.codehaus
</shadedPattern>
</relocation>
<relocation>
<pattern>com.ctc</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.azure.com.ctc
</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons</shadedPattern>
</relocation>
<relocation>
<pattern>io.netty</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.azure.io.netty
</shadedPattern>
<excludes>
<!-- netty-tcnative is a JNI wrapper around OpenSSL and its
native symbols bind to the io.netty.internal.tcnative
package name, so renaming it would break them. It is
referenced here but supplied from elsewhere, if at all;
leaving it alone keeps OpenSSL usable when the host
provides it and avoids references resolving nowhere. -->
<exclude>io.netty.internal.tcnative.**</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
</execution>
Expand Down
49 changes: 49 additions & 0 deletions fluss-filesystems/fluss-fs-cos/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,10 +257,59 @@
</filter>
</filters>
<relocations>
<!-- relocate the third-party packages bundled from
Hadoop and the cloud SDK so they cannot shadow a
downstream application's own copies, mirroring
fluss-fs-hadoop-shaded and fluss-fs-s3 -->
<relocation>
<pattern>com.google</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.cos.com.google
</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.htrace</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.cos.org.apache.htrace
</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.cos.com.fasterxml
</shadedPattern>
</relocation>
<relocation>
<pattern>org.codehaus</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.cos.org.codehaus
</shadedPattern>
</relocation>
<relocation>
<pattern>com.ctc</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.cos.com.ctc
</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons</shadedPattern>
</relocation>
<relocation>
<pattern>io.netty</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.cos.io.netty
</shadedPattern>
<excludes>
<!-- netty-tcnative is a JNI wrapper around OpenSSL and its
native symbols bind to the io.netty.internal.tcnative
package name, so renaming it would break them. It is
referenced here but supplied from elsewhere, if at all;
leaving it alone keeps OpenSSL usable when the host
provides it and avoids references resolving nowhere. -->
<exclude>io.netty.internal.tcnative.**</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
</execution>
Expand Down
40 changes: 40 additions & 0 deletions fluss-filesystems/fluss-fs-gs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,6 +424,46 @@
</filter>
</filters>
<relocations>
<!-- relocate the third-party packages bundled from
Hadoop and the cloud SDK so they cannot shadow a
downstream application's own copies, mirroring
fluss-fs-hadoop-shaded and fluss-fs-s3 -->
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.gs.com.google.common
</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.re2j</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.gs.com.google.re2j
</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.htrace</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.gs.org.apache.htrace
</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.gs.com.fasterxml
</shadedPattern>
</relocation>
<relocation>
<pattern>org.codehaus</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.gs.org.codehaus
</shadedPattern>
</relocation>
<relocation>
<pattern>com.ctc</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.gs.com.ctc
</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons</shadedPattern>
Expand Down
67 changes: 66 additions & 1 deletion fluss-filesystems/fluss-fs-hadoop-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,13 +226,78 @@
</includes>
</artifactSet>
<relocations>
<!-- we shade only the parts that are internal to Hadoop and not used / exposed downstream -->
<!-- Shade the third-party packages Hadoop drags in, so they cannot
shadow a downstream application's own copies. org.apache.hadoop
itself is deliberately left alone: the FileSystem SPI resolves
implementations by class name from configuration. -->
<!-- These patterns name individual packages rather than the
com.google / org.apache.commons prefixes on purpose. The shade
plugin rewrites every reference matching a pattern, including
references to classes this jar does not bundle, which would
leave dangling links. Hadoop here references but does not
bundle com.google.protobuf, com.google.gson, commons-cli,
commons-codec, commons-math3 and commons-net; relocating those
prefixes wholesale breaks the NameNode at runtime. Only
packages actually present below. -->
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.hadoop3.com.google.common
</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.j2objc</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.hadoop3.com.google.j2objc
</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.hadoop3.com.google.thirdparty
</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.re2j</pattern>
<shadedPattern>
org.apache.fluss.fs.shaded.hadoop3.com.google.re2j
</shadedPattern>
</relocation>
<!-- commons packages that are bundled here. The shaded target
matches what the filesystem plugins already use, so a plugin
bundling this jar ends up with one copy rather than two. -->
<relocation>
<pattern>org.apache.commons.beanutils</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.beanutils</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.collections</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.collections</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.compress</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.compress</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.configuration2</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.configuration2</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.io</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.io</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.lang3</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.lang3</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.logging</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.logging</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.text</pattern>
<shadedPattern>org.apache.fluss.shaded.org.apache.commons.text</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.htrace</pattern>
<shadedPattern>
Expand Down
Loading
Loading