CI: Add runtime-deps.txt files for all runtimes and bundles by kevinjqliu · Pull Request #16081 · apache/iceberg

kevinjqliu · 2026-04-22T21:46:22Z

Context: #16080

Add all runtime-deps.txt so that CI will start enforcing dependency changes.

I checked out main, git pull and ran

./gradlew generateRuntimeDeps -DallModules --rerun-tasks

Committed the resulting files in this PR.

Note that spark/v4.1/spark-runtime/runtime-deps.txt was added in #16080

manuzhang · 2026-04-23T01:45:56Z

no open-api module?

rdblue · 2026-04-23T16:52:51Z

Should we be publishing the iceberg-open-api module? It is for testing so it would make sense to me if we didn't publish one.

I also didn't know about that runtime module and we will need to evaluate its LICENSE and NOTICE files before including it in any more releases.

ghost · 2026-04-23T21:47:05Z

+org.apache.httpcomponents:httpcore:4.4.16
+org.apache.logging.log4j:log4j-api:2.20.0
+org.apache.logging.log4j:log4j-core:2.20.0
+org.apache.logging.log4j:log4j-slf4j-impl:2.20.0


Log4J is not included in the LICENSE file.

Opened #16105 to fix this by removing log4j.

ghost · 2026-04-23T21:48:23Z

+software.amazon.awssdk:utils:2.42.33
+software.amazon.eventstream:eventstream:1.0.1
+software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin:2.4.1
+software.amazon.s3.analyticsaccelerator:analyticsaccelerator-s3:1.3.1


I think that these are fine. I didn't check every one against the latest update (fb2c8ac3faf) because they are now grouped into high level SDK modules, but they are all ALv2 and should be okay.

ghost · 2026-04-23T22:02:11Z

@@ -0,0 +1,70 @@
+com.github.ben-manes.caffeine:caffeine:2.9.3


Note to other reviewers: The aws-bundle/LICENSE file and azure-bundle/LICENSE file both include "JCTools (via Netty)". This is correct: Netty shades org.jctools under io/netty/util/shaded, which is then shaded in org/apache/iceberg/aws/shaded.

ghost · 2026-04-23T22:47:18Z

+org.apache.avro:avro:1.12.1
+org.apache.datasketches:datasketches-java:6.2.0
+org.apache.datasketches:datasketches-memory:3.0.2
+org.apache.flink:flink-metrics-dropwizard:2.1.0


This is not listed in the LICENSE file.

I also wonder if this should be excluded and not added to LICENSE because it seems like something that should be included in the Flink runtime. I suspect that we need to add it as a compileOnly dependency or suppress it in the runtime config.

Opened a PR for this fix: #16093

ghost · 2026-04-23T22:48:13Z

@@ -0,0 +1,33 @@
+com.fasterxml.jackson.core:jackson-annotations:2.21


I'm not reviewing older versions of Flink yet. I think we should determine what needs to change for the current version and then verify the same changes on the older ones.

This file and the 2.1 file are the same. I also verified that all of the directories/packages in the Jars are the same. We should be able to copy the 2.1 LICENSE and NOTICE into 2.0.

ghost · 2026-04-23T23:00:47Z

+com.github.ben-manes.caffeine:caffeine:2.9.3
+com.github.luben:zstd-jni:1.5.7-3
+com.google.errorprone:error_prone_annotations:2.10.0
+dev.failsafe:failsafe:3.3.2


This is leaked by iceberg-aws and should not be bundled.

This is used directly by S3InputStream, which means it needs to be included when iceberg-aws is included becuase it is not provided by the AWS dependencies. I don't think this is a good reason to keep using it and that we should replace it with Tasks, unless it is doing something special.

Since this is in the license docs, I think this isn't a blocker for 1.11.0 or 1.10.2, but we should remove it to keep dependencies to a minimum.

ghost · 2026-04-23T23:01:24Z

@@ -0,0 +1,33 @@
+com.fasterxml.jackson.core:jackson-annotations:2.21


The LICENSE file stats that this contains many missing libraries:

Apache Commons [not in the Jar]

OpenTelemetry [not in the Jar]

Netty [not in the Jar]

Apache Arrow [not in the Jar]

javax.annotation-api [not in the Jar]

Apache Apache Thrift [shaded by Parquet]

Fastutil [shaded by Parquet]

Apache Hive [artifact from ORC?]

Alibaba Cloud Credentials [fixed by Aliyun: Remove leaked transitive dependencies #15858]

Alibaba Cloud Tea [fixed by Aliyun: Remove leaked transitive dependencies #15858]

Google GAX [fixed by Change bigquery to compileonly #15655]

Google Auth Library [fixed by Change bigquery to compileonly #15655]

Google Cloud BigQuery client [fixed by Change bigquery to compileonly #15655]

Google Gson [fixed by Aliyun: Remove leaked transitive dependencies #15858]

Google Protobuf [fixed by Change bigquery to compileonly #15655]

Google API Common [fixed by Change bigquery to compileonly #15655]

Google Http Client [fixed by Change bigquery to compileonly #15655]

Google Auto Valve [fixed by Change bigquery to compileonly #15655]

Google flatbuffers [fixed by Change bigquery to compileonly #15655]

Kotlin standard library [fixed by Aliyun: Remove leaked transitive dependencies #15858]

gRPC [fixed by Aliyun: Remove leaked transitive dependencies #15858]

Google APIs [fixed by Change bigquery to compileonly #15655]

Google Cloud APIs [fixed by Change bigquery to compileonly #15655]

JSpecify [fixed by Change bigquery to compileonly #15655]

Animal Sniffer Annotations [fixed by Change bigquery to compileonly #15655]

Android Annotations [fixed by Change bigquery to compileonly #15655]

Conscrypt [fixed by Change bigquery to compileonly #15655]

Perfmark [fixed by Change bigquery to compileonly #15655]

org.json. [fixed by Change bigquery to compileonly #15655]

OpenSensus [fixed by Change bigquery to compileonly #15655]

JaCoCo [fixed by Aliyun: Remove leaked transitive dependencies #15858]

JAXB [fixed by Aliyun: Remove leaked transitive dependencies #15858]

Okio [fixed by Aliyun: Remove leaked transitive dependencies #15858]

OkHttp [fixed by Aliyun: Remove leaked transitive dependencies #15858]

ThreeTen BP [fixed by Change bigquery to compileonly #15655]

The reason for most of the extras is that there were no LICENSE updates after a few recent PRs:

Aliyun: Remove leaked transitive dependencies #15858 removed leaked transitive dependencies from Aliyun

Change bigquery to compileonly #15655 removed leaked transitive dependencies from GCP

Core: Replace Failsafe with Tasks #15613 removed failsafe, but iceberg-aws still leaks it

Also, LICENSE contains Google Guava, which is present because this shades iceberg-bundled-guava. But shading in that module means we don't have it listed here (FYI).

Action items:

Find out why some libraries were there but are no longer:

Arrow, Netty, Apache Commons, OpenTelemetry, javax.annotations

Fix the Hive entry in LICENSE. Before chore: several fixes on the LICENSE/NOTICE #15449 it was clear that this was shaded by ORC. Now the only Hive reference I see is META-INF files so I think this is probably incorrect.

Remove all of the fixed dependency leaks from LICENSE and NOTICE

Apache Commons is from this commit: 760a20b

#2102 copied array methods into ArrayUtil. This isn't a big problem, but it doesn't seem worth the hassle of tracking it down in LICENSE to have array copy methods. The implementations don't match project style or provide value. A good first issue is to remove them.

OpenTelemetry, Arrow, Netty, and javax.annotations appear to have never been included in the runtime Jar through dependencies. It's unclear why they were added to LICENSE by fb2c8ac but they are not part of the runtime now and can be removed.

Opened #16102 to fix the LICENSE and NOTICE for v2.1

ghost · 2026-04-23T23:58:13Z

+com.google.cloud:google-cloud-kms:2.91.0
+com.google.cloud:google-cloud-monitoring:3.89.0
+com.google.cloud:google-cloud-storage:2.64.1
+com.google.code.findbugs:jsr305:3.0.2


Findbugs is excluded throughout the codebase because it was originally LGPL and cannot be bundled. The license issues weren't clarified, and a clean implementation was created: https://github.com/stephenc/findbugs-annotations

Although the maven metadata reports ALv2, we need to exclude it. If we need the annotations (which are not required to function), then we should use the stephenc verison.

ghost · 2026-04-24T00:02:15Z

+org.jspecify:jspecify:1.0.0
+org.slf4j:slf4j-api:2.0.17
+org.threeten:threeten-extra:1.8.0
+org.threeten:threetenbp:1.7.0


The rest of these appear to be real dependencies from GCP and correctly included in the LICENSE file.

ghost · 2026-04-24T02:00:52Z

@rdblue This is the PR to fix open-api LICENSE and NOTICE issues. Please take a look and we can discuss whether the module needs to be published there.

ghost · 2026-04-24T19:36:08Z

@@ -0,0 +1,33 @@
+com.fasterxml.jackson.core:jackson-annotations:2.21


As with 2.0, this file and the directories/packages in the Jar are the same as v2.1, so we should be able to use the same LICENSE and NOTICE.

ghost · 2026-04-24T20:23:02Z

@@ -0,0 +1,40 @@
+com.fasterxml.jackson.core:jackson-annotations:2.21


The only difference between v4.1 runtime deps and this is the Jackson version:

2,3c2,3 < com.fasterxml.jackson.core:jackson-core:2.15.2 < com.fasterxml.jackson.core:jackson-databind:2.15.2 --- > com.fasterxml.jackson.core:jackson-core:2.14.2 > com.fasterxml.jackson.core:jackson-databind:2.14.2

ghost · 2026-04-24T20:23:19Z

@@ -0,0 +1,40 @@
+com.fasterxml.jackson.core:jackson-annotations:2.21


Identical to the 4.1 deps.

ghost · 2026-04-24T20:23:25Z

@@ -0,0 +1,40 @@
+com.fasterxml.jackson.core:jackson-annotations:2.21


Identical to the 4.1 deps.

ghost · 2026-04-24T20:26:23Z

+org.threeten:threeten-extra:1.8.0
+org.threeten:threetenbp:1.7.0
+org.xerial.snappy:snappy-java:1.1.10.8
+software.amazon.awssdk.crt:aws-crt:0.43.9


Comparing this with aws-bundle's runtime deps, here are the deps in aws but not this bundle:

org.apache.logging.log4j:log4j-api org.apache.logging.log4j:log4j-core org.apache.logging.log4j:log4j-slf4j-impl software.amazon.awssdk:cloudwatch software.amazon.awssdk:cloudwatch-metric-publisher software.amazon.awssdk:s3control software.amazon.awssdk:smithy-rpcv2-protocol software.amazon.awssdk:third-party-jackson-dataformat-cbor software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin software.amazon.s3.analyticsaccelerator:analyticsaccelerator-s3

ghost · 2026-04-24T20:27:26Z

+com.github.ben-manes.caffeine:caffeine:2.9.3
+com.github.luben:zstd-jni:1.5.7-3
+com.github.pjfanning:jersey-json:1.22.0
+com.google.android:annotations:4.1.1.4


Comparing this with gcp-bundle, here are the deps in that bundle but not here:

com.google.api.grpc:proto-google-cloud-kms-v1 com.google.cloud.gcs.analytics:client com.google.cloud.gcs.analytics:gcs-analytics-core com.google.cloud:google-cloud-kms org.jspecify:jspecify

ghost · 2026-04-24T20:28:47Z

@@ -0,0 +1,233 @@
+com.azure:azure-core-http-netty:1.16.3


Comparing this with azure-bundle, here are the deps in that bundle that are not here:

com.azure:azure-security-keyvault-keys

ghost · 2026-04-24T20:37:08Z

+javax.xml.stream:stax-api:1.0-2
+net.java.dev.jna:jna-platform:5.17.0
+net.java.dev.jna:jna:5.17.0
+org.apache.arrow:arrow-format:17.0.0


Here are the dependencies in this runtime that aren't in any of the cloud provider bundles:

com.github.luben:zstd-jni com.github.pjfanning:jersey-json com.jcraft:jsch commons-cli:commons-cli commons-io:commons-io commons-net:commons-net commons-pool:commons-pool com.sun.xml.bind:jaxb-impl dev.failsafe:failsafe dnsjava:dnsjava io.airlift:aircompressor io.dropwizard.metrics:metrics-core jakarta.activation:jakarta.activation-api javax.servlet:javax.servlet-api javax.servlet.jsp:jsp-api javax.xml.bind:jaxb-api javax.xml.stream:stax-api org.apache.avro:avro org.apache.commons:commons-collections4 org.apache.commons:commons-compress org.apache.commons:commons-math3 org.apache.commons:commons-text org.apache.hadoop:hadoop-annotations org.apache.hadoop:hadoop-common org.apache.hadoop.thirdparty:hadoop-shaded-guava org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25 org.apache.httpcomponents.client5:httpclient5 org.apache.httpcomponents.core5:httpcore5 org.apache.httpcomponents.core5:httpcore5-h2 org.apache.orc:orc-core org.apache.orc:orc-shims org.apache.parquet:parquet-avro org.apache.parquet:parquet-column org.apache.parquet:parquet-common org.apache.parquet:parquet-encoding org.apache.parquet:parquet-format-structures org.apache.parquet:parquet-hadoop org.apache.parquet:parquet-jackson org.apache.parquet:parquet-variant org.bouncycastle:bcprov-jdk18on org.codehaus.jettison:jettison org.locationtech.jts:jts-core org.mongodb:bson org.roaringbitmap:RoaringBitmap org.xerial.snappy:snappy-java

ghost · 2026-04-24T21:21:00Z

+org.apache.parquet:parquet-variant:1.17.0
+org.checkerframework:checker-qual:3.19.0
+org.eclipse.collections:eclipse-collections-api:11.1.0
+org.eclipse.collections:eclipse-collections:11.1.0


This is a transitive dependency from arrow-vector:

+--- org.apache.arrow:arrow-vector:15.0.2 | +--- org.apache.arrow:arrow-format:15.0.2 | | \--- com.google.flatbuffers:flatbuffers-java:23.5.26 | +--- org.apache.arrow:arrow-memory-core:15.0.2 | +--- com.fasterxml.jackson.core:jackson-core:2.16.0 -> 2.15.2 (*) | +--- com.fasterxml.jackson.core:jackson-annotations:2.16.0 -> 2.21 | +--- com.fasterxml.jackson.core:jackson-databind:2.16.0 -> 2.15.2 (*) | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.16.0 -> 2.21.2 | | +--- com.fasterxml.jackson.core:jackson-annotations:2.21 | | +--- com.fasterxml.jackson.core:jackson-core:2.21.2 -> 2.15.2 (*) | | +--- com.fasterxml.jackson.core:jackson-databind:2.21.2 -> 2.15.2 (*) | | \--- com.fasterxml.jackson:jackson-bom:2.21.2 (*) | +--- com.google.flatbuffers:flatbuffers-java:23.5.26 | \--- org.eclipse.collections:eclipse-collections:11.1.0 | \--- org.eclipse.collections:eclipse-collections-api:11.1.0

This is missing from LICENSE.

Opened #16104 with the LICENSE and NOTICE updates.

Apparently, this was removed in later versions according to research by my Claude agent:

Eclipse Collections was added to Arrow Java in 15.0.0 (January 2024) and removed again in 16.0.0 (April 2024) — it was only present in 15.x releases (15.0.0, 15.0.1, 15.0.2).

This is not a blocker with the addition in LICENSE. But we should follow up and use a version of Arrow that doesn't include it if possible. And we should remove it from here.

ghost · 2026-04-24T21:21:20Z

+org.checkerframework:checker-qual:3.19.0
+org.eclipse.collections:eclipse-collections-api:11.1.0
+org.eclipse.collections:eclipse-collections:11.1.0
+org.eclipse.microprofile.openapi:microprofile-openapi-api:4.1.1


Transitive dependency from Nessie

ghost · 2026-04-24T22:17:03Z

Thanks, @kevinjqliu! I've merged this to get the checks running. There are now license updates or dependency changes to fix. See comments above.

add all runtime-deps

df2d4a9

kevinjqliu requested review from RussellSpitzer and rdblue April 22, 2026 21:46

github-actions Bot added spark flink AWS GCP AZURE KAFKACONNECT labels Apr 22, 2026

rdblue changed the title ~~ci: add all runtime-deps.txt files~~ CI: Add runtime-deps.txt files for all runtimes and bundles Apr 23, 2026

rdblue reviewed Apr 23, 2026

View reviewed changes

rdblue reviewed Apr 24, 2026

View reviewed changes

rdblue mentioned this pull request Apr 24, 2026

Spark 4.1: Update LICENSE and NOTICE for 1.11. #16104

Merged

rdblue approved these changes Apr 24, 2026

View reviewed changes

rdblue merged commit 65faa6f into apache:main Apr 24, 2026

ebyhr mentioned this pull request Apr 25, 2026

commit runtime-deps.txt to all artifacts #16080

Closed

adutra mentioned this pull request Apr 28, 2026

REST, OAuth2: OAuth2 AuthManager v2 - Part 1: Configuration #15703

Open

This was referenced Apr 28, 2026

Spark: Apply LICENSE and NOTICE updates to older Spark versions #16142

Merged

Kafka Connect: Update distribution LICENSE and NOTICE #16147

Closed

Flink: Apply LICENSE changes to older Flink versions #16159

Merged

kevinjqliu deleted the kevinjqliu/add-runtime-deps branch May 1, 2026 02:53

kevinjqliu mentioned this pull request May 6, 2026

GCP: Add CloudMonitoringMetricsReporter to export Iceberg metrics to Cloud Monitoring #16107

Closed

robert3005 mentioned this pull request May 11, 2026

Merge upstream changes spiraldb/iceberg#19

Merged

		@@ -0,0 +1,70 @@
		com.github.ben-manes.caffeine:caffeine:2.9.3

		@@ -0,0 +1,33 @@
		com.fasterxml.jackson.core:jackson-annotations:2.21

		@@ -0,0 +1,40 @@
		com.fasterxml.jackson.core:jackson-annotations:2.21

Conversation

kevinjqliu commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manuzhang commented Apr 23, 2026

Uh oh!

rdblue commented Apr 23, 2026

Uh oh!

ghost Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026 • edited by rdblue Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026 • edited by rdblue Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026 • edited by rdblue Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

ghost commented Apr 24, 2026

Uh oh!

ghost Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026 • edited by rdblue Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost Apr 24, 2026 • edited by rdblue Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

kevinjqliu commented Apr 22, 2026 •

edited

Loading

ghost Apr 23, 2026 •

edited by rdblue

Loading

ghost Apr 23, 2026 •

edited by rdblue

Loading

ghost Apr 24, 2026 •

edited by rdblue

Loading

ghost Apr 24, 2026 •

edited by rdblue

Loading

ghost Apr 24, 2026 •

edited by rdblue

Loading

ghost Apr 24, 2026 •

edited by rdblue

Loading