Core, Spark: Fix stale manifest_length in rewrite_table_path manifest lists by wombatu-kun · Pull Request #16910 · apache/iceberg

wombatu-kun · 2026-06-21T13:59:19Z

Problem

rewriteManifestList updated only the manifest path in each manifest-list entry and left manifest_length (field 501) at the source value. A target prefix of a different length changes the rewritten manifest's byte size, so readers that validate the field (Trino, Impala, iceberg-rust, and Iceberg itself on V4) fail with Incorrect file size.

Fix

The length is only known after the manifests are rewritten, but the action wrote the manifest lists first. rebuildMetadata now finds the manifests to rewrite through Snapshot#allManifests, rewrites them while ManifestWriter.length() records each byte length, and writes the manifest lists last.

RewriteResult carries those lengths and merges them in append, so rewriteDataManifest and rewriteDeleteManifest keep their signatures. rewriteManifestList is the only changed signature: the set of manifests to rewrite is replaced by a map from source manifest path to rewritten length, so a manifest cannot be selected for a rewrite that has no measured length. The released signature is changed in place rather than kept as a deprecated overload, since it cannot record a correct length. Two accepted breaks are recorded in .palantir/revapi.yml.

The changed signature had callers in every Spark tree, so the fix lands in 3.5, 4.0 and 4.1 together and no backport follows.

Known gap: incremental runs

A manifest carried over from an earlier increment is not rewritten again, so it keeps its source length, and each manifest list logs one info line when that happens. Closing the gap means rewriting and re-copying carried-over manifests, which is what #13720 proposed for #13719 and where its cost discussion stalled, so this PR is scoped to the complete-copy path.

Tests

assertManifestLengthsMatchOnDisk runs after copyTableFiles in every non-incremental test in TestRewriteTablePathsAction, asserting manifest_length against the on-disk size for every entry across current and historical snapshots (v2-v4). TestRewriteTablePathUtil pins the stamping, the source-path keying, the manifests the map selects for rewrite, and the append merge.

Thanks @vaultah for the incremental-mode catch.

AI Disclosure

Model: Claude Opus 5
Platform/Tool: Claude Code
Human Oversight: fully reviewed
Prompt Summary: Fix rewrite_table_path leaving the source manifest_length in the rewritten manifest lists.

leeyam24 · 2026-06-22T11:13:37Z


+    // rebuild manifest-list files last, stamping manifest_length with the rewritten manifest sizes
+    Set<RewriteResult<ManifestFile>> manifestListResults = Sets.newConcurrentHashSet();
+    Tasks.foreach(validSnapshots)


Note on performance: this loop currently defaults to single-threaded execution because PySpark/SQL cannot configure an Executor Service, directly causing slower runtimes for rewrite_table_path. I've raised a feature request to address this, and I believe it's an important bottleneck we should advocate to fix.

The threading here is unchanged by this PR - both the new discovery loop and the manifest-list loop reuse the action's existing executorService, the same as the original single loop did. Making that executor parallel by default is orthogonal to the manifest_length fix, so #16752 is the right place to address it.

Of course.
I assumed it was Out-Of-Scope, just wanted to point out a potential bottleneck.

leeyam24

I'm done reviewing, looks pretty good to me.

kevinjqliu · 2026-06-27T20:58:34Z

Thanks for the PR! I'll take a look. I reference this in #15470 (comment)

I think it would be good to include this in the upcoming patch release.

Could you rebase to resolve conflict? Might also be good to follow the pattern from #15470

wombatu-kun · 2026-06-29T02:17:23Z

Rebased onto main; the conflict with #15470 is resolved. The manifest_length fix follows the same pattern as #15470 - new overloads that record the measured length via writer.length(), threaded through a Map<String, Long> with a getOrDefault fallback, and @Deprecated on the prior signatures. Ready for another look.

vaultah · 2026-07-01T21:59:47Z

        ManifestFile newFile = file.copy();
        ((StructLike) newFile).set(0, newPath(newFile.path(), sourcePrefix, targetPrefix));
+        ((StructLike) newFile)
+            .set(1, rewrittenManifestLengths.getOrDefault(file.path(), file.length()));


This seems like an intentional decision, based on

Manifests not rewritten in an incremental run keep their original length.

but it does still mean that this procedure will silently produce incorrect manifest lists in incremental mode, if rewritten manifest lists referenced manifests that were already rewritten in a previous run.

If the rewritten manifest size changed after the prefix rewrite, which it typically does, the new manifest list is still inconsistent with the file it references.

This is a known incorrect behaviour described in #13720 (comment) and acknowledged there.

You're right - this is the same bug as your #13719 (#16905 is a later duplicate), and your #13720 went further than this PR: by rewriting every referenced manifest and requiring all lengths to be known, it kept the incremental case correct too. This PR keeps the incremental optimization (only new manifests are rewritten), so carried-over manifests fall back to file.length() and their manifest_length stays stale - the exact A/B/C case you raised with @dramaticlly. So the non-incremental path is fixed here but the incremental path is not, and closing it means re-rewriting all referenced manifests, the efficiency tradeoff that stalled #13720 at the community sync. I'd rather settle that completeness-vs-efficiency question on #13719 / dev@ than decide it unilaterally here, and I'm glad to help revive your #13720. Even as-is this PR is a strict improvement over main, where every entry kept the source length.

You're right, incremental runs are still inconsistent for carried-over manifests. This PR is now scoped to the complete-copy path, and the javadoc names the incremental case as a known gap rather than presenting it as intended behaviour. Leaving it for a follow-up.

github-actions · 2026-08-10T00:18:35Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

varun-lakhyani

Mostly looks good, Can you please check the comments @wombatu-kun
couple of them are concerns or I am just wondering, others are nits.
I will take another look at the tests soon.

Thanks for this, I am facing this issue while working on Eager fetch of parquet files, it changes how files are read and needs the correct length, so the stale manifest_length breaks it. TestRewriteTablePathsAction is the only test failing for me because of this as it reads new manifest with incorrect lengths

varun-lakhyani · 2026-08-15T17:59:06Z

+    // Read every valid snapshot's manifest list once, before writing anything. The rewritten
+    // manifest list must record each referenced manifest's rewritten length (manifest_length),
+    // which is only known after the manifests are rewritten below.
+    Map<Snapshot, List<ManifestFile>> manifestsBySnapshot = Maps.newConcurrentMap();


nit. maybe just Read every valid snapshot's manifest list works.
i think the reason for reordering doesn't belong here

Done 90d25bf. The step list in the rebuildMetadata javadoc still described the old order, so the ordering now lives there instead.

varun-lakhyani · 2026-08-15T17:59:29Z

+    // Manifests selected for rewrite. In an incremental run this is only the manifests added by the
+    // delta snapshots, so a manifest carried over from an earlier run is not rewritten here and
+    // keeps its source length in the manifest list. See the note on rewriteManifestList.
+    Set<ManifestFile> manifestFiles = Sets.newHashSet();


nit maybe code is self explaining here, if required this can be made one liner

Done 90d25bf

varun-lakhyani · 2026-08-15T18:00:12Z

            manifestFiles,
            sparkContext().broadcast(rewrittenDeleteFileSizes));

+    // rebuild manifest-list files last, stamping manifest_length with the rewritten manifest sizes


nit
similar maybe just keep rebuild manifest-list files,
updated manifest file size is the base functionality we might not need to specify separately

Done 90d25bf

varun-lakhyani · 2026-08-15T18:27:10Z

+   * @param manifestFiles the manifests the snapshot's manifest list references
   * @param manifestsToRewrite filter of manifests to rewrite.
+   * @param rewrittenManifestLengths map from source manifest path to its rewritten byte length


I would make these three same as rewriteManifestList in Util.
https://github.com/apache/iceberg/pull/16910/changes#diff-285b4520a0f058b7fe2a0103efe3f15c0021a6acd9031cdc22c3ddae6151b467R299-R307

Done 90d25bf

varun-lakhyani · 2026-08-15T19:13:53Z

+   * <p>Every entry keeps its source {@code manifest_length}, which does not match the rewritten
+   * manifest when the target prefix differs in length from the source. Callers that rewrite to a
+   * different-length prefix should use {@link #rewriteManifestList(Snapshot, FileIO, TableMetadata,
+   * List, Set, String, String, String, String, Map)} and pass the measured lengths.
+   */
+  public static RewriteResult<ManifestFile> rewriteManifestList(


Can we make it like @Deprecated and @deprecated since 1.12.0, will be removed in 1.13.0; use the overload....

The stale-size rewriteDeleteManifest in the same file is @Deprecated for the same reason (#15470).

The
note above the method already routes callers to the overload that takes the lengths. I left @deprecated off because it commits core to removing the method in a named release, which is a committer's call rather than something to fold into a bug fix - #15470's tag was set by that PR's author for their own method. Happy to add it if a committer wants it here.

Sure, I am fine with waiting to understand what commiters think.
I guess as this function returns stale lengths even when we have the updated is not the function we would like to keep.

Deprecate or even just completely remove. This code is wrong and corrupts data. Our own readers will fail on manifest-lists produced by this method in V4.

The behavior of "keeping the existing length" is wrong. We shouldn't support it. If we wanted to keep this overload in a deprecated state I would argue we should also be checking the lengths on disks of all the manifests in the staging directory and assigning based on that. If the manifest isn't in the staging directory we can fail. The deprecated version would be much less performant but would at least be correct.

Done c0337be. Removed rather than deprecated. The break is recorded in .palantir/revapi.yml.

On "we shouldn't support it" more generally: a manifest this run rewrote can no longer fall back to its source length either, that now fails. The one case left is a manifest carried over from an earlier incremental run, which this run deliberately does not rewrite, so there is nothing to measure. Closing that means rewriting carried-over manifests again and re-copying them so the target file matches the recorded length, which is what #13720 proposed for #13719 and where its cost discussion stalled.

varun-lakhyani · 2026-08-15T19:46:22Z

+   * @return the manifests referenced by the snapshot's manifest list, or an empty list if the
+   *     manifest list could not be read
+   */
+  public static List<ManifestFile> manifestsInSnapshot(


Confusing with manifestFilesInSnapshot, but this one also validates via Preconditions and throws, and both stay.
Maybe something like manifestFilesInSnapshotValidatingPrefix so the difference is visible at the callsite?

Done 90d25bf. manifestFilesInSnapshot had a single caller, so it is inlined and only manifestsInSnapshot remains.

I'm also not sure why this is public and the java doc is extremely specific to it's current usage. That's another hint to me that this is not actually a public method. I feel like we are exposing public API to fix implementation details.

Done c0337be. manifestsInSnapshot is gone; the action uses the existing Snapshot#allManifests(FileIO).

varun-lakhyani · 2026-08-15T19:54:32Z

+   * @see #rewriteDataManifestAndMeasureLength which also returns the rewritten manifest length, for
+   *     callers that need to record manifest_length in the manifest list
   */
  public static RewriteResult<DataFile> rewriteDataManifest(


Should this get @Deprecated like the old 9-arg rewriteDeleteManifest?

Same answer as the rewriteManifestList thread: @deprecated commits core to a removal release, which is a committer's call rather than part of a bug fix. The javadoc already points at the overload that returns the length.

Not needed as of c0337be: rewriteDataManifest keeps its signature and records the rewritten length in the result.

varun-lakhyani · 2026-08-15T20:11:32Z

+      result =
+          StreamSupport.stream(reader.entries().spliterator(), false)
+              .map(
+                  entry ->
+                      writeDataFileEntry(
+                          entry, snapshotIds, spec, sourcePrefix, targetPrefix, writer))
+              .reduce(new RewriteResult<>(), RewriteResult::append);
    }
+    return Pair.of(result, writer.length());


Just wondering - did you consider carrying the length on RewriteResult instead of returning a Pair? it would merge in append the same way RewriteContentFileResult does, and rewriteDataManifest / rewriteDeleteManifest could keep their names and signatures — no new methods or deprecations.
Might not be worth at this point, just wondering if it was considered.

similar to #13720

Yes, and I implemented it to check. It works: rewriteDataManifest / rewriteDeleteManifest keep their signatures and stamp the length after the reduce, and append needs no change because it is set once rather than accumulated. What killed it is that adding a field to RewriteResult changes its default serialVersionUID, so revapi reports java.class.defaultSerializationChanged and the shape only lands with a permanent accepted-break entry in .palantir/revapi.yml. Two extra methods looked like the cheaper price, so I kept the Pair.

Okay makes sense to me

Why would we care about breaking the SerialVersionUID? I can't think of a serialization boundary for this object that we want to preserve?

I think rewrite result is probably the wrong place to store it, but I also think having a pair with two very different types of return type is showing the strains on this api. It may make sense to try to just wrap OutputFile so that getLength is a noOp.

I actually changed my mind, I think this is probably a good place to store the length...

Done c0337be. RewriteResult carries a map from source manifest path to rewritten length and merges it in append.

varun-lakhyani · 2026-08-15T20:50:23Z

+        targetPrefix,
+        stagingDir,
+        outputPath,
+        Map.of(measured.path(), rewrittenLength));


nit
ImmutableMap just to be consistent?

Done 90d25bf

varun-lakhyani · 2026-08-15T20:59:57Z

+    ManifestFile unmeasured = manifests.get(1);
+
+    String manifestPath = measured.path();
+    String sourcePrefix = manifestPath.substring(0, manifestPath.indexOf("/metadata/"));


nit lastIndexOf instead of indexOf?

Done 90d25bf

varun-lakhyani · 2026-08-15T22:04:10Z

@wombatu-kun I think separating spark v3.5, v4.0 changes as backport afterwards will be good and make it easy for reviewers

wombatu-kun · 2026-08-16T13:22:17Z

Done 6e3e691 - the PR is now core plus spark/v4.1 only. The 3.5 and 4.0 backport goes up as a separate PR once this one merges.

varun-lakhyani · 2026-08-16T16:45:05Z

+      Map<String, Long> rewrittenManifestLengths) {
+    manifestFiles.stream()
+        .filter(file -> rewrittenManifestLengths.get(file.path()) == null)
+        .forEach(
+            file ->
+                LOG.warn(
+                    "No measured length for manifest {}; recording its source length {}, which "
+                        + "will not match the rewritten file at the target",
+                    file.path(),
+                    file.length()));
+
+    return writeManifestList(


I am not feeling really good about this — 2 overloaded rewriteManifestList functions further calling a 3rd writeManifestList, specifically for the log warning.
I understand you did this so the original 8-arg function doesn't put everything into log warns.

If we agree to put @Deprecated on the 8-arg, it can just call the 10-arg with ImmutableMap.of() and warn on all paths — just like the previous implementation, only without the isEmpty check.

Nit : And instead of one warning per manifest, we could aggregate to a single line per manifest list (e.g. "N of M manifests keep their source length"), so incremental runs don't pile up a warning per file per snapshot.

Not blocking from my side — happy to go either way if others feel differently.

Done c0337be

varun-lakhyani · 2026-08-16T16:49:05Z

@huaxingao @stevenzwu @dramaticlly As you guys were involved in older PR #13720 for this fix, Can you please take a look at this one.

varun-lakhyani · 2026-08-17T11:27:10Z

Just for reference — the flow before this PR and how it changes, so the correct manifest length is available at manifest-list write time.

RussellSpitzer · 2026-08-17T18:58:38Z

+   * <p>Each entry's {@code manifest_length} is taken from {@code rewrittenManifestLengths}, because
+   * replacing the path prefix changes the byte length of the rewritten manifest.
+   *
+   * <p>Known gap: an entry whose manifest is absent from the map keeps its source length, which


I'm not sure this additional java doc is very helpful. It's pretty deep in the API (an actual user will probably never get here.) We would want to keep this note if we actually wanted it in the RewriteTablePath API and not in this utility class. You can tell this is in the wrong spot because it's referring to "incremental runs" which don't actually exist in this utility class.

Done c0337be. Dropped.

Moved it to RewriteTablePath#startVersion in 4b8369e

RussellSpitzer · 2026-08-17T19:17:28Z

+    // manifest-list rewrite to record an accurate manifest_length.
+    private final Map<String, Long> rewrittenManifestLengths = Maps.newHashMap();
+
+    public Map<String, Long> rewrittenManifestLengths() {


nit but this should probably be an unmodifiable map. We are currently exposing this for read and write. So we have this pattern of

Result.getMap.put

Instead of

Result.addLength(manifestPath, length) Or what not

Done c0337be

RussellSpitzer · 2026-08-17T22:07:22Z

From architectural prospective. I think we should avoid adding new apis, and I think we can do so here by just changing the rewriteManifestList and nothing else.

I checked out @varun-lakhyani 's proposal as well,

It looks like Varun's pays a little bit in IO for each manifest which is pretty straight forward but expensive

This Pr returns the length along with the result object but changes a some return signatures and adds new methods to do it.

Ideally I think we would want to change RewriteResult so that we can keep the methods as they are, the revapi breaks from disrupting the existing APIs are I think warranted because they basically cannot be used correctly as is.

… lists Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…th manifest lists Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

…ength Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

wombatu-kun · 2026-08-18T08:02:19Z

I think we should avoid adding new apis, and I think we can do so here by just changing the rewriteManifestList and nothing else.

Ideally I think we would want to change RewriteResult so that we can keep the methods as they are, the revapi breaks from disrupting the existing APIs are I think warranted because they basically cannot be used correctly as is.

c0337be takes that shape. Since the old signature is gone, the fix is back in all three Spark trees rather than 4.1 alone.

wombatu-kun · 2026-08-18T08:14:31Z

@varun-lakhyani the trim is reverted in c0337be: removing the 8-argument rewriteManifestList leaves v3.5 and v4.0 uncompilable, so the fix has to land in all three trees together.

RussellSpitzer · 2026-08-18T15:39:10Z

                mf.path(),
                sourcePrefix));

+    long unmeasured =


I think basically we probably want to say that all rewrittenManifests must have a defined length. If a manifest path is changed, then it's length must also change. Anything else should hard fail

Something like

Preconditon
rewrittenManifestLengths.containsAll(manifestsToRewrite)

Done 4b8369e, scoped to manifestsToRewrite as in your second comment.

On this point, maybe we should just change the API entirely so that we only take this map of manifest to length. We only want to be able to rewrite entries with a length, so we may as well just drop the Set and just have Map<Manifest, Length> be the only thing we allow passing through. Then there is no ability to rewrite a manifest unless you also pass through a length.

dropping set as is could harm us.
We are iterating over List<ManifestFile> manifestFiles = manifestFilesInSnapshot(io, snapshot); which is source of manifest that might not have updated length in the map and only thing helping us fail hard using precondition is Set manifestsToRewrite which contains all that should have new length and map which contains all that have recorded new length.

If we drop set we might need precondition at some point else - while iteration -manifestFiles against the length map

If we want real fail hard - fail at all incremental rewrites too then we can drop set and put the precondition on manifestinsnapshot against the map.
This way we are 100% sure there is no way we are getting wrong length but we will loose incremental rewrite completely as it fails everytime.

Done 6f73ff1.

@varun-lakhyani over the manifests this method iterates the two are the same predicate: rebuildMetadata filters snapshot.allManifests with that set, and every manifest it then rewrites records its length on RewriteResult, so the only way into the map is having been rewritten. Carried-over manifests stay outside it by design and keep their source length.

Yeah I missed an important thing earlier - That precondition never belonged in function that's called per snapshot , when both Map and set used in it were global (same everytime). It always checked the exact same thing for each snapshot.
Agreed to remove set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

RussellSpitzer · 2026-08-24T15:06:22Z

+                measuredTargetPath));
+  }
+
+  @TestTemplate


How is this related to this change?

Done fdb2422. Removed - that precondition predates this PR, so the test was unrelated coverage.

RussellSpitzer · 2026-08-24T15:11:58Z

  }

+  @TestTemplate
+  public void testManifestLengthAfterRewrite() throws Exception {


We don't really nee individual tests for this. Every test we do that rewrites manifests should also check to see that the lengths are the same. You already have this in a helper assertManifestLengthsMatchOnDisk which we should just be calling on all of the tests (except Incremental rewrites). Then we can remove tests that are just checking this invariant which we expect to be true all the time.

https://github.com/apache/iceberg/pull/17664/changes#diff-59aa6d10abf9d930dbfe52d740ebd436610721ee31e0b999c7bed6bf0c05cd3cR1807-R1813 maybe this could help

Done fdb2422. assertManifestLengthsMatchOnDisk now runs after copyTableFiles in all 13 non-incremental tests and the dedicated test is gone.

RussellSpitzer · 2026-08-24T15:13:26Z

+  }
+
+  /** Guards the length assertions against a prefix change that moved no manifest's byte size. */
+  private void assertRewriteChangedManifestLength(Table sourceTable, Table targetTable) {


We don't really need to do this. The positive invariant (the new length matches) is enough.

Done fdb2422

RussellSpitzer · 2026-08-24T15:20:53Z

                targetPrefix,
                rewrittenDeleteFileSizes),
-            Encoders.bean(RewriteContentFileResult.class))
+            // Encoders.bean derives an empty schema here and would drop the copy plan and lengths


This feels unrelated to this change, and currently has no tests that break with it unchanged. I would change it back and have a follow up if we want to make a defensive change here. But as is I don't think this is actually broken.

Done fdb2422

RussellSpitzer

I have a few nits left. Let's see if we can keep the test coverage tight here and make sure there are no other changes leaking into this PR for unrelated work.

RussellSpitzer · 2026-08-24T15:25:22Z

For follow ups we probably need to do the following

Disable incremental mode on V4, it will never be correct. We can do this at the action level. We should fail and fail hard.
Add a warning for non-incremental mode on V4 that users should switch to using relative paths.

…ath tests Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

RussellSpitzer · 2026-08-25T21:22:49Z

Merged, Thanks @wombatu-kun for the PR and @varun-lakhyani for the review.

* Build: Bump actions/stale from 10.4.0 to 11.0.0 (#17576) Bumps [actions/stale](https://github.com/actions/stale) from 10.4.0 to 11.0.0. - [Release notes](https://github.com/actions/stale/releases) - [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/stale/compare/1e223db275d687790206a7acac4d1a11bd6fe629...4391f3da665fdf50b6810c1a66712fb9ba21aa93) --- updated-dependencies: - dependency-name: actions/stale dependency-version: 11.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump zizmorcore/zizmor-action from 0.6.1 to 0.6.2 (#17575) Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.1 to 0.6.2. - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](https://github.com/zizmorcore/zizmor-action/compare/6fc4b006235f201fdab3722e17240ab420d580e5...3dc1ecc9bcb9e94e9b2c709687979e1298497054) --- updated-dependencies: - dependency-name: zizmorcore/zizmor-action dependency-version: 0.6.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump gradle/actions/setup-gradle from 6.2.0 to 6.3.0 (#17574) Bumps [gradle/actions/setup-gradle](https://github.com/gradle/actions) from 6.2.0 to 6.3.0. - [Release notes](https://github.com/gradle/actions/releases) - [Commits](https://github.com/gradle/actions/compare/3f131e8634966bd73d06cc69884922b02e6faf92...9c971963bec38e04b3d30dcc455b5382be2fdbfb) --- updated-dependencies: - dependency-name: gradle/actions/setup-gradle dependency-version: 6.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump actions/setup-java from 5.6.0 to 5.7.0 (#17573) Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5.6.0 to 5.7.0. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](https://github.com/actions/setup-java/compare/03ad4de0992f5dab5e18fcb136590ce7c4a0ac95...b6effb05e454b25005698d916606bdc6ffcbf961) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: 5.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 8.8.0 to 8.9.0 (#17569) Build: Bump com.diffplug.spotless:spotless-plugin-gradle Bumps [com.diffplug.spotless:spotless-plugin-gradle](https://github.com/diffplug/spotless) from 8.8.0 to 8.9.0. - [Release notes](https://github.com/diffplug/spotless/releases) - [Changelog](https://github.com/diffplug/spotless/blob/main/CHANGES.md) - [Commits](https://github.com/diffplug/spotless/compare/gradle/8.8.0...gradle/8.9.0) --- updated-dependencies: - dependency-name: com.diffplug.spotless:spotless-plugin-gradle dependency-version: 8.9.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump the codeql-action group with 3 updates (#17572) Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.3 to 4.37.4 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81...f205ea1c3313d32999d8d6a48b4f6530d4437b38) Updates `github/codeql-action/analyze` from 4.37.3 to 4.37.4 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81...f205ea1c3313d32999d8d6a48b4f6530d4437b38) Updates `github/codeql-action/upload-sarif` from 4.37.3 to 4.37.4 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81...f205ea1c3313d32999d8d6a48b4f6530d4437b38) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.37.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.37.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Data, Spark: Support byte[] binary values in InternalRowConverter (#17482) * Flink: Reuse the catalog in the dynamic sink serializer cache (#17437) * Flink: Fix read split table properties being ignored (#17445) * Flink: Remove unused logger in TestIcebergCommitter (#17587) * Flink: Backport dynamic sink serializer cache catalog reuse to 2.0 and 1.20 (#17437) (#17590) * Flink: Avoid per-row field getter allocation in RowDataWrapper (#16789) * Flink: Backport read split table properties fix to 2.0 and 1.20 (#17445) (#17591) * Flink: Backport avoid per-row field getter allocation in RowDataWrapper (#17606) Backports #16789 * Core, REST: Fix delete file references for DVs in the same Puffin file (#17497) Motivation: Scan planning responses index delete files by position in the delete-files array. The index map was keyed on the delete file location alone, but a single Puffin file can hold a deletion vector for each of several data files. Every DV in that file shares a location, so each new DV entry overwrote the previous one and every task referencing a DV in that Puffin file resolved to the same last index. Changes: Extract DeleteFileSet's private wrapper into a public DeleteFileWrapper, mirroring CharSequenceWrapper, and key the index map on it so serialization uses the same delete file identity that builds the delete-files array. Fail with a clear message when a task references a delete file that is not in that array. * Docs: Document missing read table properties in configuration (#17554) Add read.split.adaptive-size.enabled, read.data-planning-mode, and read.delete-planning-mode to the read properties table with their defaults and accepted values. * Core: Support metrics with nested field in AppendBenchmark and FileGenerationUtil (#17391) * Core: Add nested field traversal in AppendBenchmark and FileGenerationUtil * expand testBoundsForNestedTypes to cover all metricsMode * Core: Resolve metrics mode by field id in FileGenerationUtil Use MetricsConfig.columnMode(int) instead of manually resolving the column name via Schema.findColumnName. The int overload performs the same id-to-name lookup internally and correctly handles column aliases, removing duplicated logic across the generator and its test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix schema used in UT --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Core: Include CRC values in DV deserialization error message (#17393) * Site: Add blog post introducing the Variant type (#17537) * Core: Add max-file-group-input-files to valid rewrite options (#17544) max-file-group-input-files is read and validated by SizeBasedFileRewritePlanner but was missing from validOptions(), so RewriteDataFilesSparkAction rejected it with "Cannot use options [max-file-group-input-files]". Flink exposes the option through RewriteDataFiles.maxFileGroupInputFiles() where it works, while Spark rejected it. The option was added in #14837 without updating validOptions(). Add the option to validOptions() and cover it with a rewrite that sets it, which fails without this change, plus a planner test that it caps the number of input files per group. * API: Respect plaintext files in EncryptingFileIO bulkDecrypt (#17592) * Respect plaintext files in EncyptingFileIO bulkDecrypt * unit tests --------- Co-authored-by: Thomas Powell <tpowell@palantir.com> * Core: Fix AvroEncoderUtil header error printing literal %02X (#17125) AvroEncoderUtil.decode validated the magic bytes with Preconditions.checkState(cond, "Unrecognized header bytes: 0x%02X 0x%02X", header0, header1). Guava's Preconditions formats messages with Strings.lenientFormat, which only understands the %s placeholder, so the %02X specifiers were emitted verbatim and the two byte arguments were appended as decimal in a trailing bracket. A mismatched header therefore reported "Unrecognized header bytes: 0x%02X 0x%02X [16, 32]" instead of the intended hex, obscuring the actual bytes. * Flink: Remove deprecated RewriteDataFiles.Builder.filter(Expression) (#17624) * Core: Fix Z-order byte encoding for floating-point values (#17071) Thanks @eye-gu for the Fix! * Parquet: Fix notNaN for columns missing from files (#17633) * Spark: Remove deprecated and unused methods in SparkRead/WriteConf and SparkSchemaUtil (#17626) Spark: Remove deprecated methods scheduled for 1.12.0 removal - SparkSchemaUtil.convertWithFreshIds(Schema, StructType) and prune(Schema, StructType, List<Expression>) in Spark 4.0 and 4.1 - SparkReadConf/SparkWriteConf constructors taking an unused branch argument in Spark 4.1 Spark 3.5 is unaffected: these overloads were never deprecated there. * Site: Update the variant blog with Apache Arrow variant support (#17640) * Data: Add Variant Support for file format tck (#17617) * Core: Call close on iterables returned by ManifestReader.liveEntries() in ManifestFilterManager (#15713) * Parquet, Spark, Flink: Type uniformity for variant shredding (#17424) * API: Avoid intermediate collections in IN predicate metrics evaluation (#17659) * Core: Always track null count in content stats, add evaluator (#17413) * API: Avoid intermediate collections in NOT IN strict metrics evaluation (#17661) * Core: Use TrackedFile constants for partition and content_stats fields (#17649) Replace the duplicated field ID and name literals with the constants that TrackedFile already declares, which both files use elsewhere. * Kafka Connect: Remove deprecated TableReference and IcebergWriterResult members (#17623) Deprecated for removal in 1.12.0: - TableReference.of(String, TableIdentifier) and the TableReference(String, List<String>, String) constructor - IcebergWriterResult(TableIdentifier, List, List, StructType) and IcebergWriterResult.tableIdentifier() Callers now construct a TableReference with an explicit table UUID; the two tests pass null to preserve their previous behavior. * BigQuery: Remove deprecated catalog property constants (#17625) PROJECT_ID, GCP_LOCATION and LIST_ALL_TABLES on BigQueryMetastoreCatalog were deprecated for removal in 1.12.0 in favor of the equivalents on BigQueryProperties. No remaining callers. * API: Add Javadoc for the Metrics constructor parameters (#17608) * API: Add Javadoc for the Metrics constructor parameters Document each parameter of the canonical Metrics constructor, including that the maps and counts may be null when unknown. * Update api/src/main/java/org/apache/iceberg/Metrics.java Co-authored-by: Anoop Johnson <git@anoopjohnson.com> * Fix format * Followup changes --------- Co-authored-by: Anoop Johnson <git@anoopjohnson.com> * Build: Bump datamodel-code-generator from 0.71.0 to 0.72.2 (#17673) Bumps [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) from 0.71.0 to 0.72.2. - [Release notes](https://github.com/koxudaxi/datamodel-code-generator/releases) - [Changelog](https://github.com/koxudaxi/datamodel-code-generator/blob/main/CHANGELOG.md) - [Commits](https://github.com/koxudaxi/datamodel-code-generator/compare/0.71.0...0.72.2) --- updated-dependencies: - dependency-name: datamodel-code-generator dependency-version: 0.72.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.6.3 to 5.6.4 (#17674) Build: Bump org.apache.httpcomponents.client5:httpclient5 Bumps [org.apache.httpcomponents.client5:httpclient5](https://github.com/apache/httpcomponents-client) from 5.6.3 to 5.6.4. - [Changelog](https://github.com/apache/httpcomponents-client/blob/rel/v5.6.4/RELEASE_NOTES.txt) - [Commits](https://github.com/apache/httpcomponents-client/compare/rel/v5.6.3...rel/v5.6.4) --- updated-dependencies: - dependency-name: org.apache.httpcomponents.client5:httpclient5 dependency-version: 5.6.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump at.yawk.lz4:lz4-java from 1.11.1 to 1.11.2 (#17675) Bumps [at.yawk.lz4:lz4-java](https://github.com/yawkat/lz4-java) from 1.11.1 to 1.11.2. - [Release notes](https://github.com/yawkat/lz4-java/releases) - [Changelog](https://github.com/yawkat/lz4-java/blob/main/CHANGES.md) - [Commits](https://github.com/yawkat/lz4-java/compare/v1.11.1...v1.11.2) --- updated-dependencies: - dependency-name: at.yawk.lz4:lz4-java dependency-version: 1.11.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump org.roaringbitmap:RoaringBitmap from 1.6.18 to 1.6.20 (#17677) Bumps [org.roaringbitmap:RoaringBitmap](https://github.com/RoaringBitmap/RoaringBitmap) from 1.6.18 to 1.6.20. - [Release notes](https://github.com/RoaringBitmap/RoaringBitmap/releases) - [Commits](https://github.com/RoaringBitmap/RoaringBitmap/commits) --- updated-dependencies: - dependency-name: org.roaringbitmap:RoaringBitmap dependency-version: 1.6.20 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump jetty from 12.1.11 to 12.1.12 (#17678) Bumps `jetty` from 12.1.11 to 12.1.12. Updates `org.eclipse.jetty.compression:jetty-compression-server` from 12.1.11 to 12.1.12 Updates `org.eclipse.jetty.compression:jetty-compression-gzip` from 12.1.11 to 12.1.12 Updates `org.eclipse.jetty.ee10:jetty-ee10-servlet` from 12.1.11 to 12.1.12 --- updated-dependencies: - dependency-name: org.eclipse.jetty.compression:jetty-compression-server dependency-version: 12.1.12 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.eclipse.jetty.compression:jetty-compression-gzip dependency-version: 12.1.12 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.eclipse.jetty.ee10:jetty-ee10-servlet dependency-version: 12.1.12 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump io.netty:netty-buffer from 4.2.16.Final to 4.2.17.Final (#17679) Bumps [io.netty:netty-buffer](https://github.com/netty/netty) from 4.2.16.Final to 4.2.17.Final. - [Release notes](https://github.com/netty/netty/releases) - [Commits](https://github.com/netty/netty/compare/netty-4.2.16.Final...netty-4.2.17.Final) --- updated-dependencies: - dependency-name: io.netty:netty-buffer dependency-version: 4.2.17.Final dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump the codeql-action group with 3 updates (#17681) Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.4 to 4.37.6 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...5595ccaf912efad79be6eef63a5619ff05969be3) Updates `github/codeql-action/analyze` from 4.37.4 to 4.37.6 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...5595ccaf912efad79be6eef63a5619ff05969be3) Updates `github/codeql-action/upload-sarif` from 4.37.4 to 4.37.6 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...5595ccaf912efad79be6eef63a5619ff05969be3) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Docs: Update Apache Fluss documentation link to correct path (#17647) * Arrow: Fix direct memory leak in row lineage vectorized readers (#17296) Co-authored-by: Angel Conde <acmanjon@amazon.com> * Build: Bump software.amazon.awssdk:bom from 2.50.2 to 2.51.3 (#17686) Bumps software.amazon.awssdk:bom from 2.50.2 to 2.51.3. --- updated-dependencies: - dependency-name: software.amazon.awssdk:bom dependency-version: 2.51.3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump com.google.cloud.gcs.analytics:gcs-analytics-core from 1.4.0 to 1.5.0 (#17687) Build: Bump com.google.cloud.gcs.analytics:gcs-analytics-core Bumps [com.google.cloud.gcs.analytics:gcs-analytics-core](https://github.com/GoogleCloudPlatform/gcs-analytics-core) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/GoogleCloudPlatform/gcs-analytics-core/releases) - [Changelog](https://github.com/GoogleCloudPlatform/gcs-analytics-core/blob/main/CHANGELOG.md) - [Commits](https://github.com/GoogleCloudPlatform/gcs-analytics-core/compare/v1.4.0...v1.5.0) --- updated-dependencies: - dependency-name: com.google.cloud.gcs.analytics:gcs-analytics-core dependency-version: 1.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Parquet: Fix notStartsWith skipping row groups with null values (#17656) The dictionary row group filter only sees non-null values, so a column whose dictionary entries all start with the prefix was reported as unmatchable even when the column contained nulls. Null values match notStartsWith, so those row groups were skipped and their rows dropped from scan results. * API: Extract superclass from StrictMetricsEvaluator (#17511) Extract the strict evaluation logic from StrictMetricsEvaluator into a new superclass, StrictEvalVisitor. StrictMetricsEvaluator now uses the visitor superclass and implements methods to access metrics by ID. This is in preparation for adding a strict stats visitor that consumes ContentStats directly. * Core: Preserve snapshot pruning when ignoring residuals in AllManifestsTable (#17522) * Core: Simplify PartitionData.hashCode to avoid per-call allocations (#17666) * Core: Simplify list handling in manifest read and write paths (#17581) * Spark: Remove deprecated SparkFilters (#17702) SparkFilters converted Spark DSv1 filters into Iceberg expressions and was deprecated as a class for removal in 1.12.0 in favor of SparkV2Filters, which converts DSv2 predicates. Nothing uses it. The only remaining references were its own test, TestSparkFilters, which is removed with it, and a TODO comment in TestRewriteDataFilesProcedure that named the class; that comment now points at SparkV2Filters. Applies to Spark 3.5, 4.0 and 4.1. * Docs: Align Flink docs with current engine (#17695) * Data: Remove deprecated GenericAppenderFactory and BaseFileWriterFactory (#17696) * Spark: Reduce TestRewriteDataFilesAction data volume to speed up CI (#16565) * Build: Remove unused Avro dependency from iceberg-orc (#17694) * Core: Treat path prefixes as literals in RewriteTablePathUtil.replacePaths (#17521) * Spark 4.0/4.1: Remove deprecated SparkTableUtil methods (#17703) * Docs: Fix "refer" grammar in maintenance, AWS, and Nessie docs (#17580) * Data, Kafka Connect: Enable Parquet variant shredding for Kafka Connect and generic Record writes (#17520) * Core: Remove deprecated methods and fields scheduled for 1.12.0 removal (#17700) * Core: Remove deprecated methods and fields scheduled for 1.12.0 removal Removes the remaining core API marked for removal in 1.12.0 that has no callers left: - SystemProperties (class) -> SystemConfigs - ThreadPools.newWorkerPool(String) and newWorkerPool(String, int) -> newExitingWorkerPool or newFixedThreadPool depending on pool lifetime - ContentCache.invalidateAll(), which was only best-effort and racy - TableProperties.MANIFEST_LISTS_ENABLED and MANIFEST_LISTS_ENABLED_DEFAULT; writing manifest lists is always enabled - AvroSchemaUtil.pruneColumns(Schema, Set, NameMapping) -> applyNameMapping plus pruneColumns(Schema, Set) - SnapshotUtil.newFiles and newFilesBetween -> SnapshotChanges with ancestorsBetween - TableScanUtil.hasDeletes(CombinedScanTask) and hasEqDeletes(CombinedScanTask) - StandardEncryptionManager(String, int, KeyManagementClient), wrapKey and unwrapKey - ManifestFiles.read(ManifestFile, FileIO), readPaths(ManifestFile, FileIO) and open(ManifestFile, FileIO) -> the overloads taking a specs map - BaseScan.io() -> table().io(); DataScan and BaseDistributedDataScan are migrated to call table().io() directly Generated-by: Claude Code * rename unit test in TestManifestReader * Docs: Align Spark docs with current engine (#17688) * AWS: Remove deprecated S3 signer classes and properties (#17627) AWS: Remove deprecated S3 signer request/response classes These five classes were deprecated for removal in 1.12.0 and are now orphaned: they reference only each other and have no remaining callers in main or test sources. S3V4RestSignerClient uses the RemoteSign* equivalents. - S3SignRequest, S3SignResponse - S3SignRequestParser, S3SignResponseParser - S3ObjectMapper * Build: Fix OpenAPI test fixtures runtime dependency (#17519) * Core: Remove deprecated DataReader in favor of PlannedDataReader (#17699) DataReader was deprecated for removal in 1.12.0; PlannedDataReader is the replacement. Also removes the deprecated RawDecoder constructor that took a read schema and reader function, superseded by RawDecoder.create(Schema, Function, Schema). No remaining references to either. TestDataReader is removed along with the class it covered; PlannedDataReader has its own coverage in TestPlannedDataReader. * API: Add tests for BinaryUtil (#17692) * API: Add tests for BinaryUtil Cover truncateBinary (partial, full, zero-length, and negative-length guard), the previously untested truncateBinaryUnsafe view, and truncateBinaryMax including the all-bytes-overflow case. * More changes * Arrow: Fix vectorized read of all-null DELTA-encoded Parquet pages (#17017) An all-null page decodes to a length stream with totalValueCount 0, so readValues wrote firstValue into a zero-length array. * Data: Split BaseFormatModelTests into read and read-write test classes (#17711) * AWS, Core: Implement RemoteSigningConfig (#17709) * Core, REST, AWS: Implement RemoteSigningConfig * Flink: Support variant for Avro readers and writers (#17737) * Docs: Fix duplicated word in Spark merge-schema description (#17746) * Core: Remove deprecated REST namespace encoding helpers (#17697) * Core: Remove deprecated REST namespace encoding helpers Removes REST API deprecated for removal in 1.12.0: - RESTUtil.NAMESPACE_JOINER / NAMESPACE_SPLITTER, superseded by namespaceToQueryParam(Namespace) / namespaceFromQueryParam(String) - RESTUtil.encodeNamespace(Namespace) / decodeNamespace(String), superseded by the two-arg overloads that take an explicit separator - RESTSessionCatalog.REST_PAGE_SIZE, superseded by RESTCatalogProperties.PAGE_SIZE - CatalogHandlers.loadTable(Catalog, TableIdentifier), superseded by the overload taking a SnapshotMode Javadoc on namespaceToQueryParam and encodeNamespace no longer cross-links the removed single-arg encodeNamespace; the wording now describes the raw versus percent-encoded separator directly. * Core: Remove deprecated public ResourcePaths(String) constructor The deprecation notice said this constructor would be made private in 1.12.0. It had no callers anywhere in the repo, so making it private would leave an unused private constructor; remove it instead. Removal and privatization are indistinguishable to external callers. * Core: Address review feedback on deprecated REST API removal Review feedback from nastra and singhpk234: - Keep the decode separator in encodeAsOldClientAndDecodeAsNewServer hardcoded as "%2E". The test only needs a separator that differs from the legacy one; it does not have to be the one RESTCatalogAdapter advertises. - Use "Removed deprecated functionality scheduled for removal in 1.12.0" as the revapi justification for all seven accepted breaks. The previous text claimed every break was part of the namespace encoding API, which was not true for REST_PAGE_SIZE, CatalogHandlers.loadTable and the ResourcePaths constructor. - Drop the javadoc note on namespaceToQueryParam(Namespace, String). The separator is an explicit argument there, so warning that it differs from encodeNamespace adds nothing. - Keep the note on the no-arg namespaceToQueryParam, which hides the separator, but point it at the surviving encodeNamespace(Namespace, String) overload. The distinction it warns about is real: namespaceToQueryParam joins raw levels, while encodeNamespace URL-encodes each level. * Core: Use transaction metadata for post-commit file cleanup (#17708) BaseTransaction resolved its newly committed snapshots through ops.current(), a catalog re-read that can be stale when the catalog caches table pointers. A missing snapshot made committedFiles return null, skipping clean-up of all uncommitted files and leaking one stale manifest list per commit retry attempt. The transaction's own committed metadata always contains the new snapshots, so resolve them there. * Core: Fail scans when position deletes/DVs don't match data file partition (#16957) * Build: Bump mkdocs-exclude-search from 0.6.6 to 0.6.7 (#17775) Bumps [mkdocs-exclude-search](https://github.com/chrieke/mkdocs-exclude-search) from 0.6.6 to 0.6.7. - [Release notes](https://github.com/chrieke/mkdocs-exclude-search/releases) - [Changelog](https://github.com/chrieke/mkdocs-exclude-search/blob/main/CHANGELOG.md) - [Commits](https://github.com/chrieke/mkdocs-exclude-search/compare/0.6.6...0.6.7) --- updated-dependencies: - dependency-name: mkdocs-exclude-search dependency-version: 0.6.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump datamodel-code-generator from 0.72.2 to 0.73.0 (#17776) Bumps [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) from 0.72.2 to 0.73.0. - [Release notes](https://github.com/koxudaxi/datamodel-code-generator/releases) - [Changelog](https://github.com/koxudaxi/datamodel-code-generator/blob/main/CHANGELOG.md) - [Commits](https://github.com/koxudaxi/datamodel-code-generator/compare/0.72.2...0.73.0) --- updated-dependencies: - dependency-name: datamodel-code-generator dependency-version: 0.73.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump io.delta:delta-standalone_2.12 from 3.3.2 to 3.3.3 (#17779) Bumps [io.delta:delta-standalone_2.12](https://github.com/delta-io/delta) from 3.3.2 to 3.3.3. - [Release notes](https://github.com/delta-io/delta/releases) - [Commits](https://github.com/delta-io/delta/compare/v3.3.2...v3.3.3) --- updated-dependencies: - dependency-name: io.delta:delta-standalone_2.12 dependency-version: 3.3.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump astral-sh/setup-uv from 9.0.0 to 10.0.1 (#17782) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 9.0.0 to 10.0.1. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](https://github.com/astral-sh/setup-uv/compare/c771a70e6277c0a99b617c7a806ffedaca235ff9...20cfd1bf945f4377ade1205e4dbc17946fc9a30d) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: 10.0.1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump the codeql-action group with 3 updates (#17781) Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.6 to 4.37.7 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd) Updates `github/codeql-action/analyze` from 4.37.6 to 4.37.7 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd) Updates `github/codeql-action/upload-sarif` from 4.37.6 to 4.37.7 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.37.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.37.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump io.delta:delta-spark_2.12 from 3.3.2 to 3.3.3 (#17780) Bumps [io.delta:delta-spark_2.12](https://github.com/delta-io/delta) from 3.3.2 to 3.3.3. - [Release notes](https://github.com/delta-io/delta/releases) - [Commits](https://github.com/delta-io/delta/compare/v3.3.2...v3.3.3) --- updated-dependencies: - dependency-name: io.delta:delta-spark_2.12 dependency-version: 3.3.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Parquet: Make createStructReader with fieldId abstract (#17698) createStructReader(List, StructType) was deprecated for removal in 1.12.0 in favor of createStructReader(List, StructType, Integer). The 2-arg form threw UnsupportedOperationException and the 3-arg form defaulted to delegating to it, so neither was usable without an override. Remove the 2-arg method and make the 3-arg form abstract. Both subclasses (GenericParquetReaders, InternalReader) already override it, and BaseParquetReaders is package-private so no other subclass can exist. * Flink: Avoid per-record Set allocation in the dynamic sink forward check (#17618) * Core: Include valid range in BasePartitionStatistics position error (#17546) * Core: Include valid range in BasePartitionStatistics position error Add the expected position range to the "Unknown position" errors thrown by getByPos and internalSet so the message points to the cause. * Address review * Build: Bump software.amazon.awssdk:bom from 2.51.3 to 2.53.1 (#17787) Bumps software.amazon.awssdk:bom from 2.51.3 to 2.53.1. --- updated-dependencies: - dependency-name: software.amazon.awssdk:bom dependency-version: 2.53.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Docs: Add MinIO AIStor to vendors.md (#17760) * Add MinIO AIStor section to vendors documentation Added information about MinIO AIStor as a software-defined object store with Apache Iceberg support. * Docs: note OAuth2 and credential vending in MinIO AIStor entry * Docs: Add OptimaFlo to vendors.md (#17725) Signed-off-by: evro <evandanrosa@gmail.com> * Core: Remove deprecated HadoopFileIO(SerializableSupplier) constructor (#17704) The constructor taking a SerializableSupplier<Configuration> was deprecated for removal in 1.12.0 in favor of HadoopFileIO(Configuration), and has no remaining callers. Since it was the delegation target, its body is inlined into the Configuration constructor. Also drops @Deprecated from serializeConfWith. That method is declared by the HadoopConfigurable interface, which is not deprecated, and it has five live consumers: SerializationUtil, IcebergInputFormat (mr) and SerializableFileIOWithSize in Spark 3.5, 4.0 and 4.1. Marking an override of a supported interface method as deprecated is misleading, so the annotation is removed rather than the method. * Connect: Fix idColumns are not read correctly from table config (#17152) * Site: Expand Variant intro blog with Iceberg table-level details (#17705) * API, Core, Spark: Add Hilbert-curve clustering strategy for `rewrite_data_files` (#16827) * Core, Spark: Fix stale manifest_length in rewrite_table_path manifest lists (#16910) * Parquet: Fix misleading AlwaysFalse placeholder exception message (#17816) * Core: Add dv_count column to PartitionsTable metadata table (#16125) * ORC: Fix timestamp nano predicate pushdown (#17750) Co-authored-by: Vova Kolmakov <wombatukun@gmail.com> * Revert "Core: Add dv_count column to PartitionsTable metadata table" (#17825) Revert "Core: Add dv_count column to PartitionsTable metadata table (#16125)" This reverts commit 68a7f2767b8da025a2663e67fc5321febf3c8dd1. * Core: Reduce visibility on scan planning response builder for specsById and deleteFiles (#17638) * Core: Add geometry bounds builder (#17509) * API: Add a builder usage example to PartitionSpec Javadoc (#17747) * Build: Clean up shaded-jar LICENSE/NOTICE and add missing third-party notices (#16543) * Spark: Make the view stored-schema cast configurable (#17499) * Core, Data, Spark, Flink: Remove of position delete files with row data (#17706) * Spec: Clarify content file uniqueness within a snapshot (#17198) * Core: Preserve trailing slash on table location in V4 manifest reader (#17807) The V4 manifest reader stripped a trailing slash from the table location before resolving relative file locations against it. Remove the strip as it should be done by the catalog during table creation. It is not the reader's responsibility * Spark: Backport view schema binding mode to Spark 3.5 and 4.0 (#17838) * API, Core, Parquet: Carry avg value sizes for v4 content stats (#17451) * API, AWS, Azure, GCP: Track read metrics for positioned, vectored, and accelerated reads (#17236) * Docs: Flink DDL documentation correction (#17854) Flink DDL documentation correction: Expand `ALTER TABLE` section to include support for adding, dropping, renaming, and modifying columns, along with clarifications on partition changes. Update Flink overview table to reflect these capabilities. * Flink: Copy Flink v2.1 to v2.3 directory * Flink: Add v2.2 directory * Flink: Adjust build scripts for Flink 2.2 and 2.3 * Flink: Code changes for Flink 2.2 and 2.3 * Flink: Remove support for Flink 2.0 * Flink: Catalog type refactor in Test and add support for REST catalog in CatalogTestBase (#17861) * Build: Add descriptions to OpenAPI spec validation tasks (#17799) * Core: Read delete manifests correctly during snapshot expiration (#17763) * Spark: Fix map value decoding in test helpers (#17721) Generated-by: Codex * Docs: Update documentation Flink versions and Flink lifecycle maintenance. (#17869) * Flink: Backport CatalogType test refactor to 1.20 * Flink: Backport REST catalog support in CatalogTestBase to 1.20 * Flink: Forward-port CatalogType test refactor to 2.3 * Flink: Forward-port REST catalog support in CatalogTestBase to 2.3 * Flink: Forward-port CatalogType test refactor to 2.2 * Flink: Forward-port REST catalog support in CatalogTestBase to 2.2 * Flink: Fix deleted rows reappearing after a failed eq-delete conversion cycle (#17630) * Build: Bump Avro Version to Avro 1.12.2 (#17188) * Build: Bump Avro Version to Avro 1.12.2 This is a smoke test PR against the staging RC1 artifacts Change-Id: I1954fdf77da9cb6b48b8f6eae9ac8536d852366f * Remove Apache staging repository from repositories Removed Apache staging repository from build.gradle as there's a full release now (same artifact tested against before) * Docs: Fix Nessie Flink Python example (#17091) * Flink: Backport eq-delete conversion replan reindex fix to 1.20 and 2.2 (#17630) * Flink: Forward-port eq-delete conversion replan reindex fix to 2.3 (#17630) * Build: Bump software.amazon.awssdk:bom from 2.53.1 to 2.54.2 (#17887) Bumps software.amazon.awssdk:bom from 2.53.1 to 2.54.2. --- updated-dependencies: - dependency-name: software.amazon.awssdk:bom dependency-version: 2.54.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump docker/setup-buildx-action from 4.2.0 to 4.3.0 (#17886) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 4.2.0 to 4.3.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/bb05f3f5519dd87d3ba754cc423b652a5edd6d2c...37fe631027851001ddb9b187196cc803df7f5f0e) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 8.9.0 to 8.10.0 (#17883) Build: Bump com.diffplug.spotless:spotless-plugin-gradle Bumps [com.diffplug.spotless:spotless-plugin-gradle](https://github.com/diffplug/spotless) from 8.9.0 to 8.10.0. - [Release notes](https://github.com/diffplug/spotless/releases) - [Changelog](https://github.com/diffplug/spotless/blob/main/CHANGES.md) - [Commits](https://github.com/diffplug/spotless/compare/gradle/8.9.0...gradle/8.10.0) --- updated-dependencies: - dependency-name: com.diffplug.spotless:spotless-plugin-gradle dependency-version: 8.10.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump the codeql-action group with 3 updates (#17885) Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.7 to 4.37.8 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd...db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28) Updates `github/codeql-action/analyze` from 4.37.7 to 4.37.8 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd...db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28) Updates `github/codeql-action/upload-sarif` from 4.37.7 to 4.37.8 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd...db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.37.8 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.37.8 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.8 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump datamodel-code-generator from 0.73.0 to 0.74.0 (#17880) Bumps [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) from 0.73.0 to 0.74.0. - [Release notes](https://github.com/koxudaxi/datamodel-code-generator/releases) - [Changelog](https://github.com/koxudaxi/datamodel-code-generator/blob/main/CHANGELOG.md) - [Commits](https://github.com/koxudaxi/datamodel-code-generator/compare/0.73.0...0.74.0) --- updated-dependencies: - dependency-name: datamodel-code-generator dependency-version: 0.74.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump jackson-bom from 2.22.1 to 2.22.2 (#17881) Bumps `jackson-bom` from 2.22.1 to 2.22.2. Updates `com.fasterxml.jackson:jackson-bom` from 2.22.1 to 2.22.2 - [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.22.1...jackson-bom-2.22.2) Updates `com.fasterxml.jackson.core:jackson-core` from 2.22.1 to 2.22.2 - [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.22.1...jackson-core-2.22.2) Updates `com.fasterxml.jackson.core:jackson-databind` from 2.22.1 to 2.22.2 - [Commits](https://github.com/FasterXML/jackson-databind/compare/jackson-databind-2.22.1...jackson-databind-2.22.2) --- updated-dependencies: - dependency-name: com.fasterxml.jackson:jackson-bom dependency-version: 2.22.2 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: com.fasterxml.jackson.core:jackson-core dependency-version: 2.22.2 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-version: 2.22.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * API, Core: Fix typos in test comments (#17889) Fix a duplicated "the", "overriden" -> "overridden", and "doesnt" -> "doesn't" in test comments. Comment-only changes, no behavior change. * Docs, Core, AWS, BigQuery, Flink: Fix miscellaneous typos in comments and docs (#17898) * Docs: Run `generateRuntimeDeps` on all modules (#17323) Updated comments for clarity on Gradle commands. * Docs: Fix grammatical typo in Spark module description (#17897) 'submodules for each spark versions' -> 'submodules for each Spark version' (number agreement + capitalization, consistent with Flink/Hive naming nearby) * Core: Fix geo bounds in v4 legacy stat maps (#17493) * Spark 4.1: Mark overrides of deprecated Spark API as deprecated (#16028) * Core: Remove ThreadPools.WORKER_THREAD_POOL_SIZE_PROP update warning log message (#17812) * Core: Remove ThreadPools.WORKER_THREAD_POOL_SIZE_PROP and correct manifest spec fallback removal version Resolves the two remaining deprecation markers in iceberg-core that promised action "in 1.12.0" but were missed by the removal PRs. Remove ThreadPools.WORKER_THREAD_POOL_SIZE_PROP. The field was deprecated in #5659 in favor of SystemConfigs.WORKER_THREAD_POOL_SIZE and retargeted from 2.0.0 to 1.12.0 by #14392. It had no callers anywhere in the repository outside its own declaration and no references in the docs, and it was the only survivor of #14392's 13-file batch after #17700 removed the rest. The corresponding java.field.removed break is recorded in .palantir/revapi.yml under the 1.11.0 baseline. Correct the ManifestReader warning for the partition-spec-from-file- metadata fallback to say 2.0.0 rather than 1.12.0. The fallback cannot be honored in 1.12.0 because its only production caller is BaseSnapshot:276, which passes a null spec map while serving Snapshot.addedDeleteFiles(FileIO) and removedDeleteFiles(FileIO). Those accessors are themselves already marked "@deprecated will be removed in 2.0.0; use SnapshotChanges#builderFor(Table) instead", and because they receive only a FileIO they structurally cannot supply a spec map. The fallback must therefore outlive 1.12.0, and 2.0.0 aligns its removal with that of its last caller. * reword revapi change * Dell: Preserve known ECS input file lengths (#17870) * Dell: Preserve known ECS input file lengths Carry known lengths into ECS input files so scan planning avoids redundant object metadata requests. Generated-by: Codex * Dell: Verify known lengths avoid metadata requests Use a mock ECS client and assert that reading a supplied file length performs no client interaction. Generated-by: Codex * Core: Add Eager fetch to parquet read path (#17284) * Spark 4.1: Add rewrite option to enable executor cache for delete files (#17868) * Spark 4.0: Add rewrite option to enable executor cache for delete files (#17907) * Spark 3.5: Add rewrite option to enable executor cache for delete files (#17909) * Core: Clean up TestTrackingBuilder (#17145) * Core, API, Parquet: Collapse redundant nested if statements (#17905) * Docs: Add max-file-group-input-files to rewrite options tables in spark-procedures.md (#17707) * Docs: Bump site Flink version to 2.3 (#17936) * Docs: Fix wrong config keys in Flink maintenance docs (#17935) * Core, Open-API, Spark, Flink: Use String.replace for literal replacements (#17484) * API: Add Javadoc for Transforms.fromString (#17693) * Spec: Add finer grained read restrictions as part of loadTable (#13879) Add an optional read-restrictions field to LoadTableResult so a catalog can return the result of evaluating fine grained access policies for the calling user. The restrictions are enforced by the client, which requires a trust relationship between the engine and the catalog; how that trust is established is out of scope for this change. ReadRestrictions carries two optional fields: - required-column-projections: a list of Action objects, each naming a column by field-id along with the transformation to apply. Action uses a discriminator on the action property, with mask-alphanum, mask-to-fixed-value, replace-with-null, show-first-4, show-last-4, truncate-to-year, truncate-to-month, sha-256-global and sha-256-query-local defined. Each action produces a value of the same type as the input column. - required-row-filter: an Expression evaluated against original, untransformed column values. Projections are applied to the rows that survive the filter. Enforcement is fail-closed. A reader that supports read restrictions must fail any read it cannot satisfy in full, including unrecognized action or expression types and actions that cannot be loaded, parsed, or evaluated, and must not silently return raw, partial, or empty results. The restrictions apply to every read performed with a given response, including subsequent planTableScan and fetchScanTasks calls, and may differ between requests or between authentication scopes. A missing or empty ReadRestrictions object imposes no restrictions. Approved on the dev list with 8 binding +1 votes and no -1 votes: https://lists.apache.org/thread/0zloqhp8wkgyn04yg69j71cwcg5n7g74 * Build: Bump mkdocs-material from 9.7.5 to 9.7.7 in /site (#17942) Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.7.5 to 9.7.7. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](https://github.com/squidfunk/mkdocs-material/compare/9.7.5...9.7.7) --- updated-dependencies: - dependency-name: mkdocs-material dependency-version: 9.7.7 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * API: Fix grammatical typos in UpdatePartitionSpec Javadoc (#17941) Correct "the a" / "the an" to "a" / "an" in the @throws documentation of addField and removeField. * Core, Hive, Spark: Auto close JsonGenerator using try-with-resources (#17931) * Core, Hive, Spark: Close JsonGenerator instead of only flushing it Ten sites created a JsonGenerator, called flush(), and left it unclosed. flush() pushes buffered content to the target and flushes the target itself, so the output was correct, but only close() returns the generator's char buffers to the thread's BufferRecycler (_releaseBuffers) and returns the recycler to the pool (IOContext.close). Without it, every serialization allocates a fresh char[4000] concat buffer instead of reusing one. Each generator now joins its enclosing try-with-resources so it closes before its target. The explicit flush() remains only where a value is read off the target inside the try: JsonUtil.generate and TableMetadataParser.toJson read StringWriter.toString(), and HMSTablePropertyHelper.hashOf reads HashWriter.getHash(). The last is ordering-sensitive, since AUTO_CLOSE_TARGET makes closing the generator close the HashWriter, and a closed HashWriter rejects getHash(). A JMH benchmark over TableMetadataParser measured a fixed 8,016 B/op allocation reduction independent of document size, which is one char[4000] plus its object header. For a ~1 KB document that is 10,616 -> 2,600 B/op and 0.860 -> 0.730 us/op; for ~41 KB it is 49,680 -> 41,664 B/op. On a failure mid-serialization, close() now auto-completes open contexts (AUTO_CLOSE_JSON_CONTENT) and flushes partial content, where the flush() was previously skipped. The resulting metadata file is orphaned either way because the commit fails. * Restore explicit generator.flush() calls Review feedback: relying on close() to flush is correct but implicit, and it contradicts JsonUtil.generate, which was cited as the pattern this change follows and which does call flush(). Restoring the calls keeps this change to a single concern: adding the generator to the enclosing try-with-resources so it is closed. The buffer-recycling win comes entirely from close() releasing the char buffer to the BufferRecycler, not from dropping the flush, so nothing measured is lost. The flush is now redundant with close() at these sites, which costs one extra flush() on the underlying stream per write. * Core, OpenAPI: Preserve avg value sizes in REST JSON (#17855) * Core, Flink, Spark, Build: Replace Guava Queues with plain JDK and bump guava from 33.6.0-jre to 33.7.1-jre (#17888) * Replace Guava Queues with plain JDK * Build: Bump guava from 33.6.0-jre to 33.7.1-jre Bumps `guava` from 33.6.0-jre to 33.7.1-jre. Updates `com.google.guava:guava` from 33.6.0-jre to 33.7.1-jre - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) Updates `com.google.guava:guava-testlib` from 33.6.0-jre to 33.7.1-jre - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-version: 33.7.1-jre dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: com.google.guava:guava-testlib dependency-version: 33.7.1-jre dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Core: Fix commit validation when a data file has multiple DVs across snapshots (#17764) * API, Core: Expose co-located deletion vector through DataFile (#17928) V4 co-locates a data file's deletion vector on the data entry. Add DataFile.deletionVector() returning the DV as a DeleteFile. Related discussion thread: https://github.com/apache/iceberg/pull/17541/changes#r3787760198 * API: Add contextual load apis (#15895) * API: Adds contextual load APIs and LoadContext Co-authored-by: Prashant Singh <prashant.singh@snowflake.com> * Spark: Add support for 4.2.0 (#14984) * Spark 4.2: Fix build failures from missed cleanups (#17952) * Docs: Note that REST catalog credentials should be redacted by engines (#17920) * Core: Avoid per-row path rehashing in toPositionIndexes (#17864) * Core: Avoid per-row path rehashing in toPositionIndexes Signed-off-by: Sergei Nikolaev <kinolaev@gmail.com> * Core: Compare paths by content when building pos delete indexes + tests Signed-off-by: Sergei Nikolaev <kinolaev@gmail.com> Co-authored-by: Anurag Mantripragada <amantripragada@apple.com> * Core: Fix Deletes.toPositionIndexes with interleaved paths test Signed-off-by: Sergei Nikolaev <kinolaev@gmail.com> * Core: Fix test names, improve assertion in TestDeletes Signed-off-by: Sergei Nikolaev <kinolaev@gmail.com> --------- Signed-off-by: Sergei Nikolaev <kinolaev@gmail.com> Co-authored-by: Anurag Mantripragada <amantripragada@apple.com> * API, Core: Add ManifestFile adapter for v4 tracked files (#17932) Reintroduces the ManifestFile adapter closed in #16867, which stalled because ManifestFile had no way to represent a manifest deletion vector. The V4 scan task planner (#17541) plans over manifests through the ManifestFile API, so this adds ManifestFile.manifestDeletionVector() and has the adapter expose the tracked file's manifest DV. * Build: Bump zizmorcore/zizmor-action from 0.6.2 to 0.6.3 (#17976) Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.2 to 0.6.3. - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](https://github.com/zizmorcore/zizmor-action/compare/3dc1ecc9bcb9e94e9b2c709687979e1298497054...70fb788f84895a7701f5643d103d587e460b5c99) --- updated-dependencies: - dependency-name: zizmorcore/zizmor-action dependency-version: 0.6.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump actions/setup-java from 5.7.0 to 6.0.0 (#17975) Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5.7.0 to 6.0.0. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](https://github.com/actions/setup-java/compare/b6effb05e454b25005698d916606bdc6ffcbf961...dd06d9cba3e5552c54d9f8ea23572deb30010f7c) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump the codeql-action group with 3 updates (#17974) Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.8 to 4.37.9 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28...cdf488f595d80d6e07e03d4674febd5ab45fa938) Updates `github/codeql-action/analyze` from 4.37.8 to 4.37.9 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28...cdf488f595d80d6e07e03d4674febd5ab45fa938) Updates `github/codeql-action/upload-sarif` from 4.37.8 to 4.37.9 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28...cdf488f595d80d6e07e03d4674febd5ab45fa938) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump org.openapitools:openapi-generator-gradle-plugin from 7.24.0 to 7.25.0 (#17973) Build: Bump org.openapitools:openapi-generator-gradle-plugin Bumps [org.openapitools:openapi-generator-gradle-plugin](https://github.com/OpenAPITools/openapi-generator) from 7.24.0 to 7.25.0. - [Release notes](https://github.com/OpenAPITools/openapi-generator/releases) - [Changelog](https://github.com/OpenAPITools/openapi-generator/blob/master/docs/release-summary.md) - [Commits](https://github.com/OpenAPITools/openapi-generator/compare/v7.24.0...v7.25.0) --- updated-dependencies: - dependency-name: org.openapitools:openapi-generator-gradle-plugin dependency-version: 7.25.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump org.xerial:sqlite-jdbc from 3.53.2.1 to 3.53.4.0 (#17972) Bumps [org.xerial:sqlite-jdbc](https://github.com/xerial/sqlite-jdbc) from 3.53.2.1 to 3.53.4.0. - [Release notes](https://github.com/xerial/sqlite-jdbc/releases) - [Changelog](https://github.com/xerial/sqlite-jdbc/blob/master/CHANGELOG) - [Commits](https://github.com/xerial/sqlite-jdbc/compare/3.53.2.1...3.53.4.0) --- updated-dependencies: - dependency-name: org.xerial:sqlite-jdbc dependency-version: 3.53.4.0 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 8.10.0 to 8.10.1 (#17970) Build: Bump com.diffplug.spotless:spotless-plugin-gradle Bumps [com.diffplug.spotless:spotless-plugin-gradle](https://github.com/diffplug/spotless) from 8.10.0 to 8.10.1. - [Release notes](https://github.com/diffplug/spotless/releases) - [Changelog](https://github.com/diffplug/spotless/blob/main/CHANGES.md) - [Commits](https://github.com/diffplug/spotless/compare/gradle/8.10.0...gradle/8.10.1) --- updated-dependencies: - dependency-name: com.diffplug.spotless:spotless-plugin-gradle dependency-version: 8.10.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump software.amazon.awssdk:bom from 2.54.2 to 2.54.7 (#17969) Bumps software.amazon.awssdk:bom from 2.54.2 to 2.54.7. --- updated-dependencies: - dependency-name: software.amazon.awssdk:bom dependency-version: 2.54.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build: Bump datamodel-code-generator from 0.74.0 to 0.76.0 (#17968) Bumps [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) from 0.74.0 to 0.76.0. - [Release notes](https://github.com/koxudaxi/datamodel-code-generator/releases) - [Changelog](https://github.com/koxudaxi/datamodel-code-generator/blob/main/CHANGELOG.md) - [Commits](https://github.com/koxudaxi/datamodel-code-generator/compare/0.74.0...0.76.0) --- updated-dependencies: - dependency-name: datamodel-code-generator dependency-version: 0.76.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] …

github-actions Bot added spark core labels Jun 21, 2026

leeyam24 reviewed Jun 22, 2026

View reviewed changes

wombatu-kun requested a review from leeyam24 June 22, 2026 13:54

leeyam24 approved these changes Jun 23, 2026

View reviewed changes

kevinjqliu mentioned this pull request Jun 27, 2026

Core: update manifest delete file size after rewrite table action #15470

Merged

kevinjqliu self-requested a review June 27, 2026 20:58

wombatu-kun force-pushed the issue/16905-manifest-length-rewrite-table-path branch from a81e706 to babe6a8 Compare June 29, 2026 01:31

danielcweeks assigned danielcweeks and unassigned danielcweeks Jun 29, 2026

danielcweeks self-requested a review June 29, 2026 18:28

vaultah reviewed Jul 1, 2026

View reviewed changes

github-actions Bot added the stale label Aug 10, 2026

wombatu-kun force-pushed the issue/16905-manifest-length-rewrite-table-path branch from 09ec66a to 06cd4ed Compare August 10, 2026 06:03

github-actions Bot removed the stale label Aug 11, 2026

varun-lakhyani reviewed Aug 15, 2026

View reviewed changes

varun-lakhyani mentioned this pull request Aug 15, 2026

Core: Add Eager fetch to parquet read path #17284

Merged

varun-lakhyani reviewed Aug 16, 2026

View reviewed changes

RussellSpitzer reviewed Aug 17, 2026

View reviewed changes

Vova Kolmakov and others added 2 commits August 18, 2026 14:15

Core, Spark: Fix stale manifest_length in rewrite_table_path manifest…

b9a5f85

… lists Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Spark: Cover all snapshots in rewrite_table_path manifest_length test

cc4a625

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Vova Kolmakov and others added 5 commits August 18, 2026 14:15

Fix build after rebase onto main

a268f3f

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Core, Spark: Record the rewritten manifest length in rewrite_table_pa…

f828f4c

…th manifest lists Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

Core, Spark: Address review feedback on rewrite_table_path manifest l…

4bd491a

…ength Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

Spark: Scope rewrite_table_path manifest length fix to Spark 4.1

3f63fa9

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

Core, Spark: Carry the rewritten manifest length on RewriteResult

c0337be

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

wombatu-kun force-pushed the issue/16905-manifest-length-rewrite-table-path branch from 6e3e691 to c0337be Compare August 18, 2026 07:30

RussellSpitzer reviewed Aug 18, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java Outdated

RussellSpitzer reviewed Aug 18, 2026

View reviewed changes

Core, API: Address review feedback on rewrite_table_path manifest length

4b8369e

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

github-actions Bot added the API label Aug 19, 2026

Core, Spark: Select manifests to rewrite by their measured length

6f73ff1

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

RussellSpitzer reviewed Aug 24, 2026

View reviewed changes

Core, Spark: Assert rewritten manifest lengths across rewrite_table_p…

fdb2422

…ath tests Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code

RussellSpitzer approved these changes Aug 25, 2026

View reviewed changes

RussellSpitzer merged commit ebebc34 into apache:main Aug 25, 2026
39 checks passed

smaheshwar-pltr mentioned this pull request Aug 26, 2026

Spark: Reject encrypted table path rewrites smaheshwar-pltr/iceberg#24

Closed

Conversation

wombatu-kun commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Known gap: incremental runs

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leeyam24 left a comment

Choose a reason for hiding this comment

Uh oh!

kevinjqliu commented Jun 27, 2026

Uh oh!

wombatu-kun commented Jun 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 10, 2026

Uh oh!

varun-lakhyani left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

varun-lakhyani Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

varun-lakhyani Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

wombatu-kun commented Jun 21, 2026 •

edited

Loading

varun-lakhyani left a comment •

edited

Loading

varun-lakhyani Aug 16, 2026 •

edited

Loading

varun-lakhyani Aug 15, 2026 •

edited

Loading

varun-lakhyani Aug 16, 2026 •

edited

Loading