Repository navigation
Add RewriteFiles support to iceberg-rust #1607
Description
Is your feature request related to a problem or challenge?
Currently, iceberg-rust does not support committing operations that both add and delete files. This feature, known as RewriteFiles in iceberg-java, is crucial for our use case, particularly for compaction.
Describe the solution you'd like
Based on my initial investigation in #1606, I believe the following three components are necessary to implement RewriteFiles in iceberg-rust:
- Process delete files: We need to add logic to handle delete files when generating snapshots.
- Implement
RewriteFilesAction: This will require creating a new transaction action,RewriteFilesAction, and related classes likeRewriteFilesOperation. - Support validation: We need to add validation for rewrite operations to prevent conflicts. Currently we don't have any validation mechnism in the codebase, so a new trait
SnapshotValidatorwould be needed.
Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg Rust community
Reacted by Danny C Jones and Ilya Pimenov
Reacted by dentiny, Vika, Timothy Maloney, Xander, Joshua Wilson, Swoorup Joshi, Netanel Malka, Danny C Jones, mattefunnel, Ilya Pimenov and 1 more
Activity
debugmiller commentedon Dec 4, 2025
I was just looking to do compaction. This would be great!
feat: Implement RewriteFiles transaction action
jdockerty commentedon Jan 13, 2026
I believe introducing this
RewriteFilesaction also means exposing the ability to mark data files asDELETEDtoo? It doesn't seem like that is currently possible withiniceberg-rust, as the only changes are additive viaFastAppend.Being able to mark data files as deleted/replaced would be very helpful for our use case too. I think doing that is part of the
RewriteFileaction, it is something our use case would be greatly helped by.Is there anyway that we can help move this along?
jdockerty commentedon Jan 20, 2026
@Xuanwo Are you able to provide any insight/guidance on ☝️?
CTTY commentedon Jan 20, 2026
Hey @jdockerty , sorry I thought I had replied but obviously I didn't.
In this issue, I meant to support
RewriteFilesas long as we can process delete files. Delete file support can be added separatelydrbothen commentedon Feb 15, 2026
Hi @CTTY, I've been working on a RewriteFiles implementation for our compaction use case and have a working version I'd like to contribute. Before opening a PR, I wanted to check with you since you own this issue:
RewriteFilesAction+ Validation Support + Process Delete Manifests #1606?RewriteFilesAction+SnapshotProducerdelete file support?The implementation covers:
SnapshotProducerchanges to supportdeleted_data_files(write deleted manifest, validate files exist, summary tracking)RewriteFilesActionthat usesOperation::Replaceper the specHappy to coordinate however works best!
CTTY commentedon Mar 16, 2026
Hey @drbothen sorry for the late reply, as the storage trait work is about to be wrapped up, I'm picking this back up! We want to get transaction API to a good state to enable compaction use cases
Make TransactionAction, SnapshotProducer public for RewriteFiles support
mattefunnel commentedon Mar 24, 2026
@CTTY (and others), we’re interested in this as well. Feel free to reach out if there’s anything I can do to help drive this forward.
brgr-s commentedon May 21, 2026
@CTTY Currently working on getting #2242 and #1606 and test it stand-alone to bin pack an unpartioned table without branching. Will get with feedback as soon as I have some, or if you have a different order in mind.
hotcache commentedon Aug 21, 2026
Hi @CTTY, @blackmwk, @dannycjones, @rexminnis,
I've been following the discussion across #1607, #2620, and #2367, and I'd
like to help move this forward. I have a working implementation that I'd like
to propose as a PR.
Approach
Based on the discussion, there were two open design questions:
MergingSnapshotProducer as a new struct vs extending existing actions —
I went with a composition-based
MergingSnapshotProducerstruct(aligning with @blackmwk's suggestion), but using Rust composition
instead of Java-style inheritance (aligning with @dannycjones's preference
for idiomatic Rust).
Cross-retry cache — I intentionally deferred this to a follow-up PR.
As @rexminnis noted, "always-rewrite-per-attempt is correct, just not
optimal." This unblocks the feature without requiring the cache design
to be settled first.
Architecture
The key design choice is that
MergingSnapshotProduceris a shared engineowned by each action via composition, not a trait or base class:
MergingSnapshotProducer (struct — shared engine)
│ - add/delete file tracking
│ - ManifestFilterManager (filters deleted files from manifests)
│ - manifest writing + summary computation
│ - delegates snapshot creation to SnapshotProducer
│
│ Owned by each action:
│
├── RewriteFilesAction (this PR)
│ operation: Replace
│ validate: fail_missing_delete_paths
│
├── OverwriteFilesAction (future)
│ operation: Overwrite
│ validate: conflict_detection_filter
│
└── RowDeltaAction (future)
operation: Overwrite
validate: no_new_deletes_on_target_files
This structure keeps future operations simple. Adding a new operation
means writing a thin struct that:
MergingSnapshotProducervalidate()rulesproducer.commit_snapshot()No shared manifest filtering, caching, or summary logic needs to be
duplicated — it's all in
MergingSnapshotProducer. For example,RewriteFilesActionis ~50 lines (excluding tests), similar to howJava's
BaseRewriteFilesis trivially thin on top ofMergingSnapshotProducer.The existing
FastAppendActionandSnapshotProducerare completelyunchanged.
Planned follow-up PRs
(
Mutex<Option<Vec<ManifestFile>>>inMergingSnapshotProducer)SnapshotValidatortrait for conflict detection(applies to both
RewriteFilesandFastAppend)OverwriteFilesAction— demonstrates extensibility of theMergingSnapshotProducerpatternI'd love to hear your thoughts before I open the PR. @CTTY, I want to
make sure this doesn't conflict with your plans — happy to coordinate.
feat(transaction): add MergingSnapshotProducer and RewriteFilesAction
feat(transaction): add MergingSnapshotProducer and RewriteFilesAction