Add support for new oci-delta version - #393
Merged
Merged
Conversation
The delta end-to-end tests built their changed layer from a dumpfile `Item::Regular` entry, whose content comes from `SmallRng::seed_from_u64(size)`. Source and target used different sizes, so their contents were unrelated and tar-diff had nothing to diff: every delta it produced consisted of a single Data op replaying the layer verbatim. The Copy, AddData and Seek paths in tar_patch_apply were never reached. Give the source and target similar contents instead. That can't be expressed as a dumpfile — the parser caps lines at 512 bytes, so multi-kilobyte inline content is impossible — so build the layer tar directly and add build_oci_layout_from_tars() to feed ready-made tars into an OCI layout. Assert on the opcodes present in the generated delta so this can't silently regress into a Data-only delta again. Assisted-by: AI Signed-off-by: Alexander Larsson <alexl@redhat.com>
tar-diff gained a zstd-based binary diff backend, selected in oci-delta with
`--binary-diff zstd`. It adds a ZstdDict op (5), whose payload is a zstd frame
compressed against the current source file used as a raw dictionary — what
`zstd --patch-from` produces — and bumps the file header to `tardf2\n\0`.
Accept both headers and implement the new op. The Rust counterpart of Go's
zstd.WithDecoderDictRaw(0, dict) is Decoder::with_ref_prefix (ZSTD_DCtx_refPrefix),
which is raw-content and single-frame; DecoderDictionary would use ZSTD_dct_auto
and misparse a source file that happens to start with the dictionary magic.
That needs zstd 0.13.2, so bump the floor.
Two details worth noting:
- The dictionary is the whole source file, and reading it leaves the cursor
at EOF. That matches the generator, which sets currentPos = sourceSize
after emitting the op, and is why it emits a Seek before the next Copy.
- libzstd refuses windows above 128 MiB by default, but the generator sizes
the window to the source file and allows up to 512 MiB, so raise
window_log_max to match. Cap the buffered dictionary at the same 512 MiB.
The end-to-end test needs an oci-delta new enough to have --binary-diff, so it
probes for it and skips otherwise; Fedora currently ships 0.3.0, which predates
the flag, but an update is comming.
Assisted-by: AI
Signed-off-by: Alexander Larsson <alexl@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
oci-delta 0.6.0 added support for zstd diffs, which are somewhat larger, but much faster to compute. This adds support for applying such diffs.
Note: There are some issues with getting the CI to test the delta code, which is fixed in #315, so that should ideally be merged first and then this rebased on top of that so that the new tests are actually run by the CI.