Skip to content

Slice stored .conda zip members instead of copying - #44

Merged
andrew merged 1 commit into
mainfrom
perf/conda-zero-copy
Sep 13, 2026
Merged

andrew merged 1 commit into
mainfrom
perf/conda-zero-copy

Conversation

@andrew

@andrew andrew commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

readCondaMember read each .tar.zst member with io.ReadAll(io.LimitReader(rc, ...)), which grows a buffer by doubling and copies the whole member. Real .conda packages store their members uncompressed (zip.Store), so f.DataOffset() locates the payload inside the outer archive bytes and raw[off : off+CompressedSize64] returns it directly. Bounds are checked against a header that lies about size or offset; non-Store members and out-of-range headers fall through to the existing Open+ReadAll path.

The stored path skips archive/zip's CRC32 check on the member; the inner zstd frame check still catches corruption.

Adds TestCondaDeflatedMemberFallback for the non-Store path and BenchmarkOpenConda (8 MiB incompressible pkg member so the stored .tar.zst is ~8 MiB). Interleaved benchstat, arm64 darwin, n=10, 2s each:

            │   before    │             after              │
            │   sec/op    │   sec/op     vs base           │
OpenConda    4.012m ± 20%   2.452m ± 16%  -38.88% (p=0.000)
            │    B/op     │    B/op      vs base           │
OpenConda    45.98Mi ± 0%   29.69Mi ± 0%  -35.42% (p=0.000)
            │  allocs/op  │  allocs/op   vs base           │
OpenConda      181.0 ± 1%    133.0 ± 0%   -26.52% (p=0.000)

The ~16 MiB B/op saving is roughly 2× the member size (io.ReadAll growth doubles at the last step). Real conda packages run to hundreds of MB per member, so the saving scales with that.

Real .conda packages store their .tar.zst members uncompressed, so
DataOffset locates the payload inside the outer bytes and a slice
returns it with no io.ReadAll copy or buffer growth. Non-Store or
out-of-range members fall back to Open+ReadAll. The stored path skips
archive/zip's CRC32; the inner zstd frame check still catches
corruption.
@andrew
andrew merged commit cba6e42 into main Sep 13, 2026
5 checks passed
@andrew
andrew deleted the perf/conda-zero-copy branch September 13, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant