Skip to content

Use klauspost/compress for tar and zip decompression - #41

Merged
andrew merged 2 commits into
mainfrom
perf/klauspost-gzip
Sep 13, 2026
Merged

andrew merged 2 commits into
mainfrom
perf/klauspost-gzip

Conversation

@andrew

@andrew andrew commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

The module already depends on klauspost/compress for zstd; its gzip subpackage is a drop-in replacement for stdlib and aliases ErrChecksum/ErrHeader to the stdlib sentinels, so no callers or existing tests change.

Interleaved benchstat, arm64 darwin, n=10:

                              │   before    │             after              │
                              │   sec/op    │   sec/op     vs base           │
OpenBytesTarGz                  552.8µ ± 6%   437.1µ ± 4%  -20.93% (p=0.000)
TarBrowse/test.tar.gz/0         12.48µ ± 5%   11.68µ ± 3%   -6.40% (p=0.005)
TarBrowse/test.tar.gz/512       17.93µ ± 4%   16.19µ ± 4%   -9.72% (p=0.000)
TarBrowse/test.tar.gz/16384     37.18µ ± 4%   33.97µ ± 3%   -8.63% (p=0.000)
TarBrowse/test.tar.gz/65536     90.14µ ± 3%   84.65µ ± 5%   -6.09% (p=0.000)
TarBrowse/test.tar.gz/1048576   1.157m ± 4%   1.068m ± 3%   -7.71% (p=0.000)
geomean                         88.47µ        79.56µ       -10.07%

An isolated decoder A/B on text-like data (25% compression ratio, closer to real source archives than the random-payload fixtures) shows 1.44–1.88×. B/op drops 0.03–9.3% across the .gz cases; allocs/op gains a fixed +2 per open from the reader setup.

Stripped binary size for a minimal consumer grows by ~84 KB (+3.1%) since archive/zip keeps stdlib compress/flate linked, so klauspost's flate is additive rather than a replacement.

Adds a multistream-gzip test (Alpine .apk shape) and a corrupt-header test; both pass against stdlib and klauspost.

For zip, reader.RegisterDecompressor(zip.Deflate, ...) swaps in klauspost's flate on the existing archive/zip reader. That bypasses archive/zip's internal flate pool, and klauspost's Reset wraps any input lacking io.ByteReader in a fresh bufio.Reader (archive/zip passes an io.SectionReader), so zip.go pools a *bufio.Reader alongside the decoder and hands that to Reset. Zero binary cost since klauspost's flate is already linked via gzip. Adds BenchmarkZipExtract (64×16 KiB deflated members, ~34% ratio):

             │   before    │             after              │
             │   sec/op    │   sec/op     vs base           │
ZipExtract     3.785m ± 3%   2.350m ± 3%  -37.91% (p=0.000)
             │    B/op     │    B/op      vs base           │
ZipExtract    50.96Ki ± 2%  48.33Ki ± 1%   -5.14% (p=0.000)
             │  allocs/op  │  allocs/op   vs base           │
ZipExtract      826.0 ± 0%    722.0 ± 0%  -12.59% (p=0.000)

conda.go is left alone: its zip members are Store, so registering a deflate decompressor there does nothing.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved blocking issues were identified.

Pull request overview

This PR switches TAR gzip decompression to Klauspost’s compatible implementation for improved performance while preserving behavior.

Changes:

  • Use klauspost/compress/gzip for TAR decompression.
  • Add multistream and corrupt-header tests.
File summaries
File Description
tar.go Uses Klauspost gzip for TAR decompression.
tar_read_test.go Adds multistream and header-error coverage.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

RegisterDecompressor bypasses archive/zip's internal flate pool and
klauspost's Reset wraps non-ByteReader inputs in a fresh bufio.Reader,
so pool a bufio.Reader alongside the decoder and hand that to Reset.
@andrew andrew changed the title Use klauspost/compress/gzip for tar decompression Use klauspost/compress for tar and zip decompression Sep 13, 2026
@andrew
andrew requested a lite review from Copilot September 13, 2026 12:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved blocking issues were identified, and all reviewed changes preserve behavior while improving decompression performance.

Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@andrew
andrew merged commit b89e711 into main Sep 13, 2026
6 checks passed
@andrew
andrew deleted the perf/klauspost-gzip branch September 13, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants