Skip to content

fix(conda): preserve gzip for large repodata indexes - #379

Merged
andrew merged 1 commit into
git-pkgs:mainfrom
abhinavgautam01:fix/conda-repodata-gzip-321
Sep 26, 2026
Merged

andrew merged 1 commit into
git-pkgs:mainfrom
abhinavgautam01:fix/conda-repodata-gzip-321

Conversation

@abhinavgautam01

Copy link
Copy Markdown
Contributor

Closes #321

Summary

Request gzip for Conda JSON repodata and preserve the compressed response through caching and delivery. This reduces transfer size and lets large indexes fit within the metadata cache limit when their compressed representation fits.

Changes

  • Request gzip for repodata.json and current_repodata.json.
  • Cache and replay compressed bytes with the correct Content-Encoding and Content-Length.
  • Preserve gzip for uncached streaming and offline cache fallback.
  • Keep repodata.json.bz2 on identity encoding.
  • Decode gzip before cooldown filtering, rejecting corrupt or truncated gzip responses.
  • Retain metadata size limits, including the decoded-size limit for cooldown filtering.

Scope and limitations

  • Encoding changes are limited to Conda JSON repodata; other ecosystems remain unchanged.
  • Upstreams that return identity responses remain supported, subject to the existing size limit.
  • Cooldown filtering still requires the decoded JSON to fit within metadata_max_size.
  • HTTP timeout settings remain unchanged.

Validation

Added regression coverage for both JSON routes, streaming, fresh-cache hits, offline fallback, identity responses, .bz2 preservation, size limits, cooldown filtering and invalid gzip.

Confirmed the new regression tests reproduce the failures against the original handler.

Local checks passed:

  • Formatting and git diff --check
  • go tool golangci-lint run ./... — 0 issues
  • go test -race ./...
  • go test ./internal/handler -run 'TestConda' -count=10
  • go vet ./...
  • go build ./...
  • go mod tidy -diff

@andrew andrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Matches the fix specified in the issue and follows the proxyCachedWithEncoding pattern already used for the Homebrew API. Good catch on the cooldown path: handleRepodata was already sending Accept-Encoding: gzip and handing the compressed body to ReadMetadata, so filtering silently fell through to writing gzip bytes under Content-Type: application/json. The decode fixes that, and ReadMetadata's limit keeps the decompressed size bounded.

@andrew
andrew merged commit b00da65 into git-pkgs:main Sep 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

conda: large repodata.json fails through the proxy

2 participants