Skip to content

fix(storage): track EOF in BlobReader to handle doubly-gzipped blobs - #18430

Open
ohmayr wants to merge 1 commit into
mainfrom
fix-storage-doubly-gzipped-eof
Open

ohmayr wants to merge 1 commit into
mainfrom
fix-storage-doubly-gzipped-eof

Conversation

@ohmayr

@ohmayr ohmayr commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Problem

When reading doubly-gzipped blobs (or objects stored with Content-Encoding: gzip), GCS decompressive transcoding can cause out-of-bounds byte range requests to return HTTP 200 OK with full body payload instead of HTTP 416 RequestRangeNotSatisfiable. Because BlobReader previously relied exclusively on catching HTTP 416 to recognize EOF, out-of-bounds reads resulted in infinite read loops.

Fix

Added client-side self._eof tracking to BlobReader:

  1. When a range download returns fewer bytes than requested (len(downloaded) < fetch_end - fetch_start), or raises RequestRangeNotSatisfiable, BlobReader sets self._eof = True.
  2. Subsequent read() calls return b"" immediately without making unnecessary HTTP requests.
  3. seek() resets self._eof = False to support repositioning and re-reading.

Verification

  • Added unit test test_read_doubly_gzipped_eof in packages/google-cloud-storage/tests/unit/test_fileio.py.
  • Executed pytest: 39/39 unit tests pass.
  • Verified against live GCS bucket gs://apache-beam-samples across all 9 test files from Issue google-cloud-storage: Failed to handle doubly-gzipped files #18423 in both raw_download=False and raw_download=True modes.

Fixes #18423

@ohmayr
ohmayr requested review from a team as code owners September 20, 2026 09:05
@ohmayr ohmayr changed the title fix(storage): track EOF in BlobReader to handle doubly-gzipped blobs (#18423) fix(storage): track EOF in BlobReader to handle doubly-gzipped blobs Sep 20, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces client-side EOF tracking in google-cloud-storage to prevent infinite read loops on doubly-gzipped files, alongside corresponding test updates. It also configures 100% code coverage requirements and updates the Nox sessions for google-crc32c. Feedback on the changes points out an issue in the check session of noxfile.py, where running coverage against the installed wheel instead of the source directory will result in 0% coverage and cause the coverage check to fail; it is recommended to remove coverage flags from this session.

Comment thread packages/google-crc32c/noxfile.py Outdated
@ohmayr
ohmayr force-pushed the fix-storage-doubly-gzipped-eof branch from 6f60e2a to 6891704 Compare September 20, 2026 09:08
…18423)

Fixes #18423. When reading doubly-gzipped blobs (or objects with Content-Encoding: gzip), out-of-bounds byte range requests may return full payload bytes instead of HTTP 416 RequestRangeNotSatisfiable.

This change adds client-side _eof tracking to BlobReader. When a range download returns fewer bytes than requested (or raises 416), BlobReader sets _eof = True so subsequent read() calls immediately return b"" without initiating redundant HTTP range requests.
@ohmayr
ohmayr force-pushed the fix-storage-doubly-gzipped-eof branch from 6891704 to 94c3c1b Compare September 20, 2026 09:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

google-cloud-storage: Failed to handle doubly-gzipped files

1 participant