fix(declarative): detect gzip payloads in GzipParser instead of trusting headers

[Airbyte Support (Airbyte-Support)]

Summary

Requested by Syed Khadeer (Airbyte support) off Zendesk ticket 18622: a customer's Connector Builder Test read of a stream that downloads a gzipped CSV fails with UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1 while a real sync of the same stream succeeds. 0x1f 0x8b is the gzip magic — compressed bytes are reaching the UTF-8 CSV parser.

Resolves https://github.com/airbytehq/oncall/issues/13362:

• https://github.com/airbytehq/oncall/issues/13362

Cause: create_gzip_decoder builds two different decoders depending on mode, and neither actually looks at the payload.

if self._emit_connector_builder_messages:      # Builder test read
    return CompositeRawDecoder(gzip_parser.inner_parser, False)   # gunzip step dropped entirely
return CompositeRawDecoder.by_headers(          # real sync
    [({"Content-Encoding", "Content-Type"}, _compressed_response_types, gzip_parser)],
    stream_response=True,
    fallback_parser=gzip_parser.inner_parser,   # no gunzip unless a header matched
)

The Builder branch assumed requests had already decompressed response.content, which only holds for transport-level Content-Encoding: gzip. When the body is a gzip payload (e.g. an S3 *.csv.gz download served as application/gzip or binary/octet-stream, as Apple App Store Connect analytics report segments are), nothing gunzips it in Builder mode. Measured on main with a single GzipDecoder(CsvDecoder), against an undecoded raw stream (decode_content=False, matching how requests constructs response.raw in production sync mode):

payload / headers	sync	Builder test read
gzip, Content-Type: application/gzip	OK	UnicodeDecodeError 0x8b
gzip, Content-Type: binary/octet-stream	UnicodeDecodeError 0x8b	UnicodeDecodeError 0x8b
gzip, Content-Encoding: gzip	OK	OK
gzip×2, Content-Encoding: gzip + Content-Type: application/gzip	UnicodeDecodeError 0x8b (only one layer stripped)	UnicodeDecodeError 0x8b

(An earlier revision of this table claimed the Content-Encoding: gzip sync path raised BadGzipFile; that measurement was taken with decode_content=True, which pre-decompressed the fixture body — the production raw path succeeded. Corrected per review.)

That asymmetry is why manifests in the wild carry double-nested GzipDecoder(GzipDecoder(CsvDecoder)) workarounds (source-amazon-ads ships one next to a # TODO Fix me) — and why the single- vs. double-nesting workarounds are mutually exclusive depending on which header the server returns. Previously reported in airbytehq/oncall#7739, #11173, #11809 and airbytehq/airbyte#56988.

Fix, in two parts:

1. Make GzipParser self-detecting — the behavior its own docstring already promised ("If the data is not gzipped, reset the pointer and pass the data to the inner parser as is") but never implemented. GzipParser.parse reads the 2-byte header (looping, since response.raw.read(2) may short-read), gunzips when it is \x1f\x8b, and otherwise hands the inner parser the untouched stream. Because the stream may be non-seekable (response.raw in sync mode), the consumed header is restored by wrapping it in a private _PrefixedStream (io.RawIOBase) inside an io.BufferedReader, which keeps it usable by both gzip.GzipFile and CsvParser's TextIOWrapper.
2. Use it in both modes of create_gzip_decoder, treating the HTTP transport layer and the gzip representation as separate layers — a response can validly carry both Content-Encoding: gzip and Content-Type: application/gzip, in which case the sync raw stream holds two gzip layers:

if self._emit_connector_builder_messages:   # requests strips the transport layer via response.content
    return CompositeRawDecoder(gzip_parser, False)

transport_gzip_parser = GzipParser(inner_parser=gzip_parser)   # strips the Content-Encoding layer, then the payload layer
return CompositeRawDecoder.by_headers(
    [
        ({"Content-Encoding"}, {"gzip"}, transport_gzip_parser),
        ({"Content-Type"}, _compressed_response_types, gzip_parser),
    ],
    stream_response=True,
    fallback_parser=gzip_parser,
)

A single GzipDecoder now works for every combination in the table above, in both modes (pass-through makes the nested parser safe when only one layer is present). Existing double-nested manifests keep working: the inner GzipParser sees already-decompressed data and passes it through instead of raising BadGzipFile.

Test plan

• New GzipParser unit tests in unit_tests/sources/declarative/decoders/test_composite_decoder.py: gzipped payload and non-gzip pass-through, each over both a seekable and a truly non-seekable stream (seek()/tell() raise io.UnsupportedOperation), a stream that returns one byte per read(), empty payload, and nested GzipParser(GzipParser(CsvParser)) over a singly-gzipped body.
• New create_gzip_decoder tests in unit_tests/sources/declarative/parsers/test_model_to_component_factory.py covering emit_connector_builder_messages True and False against each header shape above, with preload_content=False, decode_content=False fixtures matching the production raw path — including the double-layer Content-Encoding + Content-Type regression case. Builder mode had no coverage at all before.
• poetry run pytest unit_tests/sources/declarative/decoders/ unit_tests/sources/declarative/parsers/test_model_to_component_factory.py, poetry run ruff format --check . && poetry run ruff check ., poetry run mypy --config-file mypy.ini airbyte_cdk.

Link to Devin session: https://app.devin.ai/sessions/3b5024f8f9bf4de88ec1abad6fb688fb

Important

Auto-merge enabled.

This PR is set to merge automatically when all requirements are met.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1787193372-gzip-decoder-passthrough#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1787193372-gzip-decoder-passthrough

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[Copilot]

Pull request overview

This PR fixes inconsistent gzip handling in the declarative GzipDecoder between Connector Builder “Test read” and real syncs by making GzipParser detect gzip content from the payload (magic bytes) instead of relying on response headers.

Changes:

• Update GzipParser to peek the first bytes of the stream, gunzip only when gzip magic bytes are present, otherwise pass the stream through unchanged (including support for non-seekable/short-read streams via a prefixed wrapper).
• Make create_gzip_decoder use GzipParser in both Builder and sync modes (including as the by_headers fallback parser) for consistent behavior.
• Add unit tests covering gzip payload detection, pass-through behavior, short-read streams, nested gzip parsers, and factory behavior across Builder vs sync modes.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
airbyte_cdk/sources/declarative/decoders/composite_raw_decoder.py	Implements payload-based gzip detection in GzipParser and introduces a prefixed stream wrapper to support non-seekable streams.
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py	Switches gzip decoder construction to use GzipParser consistently in Builder and sync modes (including as fallback).
unit_tests/sources/declarative/decoders/test_composite_decoder.py	Adds focused GzipParser unit tests for gzip detection, short reads, non-seekable streams, empty payloads, and nested gzip parsers.
unit_tests/sources/declarative/parsers/test_model_to_component_factory.py	Adds factory-level tests ensuring create_gzip_decoder handles gzip payloads across header shapes and Builder vs sync modes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

PyTest Results (Fast)

4 363 tests  +97   4 352 ✅ +98   7m 54s ⏱️ +54s
    1 suites ± 0      11 💤  -  1 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 511654b. ± Comparison against base commit db6f309.

This pull request removes 1 and adds 98 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_missing_path_in_quota_status_response_raises_config_error

unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_duplicate_unavailable_status_codes_are_rejected
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_exhausted_tracked_token_rotates_onto_an_untracked_token_without_waiting
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_missing_path_in_quota_status_response_raises_system_error
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_missing_quota_path_always_raises[with_opt_in]
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_missing_quota_path_always_raises[without_opt_in]
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_refresh_after_exhaustion_skips_the_reseed_when_a_token_is_untracked
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_status_outside_the_opt_in_list_still_fails
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_unavailable_status_codes_are_threaded_through_the_factory
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_unavailable_status_is_untracked_and_never_blocks
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_unavailable_status_without_opt_in_still_fails
…

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 366 tests  +97   4 354 ✅ +97   9m 50s ⏱️ - 1m 14s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 511654b. ± Comparison against base commit db6f309.

This pull request removes 1 and adds 98 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_missing_path_in_quota_status_response_raises_config_error

unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_duplicate_unavailable_status_codes_are_rejected
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_exhausted_tracked_token_rotates_onto_an_untracked_token_without_waiting
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_missing_path_in_quota_status_response_raises_system_error
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_missing_quota_path_always_raises[with_opt_in]
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_missing_quota_path_always_raises[without_opt_in]
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_refresh_after_exhaustion_skips_the_reseed_when_a_token_is_untracked
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_status_outside_the_opt_in_list_still_fails
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_unavailable_status_codes_are_threaded_through_the_factory
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_unavailable_status_is_untracked_and_never_blocks
unit_tests.sources.declarative.auth.test_rate_limited_multiple_token ‑ test_unavailable_status_without_opt_in_still_fails
…

♻️ This comment has been updated with latest results.

[Patrick Nilan (pnilan)]

🤖 This PR review was generated by an AI Agent.

Requesting changes for two gzip parity/test-validity gaps and one test-hardening issue. The previously discussed typing-extensions concern is intentionally omitted because Pydantic already guarantees a sufficient version.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: caa77e87-0509-420a-be9c-4aef8a155f4d

📥 Commits

Reviewing files that changed from the base of the PR and between db6f309 and 511654b.

📒 Files selected for processing (4)

• airbyte_cdk/sources/declarative/decoders/composite_raw_decoder.py
• airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
• unit_tests/sources/declarative/decoders/test_composite_decoder.py
• unit_tests/sources/declarative/parsers/test_model_to_component_factory.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds gzip signature detection with stream-byte replay and updates decoder selection for transport and content compression. Tests cover seekable and non-seekable streams, short reads, plain payloads, nested gzip data, and HTTP responses.

Changes

Gzip decoder flow

Layer / File(s)	Summary
Stream detection and replay airbyte_cdk/sources/declarative/decoders/composite_raw_decoder.py, unit_tests/sources/declarative/decoders/test_composite_decoder.py	GzipParser inspects the gzip signature and preserves consumed bytes for non-gzip streams. Tests cover non-seekable streams, short reads, empty payloads, plain payloads, and nested gzip parsing.
Header-based decoder selection airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py, unit_tests/sources/declarative/parsers/test_model_to_component_factory.py	create_gzip_decoder separates transport-level gzip handling from content-level gzip parsing. Parameterized tests cover compression headers, nested compression, and connector-builder message settings.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 51165

The change makes gzip payload handling consistent across sync and Builder test reads, with targeted coverage for compressed, uncompressed, nested, empty, and non-seekable inputs; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: tolik0

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 5.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 20 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: detecting gzip payloads in GzipParser instead of relying on headers.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch devin/1787193372-gzip-decoder-passthrough

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}