fix(low-code): report why the API rejected the last window or page

[Baz (bazarnov)]

Overview

👉 TL;DR: When a sync keeps narrowing a request and the API still refuses it, the stream's error now records why the API refused in its internal message and stack trace, and no longer wrongly suggests a setup mistake or a cursor that does not shrink the window. Specifically, RequestWindowSplitRequiredException and PageSizeReductionRequiredException keep error_message and describe only the rejection, and the exhaustion errors raised by SimpleRetriever and PageSizeReducer.reduce add it to their internal_message.

Changes

• RequestWindowSplitRequiredException and PageSizeReductionRequiredException store error_message, and their message describes the rejection only: The API rejected the current request window of stream <name> and requires a smaller one: <error_message> (page size: ... a page of stream <name> and asked the connector for a smaller one: ...).
  • The "If this message ends a sync, the stream is not set up ..." advice is removed. These exceptions are raised on every split or reduction, and a traceback prints each chained exception's message, so the advice showed up in the stack_trace of correctly configured streams: once per split level, and in any unrelated failure inside a split child.
  • The advice stays where it applies: RequestWindowSplitNotSupportedException, PageSizeReductionNotSupportedException and the factory's config-time check.
• SimpleRetriever window split exhaustion:
  • Floor: internal_message adds ; the API rejected it: <error_message>. message is unchanged.
  • Depth cap: internal_message adds ; the API still rejected <last window>: <error_message>, and message keeps only its first sentence. The dropped sentence blamed the stream's cursor, but the built-in cursor also reaches the cap by halving correctly on long windows (a 30-day window with PT1S granularity gets there at about 42-minute windows).
• PageSizeReducer.reduce takes the caught exception's error_message and appends ; the API rejected the page: <error_message> to the internal_message of its three exhaustion errors (max attempts, transient floor, page size configured at the floor).
• Tests: two new parametrized SimpleRetriever tests, the two HttpClient exact-message asserts updated, and two end-to-end tests through a real HttpResponseFilter extended.

Review Spotlight

Reviewers with limited time, please review first:

• simple_retriever.py#L643-L680: both window split exhaustion errors
• request_window_split_exception.py#L23-L48: the control-flow message without the advice, and why
• page_size_reducer.py#L113-L169: reduce(error_message) and the first of the three errors it annotates

Risks

• Only trace text changes: the depth cap message, the internal_message of the five exhaustion errors, the message of the two control-flow exceptions, and the message of the closing During the sync, the following streams did not sync successfully error. Records, state, schemas, failure_type and request counts are unchanged.
• failure_type is deliberately untouched: exhaustion stays transient_error as decided in the feat: add SPLIT_REQUEST_WINDOW action for adaptive request-window splitting #1171 review, and a permanent rejection can later be carried by the matched filter's failure_type, which is set per response rather than per stream.
• The control-flow message is user-facing only on a stream that never handles the signal (a retriever not built on SimpleRetriever, or a Python HttpStream), which now gets the rejection instead of the setup advice. No connector relies on that path: the only connector code raising one of these exceptions is source-google-ads, inside a SimpleRetriever that handles it.
• The error handler's error_message now reaches internal_message and every traceback level, so long interpolated text makes those longer. It also reaches the user-facing message of the closing error, which is built from each stream error's internal_message for every stream failure; the stream error's own message stays free of API text. A filter without error_message contributes the default mapping's text for its status code (PayPal's gets HTTP Status Code: 400. Error: Bad request. Please check your request parameters.), or nothing when that status has no default entry (TikTok's 40067 predicate matches HTTP 200). GitHub's filter message ends with "Retrying the same page with a smaller page size.", which now also shows in its exhaustion internal_message and the closing error's message.
• Blast radius, counted on airbyte master @ fc800755 with git grep -l -E '<pattern>' origin/master -- 'airbyte-integrations/connectors/*/manifest.yaml', plus #87641 and #87630, merged after that commit and read at their merge commits:
  • SPLIT_REQUEST_WINDOW|request_window_splitting: 4 manifests (source-amplitude, source-paypal-transaction, source-twilio on SDM 7.32.0, source-tiktok-marketing on SDM 7.33.0), plus source-google-ads (airbyte-cdk ^7.33.0), whose components.py raises RequestWindowSplitRequiredException(error_message=str(error)); that text now reaches the exhaustion internal_message.
  • REDUCE_PAGE_SIZE|page_size_reduction: 1 manifest (source-github, SDM 7.32.0).
  • git grep -n -E '(raise|except).*(RequestWindowSplitRequiredException|PageSizeReductionRequiredException)' origin/master -- 'airbyte-integrations/connectors/*.py': 0 hits; google-ads' raise above landed after fc800755.
  • These connectors' tests assert only failure_type, the unchanged floor and page size messages, or their own failure_message. Their unit tests pass against this branch, run from a git archive of master @ fc800755: amplitude 21, paypal 22, twilio 38, and github 86 in test_manifest_graphql_streams.py and test_components.py, which hold its page size reduction tests (its other files flake or crash locally on main too). TikTok and google-ads assertions were read, not run.

Test plan

• Red/green: git checkout origin/main -- airbyte_cdk/sources/declarative/retrievers/page_size_reducer.py airbyte_cdk/sources/declarative/retrievers/simple_retriever.py airbyte_cdk/sources/streams/http/page_size_reduction_exception.py airbyte_cdk/sources/streams/http/request_window_split_exception.py, then python -m pytest <the tests below> >> 9 failed; git checkout HEAD -- <the same files>, same command >> 9 passed.
  • unit_tests/sources/declarative/retrievers/test_simple_retriever.py::test_given_window_still_rejected_when_splitting_is_exhausted_then_report_the_api_rejection (floor, depth cap)
  • unit_tests/sources/declarative/retrievers/test_simple_retriever.py::test_given_reductions_exhausted_when_read_records_then_report_the_api_rejection (max attempts, minimum page size reached, page size starting at the minimum)
  • unit_tests/sources/streams/http/test_http_client.py: the two exact message asserts
  • unit_tests/sources/declarative/test_concurrent_declarative_source.py::test_given_reductions_exhausted_when_read_then_emit_a_transient_error and ::test_given_split_window_still_rejected_when_read_then_partition_is_not_checkpointed
• python -m pytest unit_tests/sources/declarative/retrievers unit_tests/sources/streams/http unit_tests/sources/streams/concurrent unit_tests/sources/declarative/incremental unit_tests/sources/declarative/requesters unit_tests/sources/declarative/parsers >> 1882 passed.
• python -m pytest unit_tests/sources/declarative/test_concurrent_declarative_source.py -k "not test_read_with_concurrent_and_synchronous_streams" >> 86 passed. The 3 deselected tests crash the interpreter locally on macOS, on main too (Cached parent stream with more than one partition corrupts the shared sqlite request cache under concurrent reads (no such table: responses, file is not a database, no more rows available) #1150).
• ruff format, ruff check (0.11.5) and mypy --config-file mypy.ini (1.14.1) on the changed files: clean.

Fixes #1197

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@bazarnov/fix-split-exhaustion-error-messages#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch bazarnov/fix-split-exhaustion-error-messages

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Full)

4 990 tests  +5   4 978 ✅ +5   9m 37s ⏱️ - 6m 27s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 101a869. ± Comparison against base commit d214b0c.

[github-actions]

PyTest Results (Fast)

4 987 tests  +5   4 975 ✅ +5   10m 59s ⏱️ + 1m 26s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 101a869. ± Comparison against base commit d214b0c.