feat(low-code): response-aware ConstantBackoffStrategy with fall-through on falsy renders

[devin-ai-integration]

Overview

👉 TL;DR: Low-code connectors can now compute a constant retry wait from the failed response's body, headers or the retry attempt number. When the expression produces nothing, the strategy steps aside so the next configured backoff strategy handles the retry. Specifically, ConstantBackoffStrategy.backoff_time_in_seconds now interpolates response, headers and attempt_count (previously only config). Falsy, zero, non-numeric and UndefinedError renders return None before jitter is applied.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/17716:

• https://github.com/airbytehq/airbyte-internal-issues/issues/17716

Close on merge: no. The issue is internally filed, but its acceptance criteria also require an SDM release and one adopter migration (https://github.com/airbytehq/airbyte-internal-issues/issues/17790 to https://github.com/airbytehq/airbyte-internal-issues/issues/17793).

Part of epic https://github.com/airbytehq/airbyte-internal-issues/issues/17704 (delivery step 1). The issue was requested by the AI-fix workflow (Devin Bot).

Changes

• ConstantBackoffStrategy.backoff_time() evaluates the expression on every call with:
  • config
  • response: the parsed JSON body; {} when the body isn't JSON or the input is an exception/None (same semantic as HttpResponseFilter._safe_response_json). A JSON list is passed through as-is.
  • headers: the response headers; {} on exception.
  • attempt_count
• It returns None before jitter is applied when the render is falsy (None, "", 0), comes out as 0 seconds, is non-numeric (a warning is logged), or raises jinja UndefinedError (e.g. response.get(...) on a list body). DefaultErrorHandler / HttpClient then fall through to the next backoff_strategies entry or the default backoff. Other evaluation errors, such as ZeroDivisionError, still propagate.
• declarative_component_schema.yaml changes:
  • interpolation_context adds response, headers, attempt_count.
  • The descriptions document the fall-through behavior.
  • New examples come from adopter connectors: google-search-console, amazon-seller-partner (a safe reciprocal of x-amzn-RateLimit-Limit) and pinterest (regex_search 'Retry after N seconds' or min(2 ** attempt_count, 120)).
• models/declarative_component_schema.py: only the ConstantBackoffStrategy block is updated, to match the schema. It was hand-edited because poe assemble with the pinned datamodel_code_generator==0.26.3 causes unrelated churn across the repo.
• ModelToComponentFactory needed no change.

Example from source-google-search-console, which would replace LoadQuotaBackoffStrategy:

backoff_strategies:
  - type: ConstantBackoffStrategy
    backoff_time_in_seconds: "{{ 900 if 'load quota exceeded' in ((response.get('error') or {}).get('message') or '') | lower else 0 }}"
  - type: ConstantBackoffStrategy
    backoff_time_in_seconds: 60
    jitter_range_in_seconds: 15

Review Spotlight

Reviewers with limited time, please review first:

• constant_backoff_strategy.py#L61-L107: the new context and the fall-through rules, including the UndefinedError catch
• declarative_component_schema.yaml#L699: the documented examples

Risks / behavior changes

• 0 + jitter_range_in_seconds: this used to return random.uniform(0, 2*jitter) and never fell through. It now returns None. In the airbyte repo, only source-google-search-console sets jitter, and it uses a base of 60.
• 0 without jitter: this now returns None instead of 0.0. Nothing observable changes, because DefaultErrorHandler and HttpClient already check if backoff:.
• Non-numeric render: a value that is non-numeric (including from config) used to raise ValueError: could not convert string to float. It now logs a warning and falls through. This matches how the custom adopter classes and WaitTimeFromHeader handle unparseable values.

Design decision: no status_code in the context

LinkedInAdsDataVolumeBackoffStrategy also checks status_code == 429, and the new context doesn't expose a status code. I kept the context the same as HttpResponseFilter (response, headers):

• A backoff strategy only runs for responses the error handler already marked as retryable.
• The 45M-metric message only appears in LinkedIn's 429 body.

If exact parity is needed, the 429 check can be expressed declaratively by giving 429 its own DefaultErrorHandler (via response_filters with http_codes: [429]) inside a CompositeErrorHandler. Adding a status_code variable would be a small follow-up if reviewers prefer it.

Follow-ups

• Release the CDK and source-declarative-manifest, then migrate the adopters: https://github.com/airbytehq/airbyte-internal-issues/issues/17790, https://github.com/airbytehq/airbyte-internal-issues/issues/17791, https://github.com/airbytehq/airbyte-internal-issues/issues/17792, https://github.com/airbytehq/airbyte-internal-issues/issues/17793.
• Out of scope here: the 403 token-invalidation side effect in the amazon-seller-partner class (the epic plans a REFRESH_TOKEN_THEN_RETRY filter for it).

Declarative-First Evaluation

N/A. This change is to the declarative framework itself: it extends the built-in ConstantBackoffStrategy so connectors can drop custom components. No custom component is added.

Test Coverage

New unit tests in test_constant_backoff.py and test_default_error_handler.py. They cover:

• body-based (GSC, LinkedIn), header-based (Amazon reciprocal) and attempt_count-based (Pinterest) expressions
• non-dict and non-JSON bodies
• unguarded .get on a list body
• falsy and 0 renders, with and without jitter
• non-numeric renders
• RequestException and None input
• a factory round-trip
• DefaultErrorHandler falling through to a second strategy (GSC 900 then 60 + jitter, and {{ 0 }} + jitter falling through to ExponentialBackoffStrategy)

The 26 new tests fail on main and pass with the change. Locally, error_handlers/, test_model_to_component_factory.py and test_http_client.py all pass (586 tests), and ruff check, ruff format --check and mypy on the touched module are clean.

Link to Devin session: https://app.devin.ai/sessions/c1ef01a795c94eb3ae78f1aea714130f
Open in Devin Desktop: https://app.devin.ai/desktop/session/c1ef01a795c94eb3ae78f1aea714130f?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1791298881-response-aware-constant-backoff#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1791298881-response-aware-constant-backoff

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 050 tests  +25   5 038 ✅ +25   11m 19s ⏱️ +15s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 9aef805. ± Comparison against base commit 5381b8f.

This pull request removes 1 and adds 26 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_with_jitter_bounds[zero_base]

unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[empty_string_render-15]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[empty_string_render-None]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[none_render-15]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[none_render-None]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[plain_zero-15]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[plain_zero-None]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[zero_render-15]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[zero_render-None]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_from_factory
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_non_numeric_render_falls_through
…

[github-actions]

PyTest Results (Full)

5 053 tests  +25   5 041 ✅ +25   16m 10s ⏱️ + 3m 30s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 9aef805. ± Comparison against base commit 5381b8f.

This pull request removes 1 and adds 26 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_with_jitter_bounds[zero_base]

unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[empty_string_render-15]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[empty_string_render-None]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[none_render-15]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[none_render-None]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[plain_zero-15]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[plain_zero-None]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[zero_render-15]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_falsy_render_falls_through[zero_render-None]
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_from_factory
unit_tests.sources.declarative.requesters.error_handlers.backoff_strategies.test_constant_backoff ‑ test_constant_backoff_non_numeric_render_falls_through
…