Skip to content

fix: pass API budget to custom requesters - #1034

Merged
Daryna Ishchenko (darynaishchenko) merged 2 commits into
mainfrom
devin/1779720474-custom-requester-api-budget
May 25, 2026
Merged

Daryna Ishchenko (darynaishchenko) merged 2 commits into
mainfrom
devin/1779720474-custom-requester-api-budget

Conversation

@darynaishchenko

@darynaishchenko Daryna Ishchenko (darynaishchenko) commented May 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Pass the root declarative api_budget into custom components that accept an api_budget field, covering CustomRequester subclasses of HttpRequester instantiated through the generic custom component path.
  • Preserve connector-owned custom requester behavior: if a custom requester sets or receives its own api_budget, that value still wins; this change does not sync or override _http_client after construction.
  • Add regression coverage for both cases: a custom requester receives the manifest HttpAPIBudget when it does not override it, and a custom requester that sets its own default budget is not overwritten.
  • This fixes the source-amazon-seller-partner report creation regression where ReportCreationRequester extends HttpRequester but was not receiving the manifest API budget from the CDK.

Review & Testing Checklist for Human

  • Confirm the generic custom-component pass-through is appropriate for custom components that declare api_budget.
  • Confirm connector-specific custom requesters that intentionally override self.api_budget in __post_init__() should continue owning their own budget behavior.
  • Verify source-amazon-seller-partner picks up this CDK change in the connector dependency flow before release.
  • Optional end-to-end check: run an Amazon Seller Partner report stream sync and confirm POST /reports/2021-06-30/reports and the preflight GET /reports/2021-06-30/reports are rate-limited by the manifest api_budget.

Notes

Requested by Daryna Ishchenko.

Useful context from the related closed PR: some connectors already use custom requester API-budget overrides as a workaround or for connector-specific behavior. This PR only passes the CDK budget into the constructor path; it intentionally does not apply CDK-side _http_client sync logic.

Local checks run:

  • poetry run ruff check airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py unit_tests/sources/declarative/parsers/test_model_to_component_factory.py unit_tests/sources/declarative/parsers/testing_components.py
  • poetry run pytest unit_tests/sources/declarative/parsers/test_model_to_component_factory.py::test_api_budget_passed_to_custom_requester unit_tests/sources/declarative/parsers/test_model_to_component_factory.py::test_api_budget_does_not_override_custom_requester_default_value -q

Link to Devin session: https://app.devin.ai/sessions/2f69c7eea1164909b1e03b09862574a1
Requested by: Daryna Ishchenko (@darynaishchenko)

Summary by CodeRabbit

  • Bug Fixes

    • Fixed API budget propagation to custom component requesters. Custom requesters now properly inherit rate limiting configuration from the declarative source manifest, enabling consistent API rate limiting behavior across custom components.
  • Tests

    • Added comprehensive unit tests for API budget propagation to custom requesters, verifying budget settings are properly shared with underlying HTTP clients and do not override existing default values.

Review Change Stack

@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1779720474-custom-requester-api-budget#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1779720474-custom-requester-api-budget

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review May 25, 2026 14:50
@devin-ai-integration
devin-ai-integration Bot marked this pull request as draft May 25, 2026 14:54
@coderabbitai

coderabbitai Bot commented May 25, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Factory now conditionally injects its root api_budget into custom component constructors when the component type declares api_budget; tests verify propagation to a CustomRequester, identity-sharing with its HTTP client, and that a requester’s explicit default api_budget is not overridden.

Changes

API Budget Injection to Custom Components

Layer / File(s) Summary
Conditional api_budget injection in create_custom_component
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
create_custom_component inspects the custom component class's type hints and injects self._api_budget into kwargs when the class declares an api_budget parameter and the caller did not supply one.
Tests and testing helper for api_budget propagation
unit_tests/sources/declarative/parsers/test_model_to_component_factory.py, unit_tests/sources/declarative/parsers/testing_components.py
Adds imports for CustomRequesterModel and HttpAPIBudget, two tests (test_api_budget_passed_to_custom_requester, test_api_budget_does_not_override_custom_requester_default_value) validating propagation and non-overriding behavior, and a TestingRequesterWithDefaultBudget helper that sets api_budget = None for the override test.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Would you like me to suggest a short unit-test assertion that checks the exact number of policies more explicitly (wdyt?)

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main change: passing API budget to custom requesters, which aligns with the primary objective of the PR.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch devin/1779720474-custom-requester-api-budget

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented May 25, 2026 •

Copy link
Copy Markdown

PyTest Results (Fast)

4 071 tests  +2   4 060 ✅ +2   7m 53s ⏱️ +9s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 5fa728f. ± Comparison against base commit f67a9d9.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented May 25, 2026 •

Copy link
Copy Markdown

PyTest Results (Full)

4 074 tests  +2   4 062 ✅ +2   10m 50s ⏱️ -7s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 5fa728f. ± Comparison against base commit f67a9d9.

♻️ This comment has been updated with latest results.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

6 similar comments
@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

1 similar comment
@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

4 similar comments
@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

3 similar comments
@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

5 similar comments
@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@devin-ai-integration

This comment has been minimized.

@darynaishchenko
Daryna Ishchenko (darynaishchenko) merged commit 57110be into main May 25, 2026
34 checks passed
@darynaishchenko
Daryna Ishchenko (darynaishchenko) deleted the devin/1779720474-custom-requester-api-budget branch May 25, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants