[Daryna Ishchenko (darynaishchenko)]

Summary

Adds a new declarative authenticator, RateLimitedMultipleTokenAuthenticator, that generalizes source-github's MultipleTokenAuthenticatorWithRateLimiter so it can be reused by any connector during declarative migrations (context: https://github.com/airbytehq/airbyte-internal-issues/issues/16512).

Behavior (per outgoing request, via AuthBase.__call__):

• Requests are classified into named quota pools (quotas) via HttpRequestRegexMatchers; a pool with no matchers is the default (generalizes GitHub's hardcoded "graphql" in path REST-vs-GraphQL split).
• Each token tracks a local per-pool counter, seeded once per token from quota_status_source (e.g. GitHub GET /rate_limit) using remaining_path/reset_path/limit_path dpath extraction. Seeding is lazy (first request) and repeated after an exhaustion wait.
• Active token's counter is decremented; on exhaustion the authenticator rotates through tokens; when all are drained it sleeps until the earliest reset (heartbeat logs, floored at MIN_EXHAUSTION_WAIT = 5s so stale reset timestamps can't busy-loop the quota endpoint, bounded by max_wait_time, else raises transient_error).
• Proactive throttling: once every token drops below max(budget_min_reserve, budget_reserve_fraction * limit), delays of seconds_to_reset / total_remaining (capped at 10s) are injected for graceful degradation instead of a hard wall.

Thread safety (new vs. the source-github implementation, required for the concurrent CDK):

• Token selection + decrement happen atomically under an RLock; _acquire_call returns the token chosen under the lock and the request header is built from that value, so a request is always signed with the token whose counter was decremented (no TOCTOU with concurrent rotation). All sleeps occur outside the lock.
• Double-checked re-seed via a separate _refresh_lock so only one thread refreshes counters after an exhaustion wait.
• ModelToComponentFactory.create_rate_limited_multiple_token_authenticator caches instances keyed on the model definition, so all streams share one counter set (same idea as the shared api_budget).

Example manifest usage (source-github):

authenticator:
  type: RateLimitedMultipleTokenAuthenticator
  tokens: "{{ config['credentials']['personal_access_token'] }}"  # comma-separated PATs, or a list
  auth_method: "token"
  quota_status_source:
    type: QuotaStatusSource
    url: "{{ config.get('api_url', 'https://api.github.com') }}/rate_limit"
  quotas:
    - type: TokenQuota
      name: rest
      remaining_path: ["resources", "core", "remaining"]
      reset_path: ["resources", "core", "reset"]
      limit_path: ["resources", "core", "limit"]
    - type: TokenQuota
      name: graphql
      remaining_path: ["resources", "graphql", "remaining"]
      reset_path: ["resources", "graphql", "reset"]
      matchers:
        - type: HttpRequestRegexMatcher
          url_path_pattern: "/graphql"

Changes:

• declarative_component_schema.yaml: new RateLimitedMultipleTokenAuthenticator, QuotaStatusSource, TokenQuota definitions; added to HttpRequester.authenticator and SelectiveAuthenticator.authenticators unions (models regenerated; new classes applied on top of the checked-in file to avoid unrelated codegen churn).
• airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py: runtime implementation.
• model_to_component_factory.py: create_rate_limited_multiple_token_authenticator with token-string splitting, interpolation, and shared-instance caching.
• Unit tests: seeding/header injection, quota matching, rotation, exhaustion wait/raise, budget throttling, thread safety (no lost decrements across 200 threads; signed token matches decremented token under concurrent rotation), factory token parsing and instance sharing.

Requested by Daryna Ishchenko (darynaishchenko).

Link to Devin session: https://app.devin.ai/sessions/3ced8f6a11d44ed4abc517221294f767
Requested by: Daryna Ishchenko (@darynaishchenko)

Summary by CodeRabbit

• New Features
  • Added a declarative authenticator that rotates across multiple tokens based on quota availability.
  • Added configurable quota pools, request matching, proactive throttling, automatic quota refresh, and reset-aware waiting.
  • Enabled the authenticator in declarative HTTP requester and selective authentication configurations.
  • Added support for configuring tokens, quota status endpoints, headers, and throttling limits.
• Bug Fixes
  • Improved authentication specification generation to preserve configured values and remain consistent across repeated calls.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1783347505-rate-limited-multiple-token-authenticator#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1783347505-rate-limited-multiple-token-authenticator

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 181 tests  +23   4 169 ✅ +23   8m 2s ⏱️ +3s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 89d6750. ± Comparison against base commit 013316a.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 184 tests   4 172 ✅  12m 7s ⏱️
    1 suites     12 💤
    1 files        0 ❌

Results for commit 89d6750.

♻️ This comment has been updated with latest results.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 898e5edd-d50a-4674-a4bc-3b5eeeb68f1a

📥 Commits

Reviewing files that changed from the base of the PR and between 0cb6682 and 8ae802c.

📒 Files selected for processing (1)

• airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py

🚧 Files skipped from review as they are similar to previous changes (1)

• airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py

---

📝 Walkthrough

Walkthrough

Adds a declarative authenticator that rotates multiple tokens using quota pools, refreshes quota state, waits for resets, and applies proactive throttling. Schemas, factory construction and caching, public exports, serialization, and tests are updated.

Changes

Rate-limited multi-token authentication

Layer / File(s)	Summary
Authenticator contracts and declarative registration airbyte_cdk/sources/declarative/auth/__init__.py, airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py, airbyte_cdk/sources/declarative/declarative_component_schema.yaml, airbyte_cdk/sources/declarative/models/declarative_component_schema.py	Defines quota and authenticator models, exports the authenticator, and registers it in authentication configurations.
Quota-aware token rotation airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py, unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py	Adds quota matching, synchronized token rotation, reset waits, quota refreshes, throttling, and runtime validation tests.
Declarative factory construction and reuse airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py, unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py	Builds authenticators from declarative models, interpolates settings, parses token forms, and reuses matching instances.

Advanced authentication spec serialization

Layer / File(s)	Summary
Defensive advanced-auth enum conversion airbyte_cdk/sources/declarative/spec/spec.py, unit_tests/sources/declarative/spec/test_spec.py	Serializes a copied advanced-auth dictionary and converts enum values without mutating the configured model.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Request as PreparedRequest
  participant Auth as RateLimitedMultipleTokenAuthenticator
  participant Quota as quota-status endpoint
  Request->>Auth: authenticate request
  Auth->>Quota: fetch quota status for tokens
  Quota-->>Auth: return remaining and reset values
  Auth->>Auth: match quota pool and reserve token
  Auth-->>Request: attach authorization header

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely identifies the main change: adding the RateLimitedMultipleTokenAuthenticator component.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch devin/1783347505-rate-limited-multiple-token-authenticator

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

airbyte_cdk/sources/declarative/spec/spec.py (1)

58-69: 🗄️ Data Integrity & Integration | 🔵 Trivial | 🏗️ Heavy lift

Could we normalize a serialized copy instead of mutating the typed model?

generate_spec() assigns .value strings back into Enum-backed fields, so after the first call self.advanced_auth no longer conforms to its declared model contract. This fixes repeated serialization, but can break callers that reuse advanced_auth and expect AuthFlowType/ScopesJoinStrategy instances. Could you build the output dictionary first and normalize that copy instead, wdyt?

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@airbyte_cdk/sources/declarative/spec/spec.py` around lines 58 - 69, The
generate_spec() serialization flow currently mutates Enum-backed fields on
self.advanced_auth and its nested oauth configuration. Build or obtain a
serialized output copy first, then normalize auth_flow_type and
scopes_join_strategy on that copy only, preserving the typed AuthFlow and OAuth
connector input models for repeated callers.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py`:
- Around line 165-205: The _acquire_call exhaustion path currently applies
_max_wait_time to each sleep rather than the full retry loop. Track a single
exhaustion deadline across repeated reset/refresh iterations, bound each wait by
the remaining time, and raise the existing transient AirbyteTracedException when
that cumulative budget is exhausted; add a regression test covering repeated
zero-quota refreshes.

In `@airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py`:
- Around line 725-726: The shared authenticator registry must be reused by
nested ModelToComponentFactory instances so parent and child streams share
rate-limit counters. Update the parent-stream factory creation paths around
ModelToComponentFactory and the related factory construction at 4514-4526 to
accept and propagate the parent’s _rate_limited_authenticators mapping, rather
than initializing a fresh registry; preserve the existing local initialization
only for root factories.

In `@unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py`:
- Around line 186-211: Update
test_thread_safety_header_token_matches_decremented_token to capture exceptions
raised by each worker in make_call, using the existing thread coordination or an
equivalent mechanism, and assert that the collected exception list is empty
after all threads join. Keep the signed token collection and per-token
remaining-quota assertions unchanged.

---

Nitpick comments:
In `@airbyte_cdk/sources/declarative/spec/spec.py`:
- Around line 58-69: The generate_spec() serialization flow currently mutates
Enum-backed fields on self.advanced_auth and its nested oauth configuration.
Build or obtain a serialized output copy first, then normalize auth_flow_type
and scopes_join_strategy on that copy only, preserving the typed AuthFlow and
OAuth connector input models for repeated callers.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 53a02d6d-190c-4424-9f9f-d4b367840869

📥 Commits

Reviewing files that changed from the base of the PR and between dd9558c and 891fb08.

📒 Files selected for processing (7)

• airbyte_cdk/sources/declarative/auth/__init__.py
• airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py
• airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• airbyte_cdk/sources/declarative/models/declarative_component_schema.py
• airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
• airbyte_cdk/sources/declarative/spec/spec.py
• unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py

[Anatolii Yatsuk (tolik0)]

Deep review with local reproductions (ran the PR's tests, drove it end-to-end through ConcurrentDeclarativeSource manifests, replicated the model codegen, and compared behavior against source-github's MultipleTokenAuthenticatorWithRateLimiter). The core machinery works and thread safety is done correctly, but two must-fix items remain — see inline comments:

1. High: the shared-instance cache key is poisoned by $parameters propagation, so streams get separate counter sets in realistic manifests (reproduced on this head).
2. Medium: the checked-in generated models file is out of sync with the schema (hand-edited, not regenerated).

Also a medium on max_wait_time not being a cumulative bound (reproduced infinite reseed loop), plus several low-severity robustness notes. The substream-factory registry sharing added in d92aaac was verified working with a parent/child manifest.

[coderabbitai]

🧹 Nitpick comments (3)

unit_tests/sources/declarative/spec/test_spec.py (1)

165-180: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Cover nested scopes_join_strategy preservation.

This test does not execute the nested OAuth scope conversion. Could you add an OAuth connector input specification with a scopes_join_strategy enum, then assert that the generated value is serialized and the source model remains unchanged after both calls? wdyt?

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@unit_tests/sources/declarative/spec/test_spec.py` around lines 165 - 180,
Extend test_generate_spec_is_idempotent_and_does_not_mutate_the_model to include
an OAuth connector input specification containing a scopes_join_strategy enum.
After both generate_spec calls, assert the generated specifications serialize
and preserve that nested strategy, while also verifying the source model’s
scopes_join_strategy remains unchanged and the two generated results are
identical.

airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (1)

4585-4589: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Should this be a config error rather than a bare ValueError?

A calendar-unit max_wait_time is a manifest/config mistake. The authenticator itself reports comparable problems with AirbyteTracedException(failure_type=FailureType.config_error, ...), which yields a user-facing message instead of a system error trace. Would raising the same exception type keep the failure classification consistent, wdyt?

♻️ Proposed change

         if not isinstance(max_wait_time, datetime.timedelta):
-            raise ValueError(
-                f"max_wait_time must be a fixed-length ISO 8601 duration (e.g. 'PT2H'); "
-                f"calendar-unit durations like '{max_wait_time_str}' are not supported"
-            )
+            raise AirbyteTracedException(
+                failure_type=FailureType.config_error,
+                internal_message=(
+                    f"max_wait_time must be a fixed-length ISO 8601 duration (e.g. 'PT2H'); "
+                    f"calendar-unit durations like '{max_wait_time_str}' are not supported"
+                ),
+                message="Invalid maximum wait time: use a fixed-length duration such as 'PT2H'.",
+            )

Note: test_factory_rejects_calendar_unit_max_wait_time asserts pytest.raises(ValueError, match="calendar-unit"), so the test would need the new type and a match against the internal message.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py` around
lines 4585 - 4589, Update the calendar-unit validation in the max_wait_time
parsing path to raise AirbyteTracedException with FailureType.config_error
instead of ValueError, preserving the existing descriptive message. Adjust
test_factory_rejects_calendar_unit_max_wait_time to expect the new exception
type and match the internal error message while retaining coverage of the
calendar-unit rejection.

unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py (1)

273-276: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Could the clock patches target the authenticator module instead of the global time module?

patch("time.monotonic", ...) replaces the attribute on the shared time module, so every library in the process sees the fake clock for the duration of the test. requests_mock intercepts the HTTP call, so this currently works, but a module-scoped patch is more isolated. Would patching airbyte_cdk.sources.declarative.auth.rate_limited_multiple_token.time.monotonic / .sleep (or using monkeypatch.setattr on that module) read more precisely here, wdyt?

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py`
around lines 273 - 276, Update the clock patches in the test context around the
rate-limited authenticator to target the authenticator module’s imported time
object, using its rate_limited_multiple_token.time.monotonic and time.sleep
attributes instead of the global time module. Keep the existing fake_sleep and
clock["now"] behavior unchanged while limiting the patch scope to the code under
test.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py`:
- Around line 4585-4589: Update the calendar-unit validation in the
max_wait_time parsing path to raise AirbyteTracedException with
FailureType.config_error instead of ValueError, preserving the existing
descriptive message. Adjust test_factory_rejects_calendar_unit_max_wait_time to
expect the new exception type and match the internal error message while
retaining coverage of the calendar-unit rejection.

In `@unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py`:
- Around line 273-276: Update the clock patches in the test context around the
rate-limited authenticator to target the authenticator module’s imported time
object, using its rate_limited_multiple_token.time.monotonic and time.sleep
attributes instead of the global time module. Keep the existing fake_sleep and
clock["now"] behavior unchanged while limiting the patch scope to the code under
test.

In `@unit_tests/sources/declarative/spec/test_spec.py`:
- Around line 165-180: Extend
test_generate_spec_is_idempotent_and_does_not_mutate_the_model to include an
OAuth connector input specification containing a scopes_join_strategy enum.
After both generate_spec calls, assert the generated specifications serialize
and preserve that nested strategy, while also verifying the source model’s
scopes_join_strategy remains unchanged and the two generated results are
identical.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1af9027d-f2f9-45cb-a745-af3b97a80f0d

📥 Commits

Reviewing files that changed from the base of the PR and between 891fb08 and 0cb6682.

📒 Files selected for processing (7)

• airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py
• airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• airbyte_cdk/sources/declarative/models/declarative_component_schema.py
• airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
• airbyte_cdk/sources/declarative/spec/spec.py
• unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py
• unit_tests/sources/declarative/spec/test_spec.py

🚧 Files skipped from review as they are similar to previous changes (2)

• airbyte_cdk/sources/declarative/models/declarative_component_schema.py
• airbyte_cdk/sources/declarative/declarative_component_schema.yaml