fix(oauth): surface token refresh retry exhaustion as transient_error

[devin-ai-integration]

Summary

Resolves https://github.com/airbytehq/oncall/issues/13684:

• https://github.com/airbytehq/oncall/issues/13684

Close on merge: yes — internally filed, no customer to confirm with.

Related to https://github.com/airbytehq/oncall/issues/13683:

• https://github.com/airbytehq/oncall/issues/13683

When an OAuth token endpoint keeps returning 429/5xx for the whole 300s refresh backoff window, the raw DefaultBackoffException(failure_type=transient_error) escaped the authenticator, and ConcurrentReadProcessor.on_exception re-wrapped it as system_error with the message An unexpected error occurred in stream <stream>: DefaultBackoffException. Provider outages (seen with source-gitlab during a GitLab.com 50X incident) were therefore routed to oncall as system errors.

Fix

1. AbstractOauth2Authenticator (covers Oauth2Authenticator, SingleUseRefreshTokenOauth2Authenticator, DeclarativeOauth2Authenticator, and the declarative single-use variant — all go through _make_handled_request):

   @backoff.on_exception(...)            # unchanged decorator + body, renamed
   def _make_handled_request_with_backoff(self): ...
   
   def _make_handled_request(self):      # same name/callers as before
       try:
           return self._make_handled_request_with_backoff()
       except DefaultBackoffException as e:   # only raised once retries are exhausted
           raise AirbyteTracedException(
               message=<deterministic per status, see below>,
               internal_message=f"OAuth token refresh retries were exhausted. {self._build_provider_response_info(e)}",
               failure_type=e.failure_type or FailureType.transient_error,
           ) from e

   User-facing messages:

   • 429 → OAuth token refresh failed because the provider's token endpoint rate limited the request (HTTP 429).
   • 5xx → OAuth token refresh failed because the provider's token endpoint was temporarily unavailable (HTTP 503).

   No class names, URLs, or response bodies in the message. The internal message goes through _build_provider_response_info (redacts credentials and truncates) rather than str(e), because with send_refresh_request_as_query_params the request URL carries the refresh token.

2. As a backstop, ConcurrentReadProcessor.on_exception now uses BaseBackoffException.failure_type (when set) for non-ATE exceptions instead of always using system_error. The templated message is unchanged, and AirbyteTracedException.from_exception is not touched. Only raw backoff exceptions that reach the stream thread are affected, since HttpClient._send_with_retry already converts its own.

Unchanged: refresh-token rejection (config_error), network-error handling (ConnectionError/timeouts still propagate out of _make_handled_request, then get wrapped as transient_error in refresh_access_token), and the generic system_error fallback. This does not touch http_client.py, so it should not conflict with #1115.

Reproduction (before fix): I used a DeclarativeOauth2Authenticator against a mocked token endpoint that always returns a 503 HTML page, with time frozen and time.sleep advancing the clock. Backoff gave up after 11 tries (Giving up _make_handled_request(...) after 11 tries (DefaultBackoffException ...)), and on_exception emitted An unexpected error occurred in stream users: DefaultBackoffException / system_error. After the fix the same script emits the 503 message above with transient_error.

Out of scope (for follow-up): In the same incident, source-gitlab's manifest maps every 401 to FAIL/config_error ("Unable to refresh the access_token, please re-authenticate..."), which set the job-level failure to config_error while the token endpoint was down. That is a connector manifest change, not part of this PR.

Test Coverage

New tests. Each one fails on main and passes with this change:

• test_requests_native_auth.py::TestRefreshAccessTokenRetryExhaustion: Oauth2Authenticator with 503/500/429, and SingleUseRefreshTokenOauth2Authenticator with 503. Checks for ATE + transient_error + the exact message, no class name/URL/HTML in the message, HTTP <code> + body in internal_message, __cause__ is DefaultBackoffException, and that the request was retried. This class is not frozen with freezegun because the class-level freeze in the neighbouring classes would block mock_sleep from moving the backoff clock forward.
• declarative/auth/test_oauth.py: the same 503 case for DeclarativeOauth2Authenticator.
• test_concurrent_read_processor.py::test_on_exception_non_ate_backoff_exception_preserves_failure_type: transient_error→transient_error, config_error→config_error, None→system_error. The templated message is unchanged.

Local run: requests_native_auth/, declarative/auth/test_oauth.py, test_concurrent_read_processor.py, and test_http_client.py all pass (274 passed). ruff check, ruff format --check, and mypy on the changed source files are clean.

Declarative-First Evaluation

Not applicable: this is a CDK runtime change, not a connector manifest change.

Link to Devin session: https://app.devin.ai/sessions/9afc93d4b13248feb7f6acc6f174e7ed
Open in Devin Desktop: https://app.devin.ai/desktop/session/9afc93d4b13248feb7f6acc6f174e7ed?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1791877713-oauth-refresh-retry-exhaustion-transient#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1791877713-oauth-refresh-retry-exhaustion-transient

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 069 tests  +6   5 057 ✅ +6   9m 14s ⏱️ - 2m 7s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit f10b555. ± Comparison against base commit 42f6023.

[github-actions]

PyTest Results (Full)

5 072 tests  +6   5 060 ✅ +7   12m 20s ⏱️ + 2m 33s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌  - 1 

Results for commit f10b555. ± Comparison against base commit 42f6023.