Skip to content

fix: emit specific error message when API rate limit retries are exhausted #1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Patrick Nilan (pnilan) merged 3 commits into from
Apr 28, 2026
Merged

fix: emit specific error message when API rate limit retries are exhausted #1004

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
caa9f4e
fix: emit specific error message when API rate limit retries are exha…
devin-ai-integration[bot] Apr 28, 2026
0d87243
fix: update remaining test asserting old 429 message
devin-ai-integration[bot] Apr 28, 2026
7221854
style: fix ruff formatting
devin-ai-integration[bot] Apr 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion airbyte_cdk/sources/streams/http/http_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,22 @@ def _send_with_retry(

return response
except BaseBackoffException as e:
self._logger.error(f"Retries exhausted with backoff exception.", exc_info=True)
self._logger.error("Retries exhausted with backoff exception.", exc_info=True)

is_rate_limited = (
isinstance(e.response, requests.Response)
and e.response.status_code == requests.codes.too_many_requests
)

if is_rate_limited:
raise AirbyteTracedException(
internal_message=f"Rate limit retry budget exhausted. Last exception: {e}",
message="API rate limit exceeded.",
failure_type=FailureType.transient_error,
exception=e,
stream_descriptor=StreamDescriptor(name=self._name),
)

raise AirbyteTracedException(
internal_message=f"Exhausted available request attempts. Exception: {e}",
message=f"Exhausted available request attempts. Please see logs for more details. Exception: {e}",
Expand Down
5 changes: 3 additions & 2 deletions unit_tests/sources/streams/http/test_http.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,8 @@ def get_error_handler(self) -> Optional[ErrorHandler]:
send_mock = mocker.patch.object(requests.Session, "send", return_value=req)

with pytest.raises(
AirbyteTracedException, match="Exception: HTTP Status Code: 429. Error: Too many requests."
AirbyteTracedException,
match="API rate limit exceeded.",
):
Comment on lines 224 to 227

Copilot AI Apr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pytest.raises(..., match=...) uses a regex. The trailing . in "API rate limit exceeded." will match any character, so this assertion could pass even if the message changes slightly. Consider escaping the dot (e.g., matching a literal period) or using re.escape(...) to ensure the test asserts the exact message.

Copilot uses AI. Check for mistakes.
list(stream.read_records(SyncMode.full_refresh))
if retries <= 0:
Expand Down Expand Up @@ -316,7 +317,7 @@ def test_raise_on_http_errors_off_429(mocker):
mocker.patch.object(requests.Session, "send", return_value=req)
with pytest.raises(
AirbyteTracedException,
match="Exhausted available request attempts. Please see logs for more details. Exception: HTTP Status Code: 429. Error: Too many requests.",
match="API rate limit exceeded.",
):
Comment on lines 318 to 321

Copilot AI Apr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pytest.raises(..., match=...) treats match as a regex; the . at the end of "API rate limit exceeded." matches any character. Escaping the period (or using re.escape) would make this assertion stricter and avoid accidental false positives.

Copilot uses AI. Check for mistakes.
stream.exit_on_rate_limit = True
list(stream.read_records(SyncMode.full_refresh))
Expand Down
Loading