Skip to content

Fix streaming request failover to retry on error status codes#7

Merged
GoJam11 merged 1 commit into
mainfrom
claude/issue-6-verification-55oec3
Jun 30, 2026
Merged

Fix streaming request failover to retry on error status codes#7
GoJam11 merged 1 commit into
mainfrom
claude/issue-6-verification-55oec3

Conversation

@GoJam11

@GoJam11 GoJam11 commented Jun 30, 2026

Copy link
Copy Markdown
Owner

Summary

This PR fixes the failover behavior for streaming requests to properly retry and fall back when the upstream returns error status codes before any response body is streamed to the client.

Key Changes

  • Removed incorrect streaming request check: Deleted the !retryableStreamRequest condition that was preventing status-based failover for streaming requests. Error status codes are returned before the response body is streamed, making retries safe at that point.
  • Updated test expectations:
    • Changed test "does NOT retry a streaming request on 5xx" to "retries a streaming request when the upstream returns an error status before the first byte" to reflect the corrected behavior
    • Added new test "falls over a streaming request to the next provider on 5xx" to verify model fallback works for streaming requests
  • Added clarifying comments: Documented that error status codes arrive before streaming begins, so failover is safe; only after streaming has started (with 2xx status) should retries be skipped

Implementation Details

The key insight is that when an upstream returns an error status code (4xx, 5xx), the response headers and status are sent before any body content. For streaming requests, this means the error status arrives before the first byte of the stream is sent to the client, so the response can be safely cancelled and retried/fallen back to another provider. Only after a successful (2xx) response has started streaming should retries be disabled, but those cases won't trigger failover anyway since they have success status codes.

https://claude.ai/code/session_01B1f8CF7YQmp9EU3m8yJYFr

此前状态码级别的故障转移被 `!retryableStreamRequest` 一刀切挡掉,
导致流式请求(stream: true)即使在开流前收到 429 / 5xx,也不会按
priority 降级到下一个渠道,而是直接把错误透传给客户端(issue #6)。

错误状态码是在响应体转发给客户端之前返回的(随后会 cancel 掉上游
body),此时重试 / 回退是安全的;真正不可重试的是“已经开始流式响应体
后才出错”,而那种情况状态码是 2xx,本就不会命中 failover。因此去掉该
前置条件即可,无需新增开关。

新增两条流式 failover 端到端测试:开流前 429 重试成功、5xx 按 priority
回退到下一渠道。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B1f8CF7YQmp9EU3m8yJYFr
@GoJam11 GoJam11 merged commit 3f892c5 into main Jun 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants