diff --git a/README.ja.md b/README.ja.md index ce68f0046..ae38af891 100644 --- a/README.ja.md +++ b/README.ja.md @@ -91,7 +91,7 @@ shunt add upstream https://provider.example/docs --print | claude ## プロバイダー -プロバイダーは、順序付き `[[upstreams]]` エントリまたはレガシーな `[providers.]` TOML テーブルです(YAML では、それぞれ対応する sequence または mapping のエントリ)。2 種類のアダプターでほとんどの上流をカバーします。`kind = "anthropic"`(上流が Anthropic Messages を話す場合。別のキーを付けてパススルー可能)と `kind = "responses"`(上流が OpenAI Responses API を話す場合。shunt が Anthropic Messages ⇄ Responses をストリーミング込みで変換)です。3 つ目のネイティブな種類である `kind = "cursor"` は、Cursor の ConnectRPC/protobuf AgentService をブリッジし、Cursor サブスクリプションを同じ Anthropic Messages インターフェース経由で利用できるようにします。 +プロバイダーは、順序付き `[[upstreams]]` エントリまたはレガシーな `[providers.]` TOML テーブルです(YAML では、それぞれ対応する sequence または mapping のエントリ)。2 種類のアダプターでほとんどの上流をカバーします。`kind = "anthropic"`(上流が Anthropic Messages を話す場合。別のキーを付けてパススルー可能)と `kind = "responses"`(上流が OpenAI Responses API を話す場合。shunt が Anthropic Messages ⇄ Responses をストリーミング込みで変換し、Responses API には `stop` パラメーターがないため `stop_sequences` をゲートウェイ側でエミュレート)です。3 つ目のネイティブな種類である `kind = "cursor"` は、Cursor の ConnectRPC/protobuf AgentService をブリッジし、Cursor サブスクリプションを同じ Anthropic Messages インターフェース経由で利用できるようにします。 順序付きアップストリームにより、プロバイダー間のフェイルオーバーが可能になります。宣言順が試行順となり、モデルの `upstream_model` マップが参加するエントリを選択して、公開 id を各バックエンドの id にマッピングします。 diff --git a/README.ko.md b/README.ko.md index 76a5ad565..f2952484d 100644 --- a/README.ko.md +++ b/README.ko.md @@ -92,7 +92,7 @@ shunt add upstream https://provider.example/docs --print | claude ## 프로바이더 -프로바이더는 순서가 있는 `[[upstreams]]` 항목 또는 레거시 `[providers.]` TOML 테이블입니다(YAML에서는 각각 해당 sequence 또는 mapping의 항목). 두 가지 어댑터 종류가 대부분의 업스트림을 커버합니다. `kind = "anthropic"`(업스트림이 Anthropic Messages를 사용하며, 필요하면 다른 키로 패스스루)와 `kind = "responses"`(업스트림이 OpenAI Responses API를 사용하며, shunt가 Anthropic Messages ⇄ Responses를 스트리밍 포함하여 변환)입니다. 세 번째 네이티브 종류인 `kind = "cursor"`는 Cursor의 ConnectRPC/protobuf AgentService를 브리지하여 Cursor 구독을 동일한 Anthropic-Messages 인터페이스로 사용할 수 있게 합니다. +프로바이더는 순서가 있는 `[[upstreams]]` 항목 또는 레거시 `[providers.]` TOML 테이블입니다(YAML에서는 각각 해당 sequence 또는 mapping의 항목). 두 가지 어댑터 종류가 대부분의 업스트림을 커버합니다. `kind = "anthropic"`(업스트림이 Anthropic Messages를 사용하며, 필요하면 다른 키로 패스스루)와 `kind = "responses"`(업스트림이 OpenAI Responses API를 사용하며, shunt가 Anthropic Messages ⇄ Responses를 스트리밍 포함하여 변환하고, Responses API에는 `stop` 파라미터가 없으므로 `stop_sequences`를 게이트웨이 측에서 에뮬레이션)입니다. 세 번째 네이티브 종류인 `kind = "cursor"`는 Cursor의 ConnectRPC/protobuf AgentService를 브리지하여 Cursor 구독을 동일한 Anthropic-Messages 인터페이스로 사용할 수 있게 합니다. 순서가 있는 업스트림은 프로바이더 간 페일오버를 지원합니다. 선언 순서가 시도 순서이며, 모델의 `upstream_model` 맵은 참여할 항목을 선택하고 공개 id를 각 백엔드 id에 매핑합니다. diff --git a/README.md b/README.md index 20fc26c86..d45247868 100644 --- a/README.md +++ b/README.md @@ -91,7 +91,7 @@ The command is offline and read-only: it prints guidance but never edits files, ## Providers -A provider is either an ordered `[[upstreams]]` entry or a legacy `[providers.]` TOML table (under YAML, an entry in the corresponding sequence or mapping). Two adapter kinds cover most upstreams: `kind = "anthropic"` (the upstream speaks Anthropic Messages; passed through, optionally with a different key) and `kind = "responses"` (the upstream speaks the OpenAI Responses API; shunt translates Anthropic Messages ⇄ Responses, streaming included). A third native kind, `kind = "cursor"`, bridges Cursor's ConnectRPC/protobuf AgentService so a Cursor subscription is reachable through the same Anthropic-Messages interface. +A provider is either an ordered `[[upstreams]]` entry or a legacy `[providers.]` TOML table (under YAML, an entry in the corresponding sequence or mapping). Two adapter kinds cover most upstreams: `kind = "anthropic"` (the upstream speaks Anthropic Messages; passed through, optionally with a different key) and `kind = "responses"` (the upstream speaks the OpenAI Responses API; shunt translates Anthropic Messages ⇄ Responses, streaming included, and emulates `stop_sequences` gateway-side because the Responses API has no `stop` parameter). A third native kind, `kind = "cursor"`, bridges Cursor's ConnectRPC/protobuf AgentService so a Cursor subscription is reachable through the same Anthropic-Messages interface. Ordered upstreams enable cross-provider failover. Declaration order is the attempt order; a model's `upstream_model` map selects the participating entries and maps its public id to each backend's id: diff --git a/README.zh-CN.md b/README.zh-CN.md index 18edad0a0..fbaedf2b3 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -89,7 +89,7 @@ shunt add upstream https://provider.example/docs --print | claude ## 提供方 -一个提供方可以是有序的 `[[upstreams]]` 条目,也可以是旧式 `[providers.]` TOML 表(在 YAML 中,分别对应 sequence 或 mapping 中的条目)。两种适配器类型即可覆盖大多数上游:`kind = "anthropic"`(上游讲 Anthropic Messages;透传,可选择换用不同的密钥)和 `kind = "responses"`(上游讲 OpenAI Responses API;shunt 在 Anthropic Messages ⇄ Responses 之间转换,含流式传输)。第三种原生类型 `kind = "cursor"` 桥接 Cursor 的 ConnectRPC/protobuf AgentService,使 Cursor 订阅可通过同一套 Anthropic-Messages 接口访问。 +一个提供方可以是有序的 `[[upstreams]]` 条目,也可以是旧式 `[providers.]` TOML 表(在 YAML 中,分别对应 sequence 或 mapping 中的条目)。两种适配器类型即可覆盖大多数上游:`kind = "anthropic"`(上游讲 Anthropic Messages;透传,可选择换用不同的密钥)和 `kind = "responses"`(上游讲 OpenAI Responses API;shunt 在 Anthropic Messages ⇄ Responses 之间转换,含流式传输,并且由于 Responses API 没有 `stop` 参数,`stop_sequences` 由网关侧模拟)。第三种原生类型 `kind = "cursor"` 桥接 Cursor 的 ConnectRPC/protobuf AgentService,使 Cursor 订阅可通过同一套 Anthropic-Messages 接口访问。 有序上游支持跨提供方故障转移。声明顺序就是尝试顺序;模型的 `upstream_model` 映射选择参与的条目,并将其公开 id 映射到各后端的 id: diff --git a/docs/m1-responses-translation.md b/docs/m1-responses-translation.md index bfeea30a9..8a0ac3124 100644 --- a/docs/m1-responses-translation.md +++ b/docs/m1-responses-translation.md @@ -226,6 +226,66 @@ instruction on how the agent should proceed. Claude Code surfaces only the error `detailed_explanation` / `error_type` fields are not forwarded — upstream treats them as sensitive and never persists them. +## 8b. Emulated `stop_sequences` (issue #605) + +The Responses API has **no `stop` parameter** — Chat Completions does, Responses does not — so a +client's Anthropic `stop_sequences` cannot be forwarded. Before #605 the field was simply never +read, and every Responses upstream silently ignored it; Claude Code's auto-mode permission +classifier sends `stop_sequences: [""]` / `[""]` and the trailing text past the +stop broke its parser, costing a retry per classification. + +shunt emulates them inside the translation instead. `forward` reads the field once +(`adapters/responses/mod.rs`), threads it through `TurnOptions` → `RelayOptions` → +`AnthropicSseMachine::with_stop_sequences`, and **never** adds it to the upstream body. + +**Scanner** (`model/stop_sequences.rs`). Assistant text deltas only — reasoning summaries +(`response.reasoning_summary_text.delta`) and tool-call arguments +(`response.function_call_arguments.delta`) are never scanned, so a stop string inside a thinking +block or a tool argument cannot end the turn. Each text delta is appended to a holdback, which is +searched for the earliest occurrence of any stop sequence (earliest start byte wins; ties break by +the client's order). Without a match, everything but the longest holdback suffix that is a *proper* +prefix of some stop sequence is emitted immediately — so at most `max_len - 1` bytes are ever +buffered and streaming is preserved, while a stop split across two deltas is still caught. Slicing +only happens on `char` boundaries, so a multi-byte code point is never split. + +**Termination.** On a match the machine emits the text before it, closes the open block, and sends +`message_delta` with `stop_reason: "stop_sequence"` and `stop_sequence: ""`, then +`message_stop`; `stopped` makes every later event — `response.completed` included — a no-op. Text +after the stop is neither streamed nor accumulated, so the non-streaming `final_json` +reconstruction is truncated at exactly the same point and reports the same `stop_reason` / +`stop_sequence`. A held-back prefix that never completes is ordinary output: `close_any` flushes it +as a normal `text_delta` before the block's `content_block_stop`, on every path that closes a text +block. + +**Abort.** Every transport but one drops the upstream the moment the stop fires: `sse_parse.rs` +drops the `reqwest` byte stream with the final chunk (streaming HTTP), and `ws_stream.rs` drops +the `CodexWsEvents` receiver (both websocket paths), which makes the codex_ws reader abandon the +turn and evict the socket — correct, since a half-consumed turn must not be pooled. The exception +is non-streaming HTTP: `json_response` reads the whole body with `upstream.text()` before the +machine sees an event, so it truncates at the match like every other path but cannot cut the +upstream short. That is the pre-existing shape of the non-streaming relay, not something the stop +emulation introduced. + +Each transport keys the abort on the stop sequence specifically rather than on "the machine is +stopped": on a *normally* completed turn the upstream is not mid-turn, and aborting it there would +evict a healthy pooled socket (websocket) or close a connection reqwest could otherwise return to +its idle pool (HTTP). So an ordinary terminal — `response.completed` / `response.done` / +`response.incomplete`, or a backend error event — leaves the upstream to finish; on streaming HTTP +the rest of the body goes to `spawn_terminal_drain`, a detached read bounded by +`TERMINAL_DRAIN_BUDGET`, so the connection still returns to the idle pool. The trailing bytes +translate to nothing, since `stopped` already makes every later event a no-op. + +**Usage caveat.** The stop makes every later event a no-op, so the upstream's `response.completed` +usage is never applied — on the aborting transports it never arrives, and on non-streaming HTTP it +is read but discarded. `usage_value`'s existing estimate substitution applies instead, so the turn +reports the local input estimate and `output_tokens: 0`: in `message_delta.usage` when streaming, +in the final JSON's `usage` when not. A non-streaming turn emits no `message_start` and normally +skips the estimate altogether, so `forward` widens its gate to cover a request carrying +`stop_sequences` — otherwise a stopped turn would report `input_tokens: 0` for a non-empty prompt. +The estimate stays subject to the same `count_tokens = "tiktoken"` opt-in as every other path. +Tokens the upstream generated between the match and the abort are still billed upstream — +negligible for the short completions stop sequences are used for. + ## 9. Test targets (M1) - `insta` snapshots: request translation for (plain text, multi-turn, tool_use+tool_result diff --git a/site/src/content/docs/ja/providers/anthropic.mdx b/site/src/content/docs/ja/providers/anthropic.mdx index b013a7d34..81ddd5e2e 100644 --- a/site/src/content/docs/ja/providers/anthropic.mdx +++ b/site/src/content/docs/ja/providers/anthropic.mdx @@ -156,9 +156,11 @@ classifier_model = "claude-sonnet-5" このキーは `kind = "anthropic"` のプロバイダーでのみ受け付けられ、**そのプロバイダー内でのモデルの 差し替え**であって、別のプロバイダーへのルートではありません。分類器リクエストは `stop_sequences` -を運びますが、shunt の Responses 変換はこのフィールドを落とします。Sonnet エイリアス経由で Codex -の上流に向けたところ、分類器は第 1 段階をリトライし、パーミッション確認のたびに 5〜7 秒余計に -かかりました。キーを設定しない場合(デフォルト)は、クライアントが要求したモデルのまま動きます。 +に依存します。shunt の Responses 変換がこのフィールドを落としていた頃は、Sonnet エイリアス経由で +Codex の上流に向けたところ、分類器は第 1 段階をリトライし、パーミッション確認のたびに 5〜7 秒余計に +かかりました。現在は shunt が [Responses 経路で `stop_sequences` をエミュレート](/ja/providers/openai/#ストップシーケンス) +するため、その失敗はもう当てはまりませんが、このキーは引き続きプロバイダー内での差し替えです。 +キーを設定しない場合(デフォルト)は、クライアントが要求したモデルのまま動きます。 ゲートウェイログインのデプロイメントでは、代わりに[マネージド設定](/ja/guides/gateway-login/#管理設定とモデルポリシー)で エイリアスをクライアントへ配信することもできます。 diff --git a/site/src/content/docs/ja/providers/openai.mdx b/site/src/content/docs/ja/providers/openai.mdx index 9ffd540ba..1cdc5f9ee 100644 --- a/site/src/content/docs/ja/providers/openai.mdx +++ b/site/src/content/docs/ja/providers/openai.mdx @@ -106,6 +106,31 @@ ChatGPT / Codex バックエンドです — そのためネイティブが自 `tool_search = true` が必要です。ここで `tool_search = false` を設定すると、代わりにテキストシムを強制します。 [ツール検索](/ja/guides/codex/#ツール検索)を参照してください。 +## ストップシーケンス + +Responses API には **`stop` パラメーターがありません** — Chat Completions にはありますが、Responses +にはありません — そのため、クライアントの Anthropic `stop_sequences` をアップストリームへ渡すことは +できません。代わりに shunt が Responses の変換内でゲートウェイ側からこれをエミュレートします。 +すべての Responses プロバイダーと、すべてのトランスポート(HTTP と WebSocket、ストリーミングと +非ストリーミング)が対象です。 + +- アシスタントのテキストは、いずれかのストップシーケンスが最初に現れた位置で打ち切られ、それ以降の + テキストがクライアントへ届くことはありません。 +- ターンは `stop_reason: "stop_sequence"` と、一致した文字列を示す `stop_sequence` で終了します。 + ネイティブの Anthropic アップストリームが報告するのとまったく同じ形です。 +- 一致した時点でアップストリームのリクエストを中断し、走らせたままにはしません — ストリーミング + HTTP と 2 つの WebSocket パスが対象です。非ストリーミング HTTP は例外で、レスポンスボディを + すべて読み終えてから変換するため、打ち切り自体は他のパスと同じですが、アップストリームは最後 + まで実行されます。一致から中断までの間にアップストリームが生成したトークンは、引き続き + アップストリーム側で課金されます。ストップシーケンスを使う典型的な短い応答では、無視できる量 + です。 + +注意点がひとつあります。ストップによって、アップストリームの `response.completed` の使用量が反映 +される前にターンが終わるため、shunt は自前のローカル入力推定値で代替し、`output_tokens: 0` を報告 +します — ストリーミングのターンでは `message_delta.usage` に、非ストリーミングのターンでは +レスポンスの `usage` に入ります。この推定値にはプロバイダーの `count_tokens = "tiktoken"` の +オプトインが必要で、有効にしていない場合、ストップしたターンは `input_tokens: 0` を報告します。 + ## 検証 ```bash diff --git a/site/src/content/docs/ja/reference/configuration.md b/site/src/content/docs/ja/reference/configuration.md index 06e82050e..f2964fea1 100644 --- a/site/src/content/docs/ja/reference/configuration.md +++ b/site/src/content/docs/ja/reference/configuration.md @@ -336,14 +336,14 @@ origin に関係なく、保持された各スロットはそのスロットが | キー | 値 | 意味 | | :-- | :-- | :-- | -| `kind` | `anthropic` \| `responses` \| `cursor` \| `gemini` \| `antigravity` \| `antigravity_cli` | 上流プロトコル / アダプター。`anthropic` = Messages API(パススルー、オプションで再キー付け)。`responses` = Anthropic Messages を OpenAI Responses API へ変換。`cursor` = ネイティブな Cursor ConnectRPC/protobuf AgentService アダプター。`gemini` = Anthropic Messages を Google Code Assist バックエンドの Gemini `generateContent`/`streamGenerateContent` へ変換。`antigravity` = Google Antigravity バックエンドに HTTP で接続。`gemini` と同じ Code Assist プロトコルを話しますが、Antigravity のサブスクリプショントークンで認証し、プロジェクト探索では `ideType: ANTIGRAVITY` として自身を識別します。`antigravity_cli` = **非推奨** — 上流を持たず、ローカルの Antigravity CLI バイナリ(`agy`)をサブプロセスとして実行。`agy` が自身のツール呼び出しを解決し、`tool_use` ブロックを返せないため、実際にツール呼び出しを要求するリクエスト(空でない `tools` 配列、または `any`・`tool` の `tool_choice`)は、テキストとして黙って応答するのではなく `400 invalid_request_error` で拒否されます。`tool_choice: none`(`tools` と併用していても)、ツールのない `tool_choice: auto`、空の `tools: []` はいずれもツール呼び出しを強制しないため受け付けられます。 | +| `kind` | `anthropic` \| `responses` \| `cursor` \| `gemini` \| `antigravity` \| `antigravity_cli` | 上流プロトコル / アダプター。`anthropic` = Messages API(パススルー、オプションで再キー付け)。`responses` = Anthropic Messages を OpenAI Responses API へ変換(Responses API には `stop` パラメーターがないため、`stop_sequences` は黙って捨てられるのではなく、変換内でゲートウェイ側からエミュレートされます)。`cursor` = ネイティブな Cursor ConnectRPC/protobuf AgentService アダプター。`gemini` = Anthropic Messages を Google Code Assist バックエンドの Gemini `generateContent`/`streamGenerateContent` へ変換。`antigravity` = Google Antigravity バックエンドに HTTP で接続。`gemini` と同じ Code Assist プロトコルを話しますが、Antigravity のサブスクリプショントークンで認証し、プロジェクト探索では `ideType: ANTIGRAVITY` として自身を識別します。`antigravity_cli` = **非推奨** — 上流を持たず、ローカルの Antigravity CLI バイナリ(`agy`)をサブプロセスとして実行。`agy` が自身のツール呼び出しを解決し、`tool_use` ブロックを返せないため、実際にツール呼び出しを要求するリクエスト(空でない `tools` 配列、または `any`・`tool` の `tool_choice`)は、テキストとして黙って応答するのではなく `400 invalid_request_error` で拒否されます。`tool_choice: none`(`tools` と併用していても)、ツールのない `tool_choice: auto`、空の `tools: []` はいずれもツール呼び出しを強制しないため受け付けられます。 | | `base_url` | URL | 上流のベース。shunt がエンドポイントパスを追加します。`kind = "cursor"` ではログイン/トークン更新用エンドポイントにのみ使われ、エージェント/推論ホストは選択しません。 | | `auth` | `passthrough` \| `api_key` \| `chatgpt_oauth` \| `claude_oauth` \| `xai_oauth` \| `cursor_oauth` \| `google_oauth` \| `antigravity_oauth` \| `none` | `passthrough` はクライアント自身の credential を転送。`api_key` は `api_key_env` からキーを注入。`chatgpt_oauth` は `~/.codex/auth.json` を再利用。`claude_oauth` は明示的な Anthropic アカウントから選択。`xai_oauth` は `shunt login xai` からの `~/.shunt/xai-auth.json` を再利用(HTTPS 上の x.ai/grok.com ホストへのみ送信)。`cursor_oauth` は `~/.shunt/cursor-auth.json`(`shunt login cursor`)を再利用。`google_oauth` は gemini CLI ログインの `~/.gemini/oauth_creds.json` を再利用し、`kind = "gemini"` でのみ有効。`antigravity_oauth` は `shunt login antigravity` からの `~/.shunt/antigravity-auth.json` を再利用し、`kind = "antigravity"` でのみ有効で、`google_oauth` とは**互換性がありません** — Antigravity は Gemini CLI のトークンには含まれない 2 つのスコープ(`cclog`、`experimentsandconfigs`)を要求します。`none` は認証すべき上流を持たないアダプター(`kind = "antigravity_cli"`)向けに、credential を一切送信しません。 | | `api_key_env` | 環境変数名 | `auth = "api_key"` のとき、キーを読み取る場所。この値自体も `${VAR}` / `${file:...}` で書けます([Secret 参照](#secret-参照)を参照)。 | | `api_key_header` | `bearer`(デフォルト) \| `x_api_key` | 注入されたキーを送るヘッダー。 | | `effort` | `low` … `max` | オプションのデフォルト reasoning エフォート(`responses` プロバイダー)。`kind = "antigravity"` にも適用され、サフィックスのない `gemini-*` の `upstream_model` にカタログの effort サフィックスとして付与されます。 | | `count_tokens` | `tiktoken`(デフォルト) \| `estimate` | `responses` および `cursor` provider: ローカルの tiktoken カウント vs. `501 not_supported` フォールバック([詳細](/ja/guides/effort-and-context/#トークンカウントcount_tokens))。 | -| `classifier_model` | モデル id | `anthropic` provider 専用。Claude Code のオートモード権限分類器リクエストが使う上流モデル。対象はリクエストの形だけで判定され、それ以外のリクエストはクライアントが要求したモデルのままです。**この provider 内での**差し替えであって、別の provider へのルートではありません — 分類器リクエストは `stop_sequences` を運びますが、Responses 変換はこのフィールドを落とすためです。デフォルトは未設定。[Anthropic → オートモードの分類器](/ja/providers/anthropic/#オートモードの分類器) を参照。 | +| `classifier_model` | モデル id | `anthropic` provider 専用。Claude Code のオートモード権限分類器リクエストが使う上流モデル。対象はリクエストの形だけで判定され、それ以外のリクエストはクライアントが要求したモデルのままです。**この provider 内での**差し替えであって、別の provider へのルートではありません — このキーは `anthropic` の上流でのみ受け付けられます。デフォルトは未設定。[Anthropic → オートモードの分類器](/ja/providers/anthropic/#オートモードの分類器) を参照。 | | `tool_search` | 未設定(「auto」、デフォルト) \| `true` \| `false` | gpt-5.4+ モデルかつフレーバーが xAI/Grok でない場合に、Claude Code のツール検索へネイティブなクライアント実行 `tool_search` プロトコルを使う。未設定時は、すでに動作確認済みのホスト — ChatGPT/Codex バックエンドと `api.openai.com` — でのみネイティブがデフォルトになり、LiteLLM・vLLM・OpenRouter・自前ホストのプロキシなど他のすべての OpenAI 互換エンドポイントはテキストベースのシムのまま。検証済みのカスタムエンドポイントをネイティブへオプトインするには `true`、常にシムを強制するには `false` を設定する。[Codex → ツール検索](/ja/guides/codex/#ネイティブプロトコル) を参照。 | 名前だけのエントリーは、`shunt login claude --name --mode oauth|import|setup-token` で作成した `~/.shunt/accounts/claude/.json` を読み取ります。対話型 CLI はこの 3 つの mode を提示し、リフレッシュ可能な OAuth を推奨します。`--long-lived` は `--mode setup-token` の deprecated alias です。`SHUNT_CLAUDE_ACCOUNTS_DIR` でストアディレクトリを上書きできます。リフレッシュ可能な OAuth/import ファイルは provider が refresh token をローテーションすると同じ場所に更新されるため、ファイルごとに稼働中の owner は 1 つだけにしてください。複数の shunt プロセスで共有したり、独立してコピーしたりしないでください。プロセスごとに個別にプロビジョニングするか、適切な場合は静的な setup token を使ってください。 diff --git a/site/src/content/docs/ko/providers/anthropic.mdx b/site/src/content/docs/ko/providers/anthropic.mdx index 8c603fe74..8bc741cb7 100644 --- a/site/src/content/docs/ko/providers/anthropic.mdx +++ b/site/src/content/docs/ko/providers/anthropic.mdx @@ -148,9 +148,11 @@ classifier_model = "claude-sonnet-5" ``` 이 키는 `kind = "anthropic"` 프로바이더에서만 받으며, **해당 프로바이더 안에서의 모델 리맵**이지 -다른 프로바이더로 가는 라우트가 아닙니다. 분류기 요청에는 `stop_sequences`가 실려 있는데 shunt의 -Responses 변환은 이 필드를 떨어뜨립니다: Sonnet 별칭을 통해 Codex 업스트림을 가리키게 했더니 -분류기가 첫 단계를 재시도했고 권한 확인마다 5~7초가 더 걸렸습니다. 키를 설정하지 않으면(기본값) +다른 프로바이더로 가는 라우트가 아닙니다. 분류기 요청은 `stop_sequences`에 의존합니다. shunt의 +Responses 변환이 이 필드를 떨어뜨리던 시절에는, Sonnet 별칭을 통해 Codex 업스트림을 가리키게 했더니 +분류기가 첫 단계를 재시도했고 권한 확인마다 5~7초가 더 걸렸습니다. 지금은 shunt가 +[Responses 경로에서 `stop_sequences`를 에뮬레이션](/ko/providers/openai/#정지-시퀀스)하므로 그 실패는 +더 이상 해당되지 않지만, 이 키는 그대로 프로바이더 내부 리맵입니다. 키를 설정하지 않으면(기본값) 분류기는 클라이언트가 요청한 모델을 그대로 씁니다. 게이트웨이 로그인 배포에서는 대신 [관리형 설정](/ko/guides/gateway-login/#관리형-설정과-모델-정책)으로 별칭을 클라이언트에 diff --git a/site/src/content/docs/ko/providers/openai.mdx b/site/src/content/docs/ko/providers/openai.mdx index 1f82f0eff..db0d6d04f 100644 --- a/site/src/content/docs/ko/providers/openai.mdx +++ b/site/src/content/docs/ko/providers/openai.mdx @@ -102,6 +102,29 @@ Responses 변환은 프로바이더별 또는 라우트별 추론 노력 다이 않으며 옵트인하려면 `tool_search = true`가 필요합니다. 여기서 `tool_search = false`로 설정하면 대신 텍스트 심을 강제할 수 있습니다. [도구 검색](/ko/guides/codex/#도구-검색)을 참고하세요. +## 정지 시퀀스 + +Responses API에는 **`stop` 파라미터가 없습니다** — Chat Completions에는 있지만 Responses에는 +없습니다 — 따라서 클라이언트의 Anthropic `stop_sequences`는 업스트림으로 전달할 수 없습니다. +대신 shunt가 Responses 변환 과정에서 게이트웨이 측으로 이를 흉내 냅니다. 모든 Responses +프로바이더와 모든 전송 방식(HTTP와 WebSocket, 스트리밍과 비스트리밍)에 적용됩니다. + +- 어시스턴트 텍스트는 정지 시퀀스가 처음 나타나는 지점에서 잘리며, 그 뒤의 텍스트는 클라이언트에 + 전달되지 않습니다. +- 턴은 `stop_reason: "stop_sequence"`와 일치한 문자열을 담은 `stop_sequence`로 종료됩니다. 네이티브 + Anthropic 업스트림이 보고하는 방식과 동일합니다. +- 일치 시점에 업스트림 요청을 중단하며, 그대로 두지 않습니다 — 스트리밍 HTTP와 두 WebSocket + 경로가 해당합니다. 비스트리밍 HTTP는 예외로, 응답 본문을 모두 읽은 뒤에야 변환하므로 자르는 + 동작은 다른 경로와 같지만 업스트림은 끝까지 진행됩니다. 일치 시점부터 중단까지 업스트림이 + 생성한 토큰은 여전히 업스트림에서 과금되지만, 정지 시퀀스를 주로 쓰는 짧은 응답에서는 무시할 + 만한 양입니다. + +한 가지 알아둘 점이 있습니다. 정지 때문에 업스트림의 `response.completed` 사용량이 반영되기 전에 +턴이 끝나므로, shunt는 자체 로컬 입력 추정치로 대체하고 `output_tokens: 0`을 보고합니다 — +스트리밍 턴에서는 `message_delta.usage`에, 비스트리밍 턴에서는 응답의 `usage`에 담깁니다. 이 +추정치에는 프로바이더의 `count_tokens = "tiktoken"` 옵트인이 필요하며, 이를 켜지 않으면 정지된 +턴은 `input_tokens: 0`을 보고합니다. + ## 검증 ```bash diff --git a/site/src/content/docs/ko/reference/configuration.md b/site/src/content/docs/ko/reference/configuration.md index 29bb419a5..5dc2e0e9c 100644 --- a/site/src/content/docs/ko/reference/configuration.md +++ b/site/src/content/docs/ko/reference/configuration.md @@ -355,7 +355,7 @@ origin과 무관하게, 유지된 각 슬롯은 그 슬롯이 실제로 담고 | 키 | 값 | 의미 | | :-- | :-- | :-- | -| `kind` | `anthropic` \| `responses` \| `cursor` \| `gemini` \| `antigravity` \| `antigravity_cli` | 업스트림 프로토콜 / 어댑터. `anthropic` = Messages API(패스스루, 선택적으로 키 재설정); `responses` = Anthropic Messages를 OpenAI Responses API로 변환; `cursor` = 네이티브 Cursor ConnectRPC/protobuf AgentService 어댑터; `gemini` = Anthropic Messages를 Google Code Assist 백엔드의 Gemini `generateContent`/`streamGenerateContent`로 변환; `antigravity` = Google Antigravity 백엔드에 HTTP로 접속하며, `gemini`와 동일한 Code Assist 프로토콜을 사용하되 Antigravity 구독 토큰으로 인증하고 프로젝트 디스커버리에서 `ideType: ANTIGRAVITY`로 자신을 식별; `antigravity_cli` = **더 이상 사용되지 않음** — 업스트림 없이 로컬 Antigravity CLI 바이너리(`agy`)를 서브프로세스로 실행. `agy`가 자체 도구 호출을 처리하며 `tool_use` 블록을 반환할 수 없기 때문에, 실제로 도구 호출을 요구하는 요청 — 비어 있지 않은 `tools` 배열 또는 `any`나 `tool` 값의 `tool_choice` — 은 텍스트로 조용히 응답하지 않고 `400 invalid_request_error`로 거부됩니다. `tool_choice: none`(`tools`와 함께 있어도), 도구가 없는 `tool_choice: auto`, 빈 `tools: []`는 도구 호출을 요구하지 않으므로 모두 허용됩니다. | +| `kind` | `anthropic` \| `responses` \| `cursor` \| `gemini` \| `antigravity` \| `antigravity_cli` | 업스트림 프로토콜 / 어댑터. `anthropic` = Messages API(패스스루, 선택적으로 키 재설정); `responses` = Anthropic Messages를 OpenAI Responses API로 변환(Responses API에는 `stop` 파라미터가 없으므로 `stop_sequences`는 조용히 버려지지 않고 변환 과정에서 게이트웨이 측으로 흉내 냅니다); `cursor` = 네이티브 Cursor ConnectRPC/protobuf AgentService 어댑터; `gemini` = Anthropic Messages를 Google Code Assist 백엔드의 Gemini `generateContent`/`streamGenerateContent`로 변환; `antigravity` = Google Antigravity 백엔드에 HTTP로 접속하며, `gemini`와 동일한 Code Assist 프로토콜을 사용하되 Antigravity 구독 토큰으로 인증하고 프로젝트 디스커버리에서 `ideType: ANTIGRAVITY`로 자신을 식별; `antigravity_cli` = **더 이상 사용되지 않음** — 업스트림 없이 로컬 Antigravity CLI 바이너리(`agy`)를 서브프로세스로 실행. `agy`가 자체 도구 호출을 처리하며 `tool_use` 블록을 반환할 수 없기 때문에, 실제로 도구 호출을 요구하는 요청 — 비어 있지 않은 `tools` 배열 또는 `any`나 `tool` 값의 `tool_choice` — 은 텍스트로 조용히 응답하지 않고 `400 invalid_request_error`로 거부됩니다. `tool_choice: none`(`tools`와 함께 있어도), 도구가 없는 `tool_choice: auto`, 빈 `tools: []`는 도구 호출을 요구하지 않으므로 모두 허용됩니다. | | `base_url` | URL | 업스트림 base; shunt가 엔드포인트 경로를 붙입니다. `kind = "cursor"`에서는 로그인/토큰 갱신 엔드포인트에만 사용되며 에이전트/추론 호스트를 선택하지 않습니다. | | `auth` | `passthrough` \| `api_key` \| `chatgpt_oauth` \| `claude_oauth` \| `xai_oauth` \| `cursor_oauth` \| `google_oauth` \| `antigravity_oauth` \| `none` | `passthrough`는 클라이언트 본인의 credential을 전달; `api_key`는 `api_key_env`의 키를 주입; `chatgpt_oauth`는 `~/.codex/auth.json`을 재사용; `claude_oauth`는 명시적 Anthropic 계정에서 선택; `xai_oauth`는 `shunt login xai`의 `~/.shunt/xai-auth.json`을 재사용(HTTPS를 통한 x.ai/grok.com 호스트에만 전송); `cursor_oauth`는 `~/.shunt/cursor-auth.json`을 재사용(`shunt login cursor`); `google_oauth`는 gemini CLI 로그인의 `~/.gemini/oauth_creds.json`을 재사용하며 `kind = "gemini"`에서만 유효; `antigravity_oauth`는 `shunt login antigravity`의 `~/.shunt/antigravity-auth.json`을 재사용하며 `kind = "antigravity"`에서만 유효하고, `google_oauth`와 **호환되지 않습니다** — Antigravity는 Gemini CLI 토큰에 없는 두 스코프(`cclog`, `experimentsandconfigs`)를 요청합니다; `none`은 인증할 업스트림이 없는 어댑터(`kind = "antigravity_cli"`)를 위해 크리덴셜을 전혀 보내지 않습니다. | | `api_key_env` | env 변수 이름 | `auth = "api_key"`일 때 키를 읽어오는 곳. 이 값 자신도 `${VAR}` / `${file:...}`로 쓸 수 있음([Secret 참조](#secret-참조) 참고). | @@ -363,7 +363,7 @@ origin과 무관하게, 유지된 각 슬롯은 그 슬롯이 실제로 담고 | `accounts` | 계정 테이블 배열 | Anthropic OAuth 계정 풀. `kind = "anthropic"`이고 `auth = "claude_oauth"`일 때만 유효; 아래 참고. | | `effort` | `low` … `max` | 선택적 기본 추론 노력(`responses` 프로바이더). `kind = "antigravity"`에도 적용되며, 접미사가 없는 `gemini-*` `upstream_model`에 카탈로그의 effort 접미사로 붙습니다. | | `count_tokens` | `tiktoken`(기본) \| `estimate` | `responses` 및 `cursor` provider: 로컬 tiktoken 카운트 대 `501 not_supported` fallback([상세](/ko/guides/effort-and-context/#토큰-카운팅-count_tokens)). | -| `classifier_model` | 모델 id | `anthropic` provider 전용. Claude Code 자동 모드 권한 분류기 요청이 사용할 업스트림 모델이며, 대상은 요청 모양만으로 식별합니다 — 나머지 요청은 모두 클라이언트가 요청한 모델을 그대로 씁니다. **이 provider 안에서의** 리맵이지 다른 provider로 가는 라우트가 아닙니다: 분류기 요청에는 `stop_sequences`가 실려 있고 Responses 변환은 이 필드를 떨어뜨리기 때문입니다. 기본값은 설정 안 함. [Anthropic → 자동 모드 분류기](/ko/providers/anthropic/#자동-모드-분류기) 참고. | +| `classifier_model` | 모델 id | `anthropic` provider 전용. Claude Code 자동 모드 권한 분류기 요청이 사용할 업스트림 모델이며, 대상은 요청 모양만으로 식별합니다 — 나머지 요청은 모두 클라이언트가 요청한 모델을 그대로 씁니다. **이 provider 안에서의** 리맵이지 다른 provider로 가는 라우트가 아닙니다 — 이 키는 `anthropic` 업스트림에서만 받습니다. 기본값은 설정 안 함. [Anthropic → 자동 모드 분류기](/ko/providers/anthropic/#자동-모드-분류기) 참고. | | `tool_search` | 미설정("auto", 기본) \| `true` \| `false` | gpt-5.4+ 모델이면서 계열이 xAI/Grok이 아닐 때 Claude Code의 도구 검색에 네이티브 클라이언트 실행 `tool_search` 프로토콜을 사용합니다. 미설정 시에는 이미 검증된 호스트 — ChatGPT/Codex 백엔드와 `api.openai.com` — 에서만 기본으로 네이티브를 사용하고, LiteLLM·vLLM·OpenRouter·자체 호스팅 프록시 등 그 외 모든 OpenAI 호환 엔드포인트는 텍스트 shim을 유지합니다. 검증된 커스텀 엔드포인트를 네이티브에 옵트인하려면 `true`로, shim을 항상 강제하려면 `false`로 설정하세요. [Codex → 도구 검색](/ko/guides/codex/#네이티브-프로토콜)을 참고하세요. | 이름만 있는 항목은 `shunt login claude --name --mode `(``는 `oauth`, `import`, `setup-token` 중 하나)로 만든 `~/.shunt/accounts/claude/.json`을 읽습니다. 대화형 CLI는 이 세 mode를 묻고 갱신 가능한 OAuth를 권장합니다. `--long-lived`는 `--mode setup-token`의 deprecated alias입니다. `SHUNT_CLAUDE_ACCOUNTS_DIR`로 스토어 디렉터리를 재정의할 수 있습니다. `[[providers..accounts]]`에 명시적으로 나열된 계정 목록이 비어 있으면 스토어 디렉터리의 유효한 계정 파일을 모두 스캔합니다. 갱신 가능한 OAuth/import 파일은 provider가 refresh token을 회전할 때 제자리에서 갱신되므로 파일마다 활성 owner가 하나만 있어야 합니다. 실행 중인 여러 shunt 프로세스에서 파일을 공유하거나 독립적으로 복사하지 마세요. 프로세스마다 별도로 프로비저닝하거나, 적절한 경우 정적 setup token을 사용하세요. diff --git a/site/src/content/docs/providers/anthropic.mdx b/site/src/content/docs/providers/anthropic.mdx index 6e71839c1..150c8f0df 100644 --- a/site/src/content/docs/providers/anthropic.mdx +++ b/site/src/content/docs/providers/anthropic.mdx @@ -152,10 +152,12 @@ classifier_model = "claude-sonnet-5" ``` The key is accepted on `kind = "anthropic"` providers only, and it is a **model remap inside that -provider**, never a route to a different one. The classifier request carries `stop_sequences`, which -shunt's Responses translation drops: pointed at a Codex upstream through the Sonnet alias, the -classifier retried its first stage and every permission check took 5–7 s longer. An unset key (the -default) leaves the classifier on whatever model the client asked for. +provider**, never a route to a different one. The classifier request depends on `stop_sequences`: +while shunt's Responses translation still dropped that field, pointing the classifier at a Codex +upstream through the Sonnet alias made it retry its first stage, and every permission check took +5–7 s longer. shunt now [emulates `stop_sequences` on the Responses path](/providers/openai/#stop-sequences), +so that particular failure no longer applies — the key stays an in-provider remap regardless. An +unset key (the default) leaves the classifier on whatever model the client asked for. On a gateway-login deployment you can push the alias to the client instead, through [managed settings](/guides/gateway-login/#managed-settings-and-model-policy): diff --git a/site/src/content/docs/providers/openai.mdx b/site/src/content/docs/providers/openai.mdx index df79719c3..f4bf43999 100644 --- a/site/src/content/docs/providers/openai.mdx +++ b/site/src/content/docs/providers/openai.mdx @@ -104,6 +104,29 @@ a custom OpenAI-compatible endpoint (LiteLLM, vLLM, OpenRouter, self-hosted) isn same way and needs `tool_search = true` to opt in. Set `tool_search = false` here to force the text shim instead. See [Tool search](/guides/codex/#tool-search). +## Stop sequences + +The Responses API has **no `stop` parameter** — Chat Completions has one, Responses does not — so +a client's Anthropic `stop_sequences` cannot be forwarded upstream. shunt emulates them +gateway-side in the Responses translation instead, on every Responses provider and every +transport (HTTP and WebSocket, streaming and non-streaming): + +- assistant text is truncated at the first occurrence of any stop sequence, and the text after it + never reaches the client; +- the turn ends with `stop_reason: "stop_sequence"` and `stop_sequence` naming the match, exactly + as a native Anthropic upstream would report it; +- the upstream request is aborted at the match rather than left running — on streaming HTTP and on + both WebSocket paths. Non-streaming HTTP is the exception: it reads the whole response body + before translating any of it, so it truncates like the others but lets the upstream finish. + Tokens the upstream generated between the match and the abort are still billed by the upstream — + negligible for the short completions stop sequences are typically used for. + +One consequence to be aware of: the stop ends the turn before the upstream's `response.completed` +usage can be applied, so shunt substitutes its own local input estimate and reports +`output_tokens: 0` — in `message_delta.usage` on a streaming turn, in the response's `usage` on a +non-streaming one. The estimate needs the provider's `count_tokens = "tiktoken"` opt-in; without +it, a stopped turn reports `input_tokens: 0`. + ## Verify ```bash diff --git a/site/src/content/docs/reference/configuration.md b/site/src/content/docs/reference/configuration.md index 1befb0856..dc5cb549f 100644 --- a/site/src/content/docs/reference/configuration.md +++ b/site/src/content/docs/reference/configuration.md @@ -470,7 +470,7 @@ Each provider is a table under a name of your choosing. Built-ins (`anthropic`, | Key | Values | Meaning | | :-- | :-- | :-- | -| `kind` | `anthropic` \| `responses` \| `cursor` \| `gemini` \| `antigravity` \| `antigravity_cli` | Upstream protocol / adapter. `anthropic` = Messages API (passed through, optionally re-keyed); `responses` = Anthropic Messages translated to the OpenAI Responses API; `cursor` = the native Cursor ConnectRPC/protobuf AgentService adapter; `gemini` = Anthropic Messages translated to Gemini `generateContent`/`streamGenerateContent` on the Google Code Assist backend; `antigravity` = the Google Antigravity backend over HTTP, which speaks the same Code Assist protocol as `gemini` but authenticates with an Antigravity subscription token and identifies itself as `ideType: ANTIGRAVITY` during project discovery; `antigravity_cli` = **deprecated** — no upstream at all, running the local Antigravity CLI binary (`agy`) as a subprocess. Because `agy` resolves its own tool calls and no `tool_use` block can ever be returned, a request that asks for one — a non-empty `tools` array, or a `tool_choice` of `any` or `tool` — is refused with a `400 invalid_request_error` rather than silently answered as text. `tool_choice: none` (even alongside `tools`), `tool_choice: auto` with no tools, and an empty `tools: []` all oblige nothing and are accepted. | +| `kind` | `anthropic` \| `responses` \| `cursor` \| `gemini` \| `antigravity` \| `antigravity_cli` | Upstream protocol / adapter. `anthropic` = Messages API (passed through, optionally re-keyed); `responses` = Anthropic Messages translated to the OpenAI Responses API (the Responses API has no `stop` parameter, so `stop_sequences` is emulated gateway-side in the translation rather than silently dropped); `cursor` = the native Cursor ConnectRPC/protobuf AgentService adapter; `gemini` = Anthropic Messages translated to Gemini `generateContent`/`streamGenerateContent` on the Google Code Assist backend; `antigravity` = the Google Antigravity backend over HTTP, which speaks the same Code Assist protocol as `gemini` but authenticates with an Antigravity subscription token and identifies itself as `ideType: ANTIGRAVITY` during project discovery; `antigravity_cli` = **deprecated** — no upstream at all, running the local Antigravity CLI binary (`agy`) as a subprocess. Because `agy` resolves its own tool calls and no `tool_use` block can ever be returned, a request that asks for one — a non-empty `tools` array, or a `tool_choice` of `any` or `tool` — is refused with a `400 invalid_request_error` rather than silently answered as text. `tool_choice: none` (even alongside `tools`), `tool_choice: auto` with no tools, and an empty `tools: []` all oblige nothing and are accepted. | | `base_url` | URL | Upstream base; shunt appends the endpoint path. For `kind = "cursor"`, this is the login/token-refresh surface only; it does not select the agent/inference host. For `kind = "antigravity"`, credential-path project discovery (`loadCodeAssist`) also addresses this host when a stored credential has no cached project ID — not just inference — as does first-time onboarding (`onboardUser`), except on the production default host, which is onboarded through its own `daily-` control-plane host. When several Antigravity upstreams are configured, [`shunt login antigravity`](/reference/cli/#shunt-login-antigravity) runs that discovery against the one your routes select. | | `auth` | `passthrough` \| `api_key` \| `chatgpt_oauth` \| `claude_oauth` \| `kimi_oauth` \| `xai_oauth` \| `cursor_oauth` \| `google_oauth` \| `antigravity_oauth` \| `none` | `passthrough` forwards the client's own credential; `api_key` injects a key from `api_key_env`; `chatgpt_oauth` reuses `~/.codex/auth.json`; `claude_oauth` selects from explicit Anthropic accounts; `kimi_oauth` selects from explicit Kimi Code accounts (`shunt login kimi`), valid only with `kind = "anthropic"` and a `kimi.com` `base_url`; `xai_oauth` reuses `~/.shunt/xai-auth.json` from `shunt login xai` (only sent to x.ai/grok.com hosts over HTTPS); `cursor_oauth` reuses `~/.shunt/cursor-auth.json` (`shunt login cursor`); `google_oauth` reuses the gemini CLI login in `~/.gemini/oauth_creds.json` and is valid only with `kind = "gemini"`; `antigravity_oauth` reuses `~/.shunt/antigravity-auth.json` (`shunt login antigravity`), is valid only with `kind = "antigravity"`, and is **not** interchangeable with `google_oauth` — Antigravity requests two scopes (`cclog`, `experimentsandconfigs`) a Gemini CLI token never carries; `none` sends no credential at all, for adapters with no upstream to authenticate against (`kind = "antigravity_cli"`). | | `api_key_env` | env var name | Where the key is read from, when `auth = "api_key"`. Its own value can also be written as `${VAR}` / `${file:...}` (see [Secret references](#secret-references)). | @@ -479,7 +479,7 @@ Each provider is a table under a name of your choosing. Built-ins (`anthropic`, | `effort` | `low` … `max` | Optional default reasoning effort (`responses` providers). Also applies to `kind = "antigravity"`, where it is appended to a bare `gemini-*` `upstream_model` as the catalog's effort suffix. | | `service_tier` | `fast` \| `priority` \| `flex` \| `default` | Optional default Codex "Fast" mode opt-in (`responses` providers) — sent as the Responses API `service_tier` field. `fast` normalizes to `priority`; `default` is a client-only sentinel that is never sent on the wire. Off by default. Withheld for the `xai`/`grok` flavors even when configured (xAI 400s on it). A route-level `service_tier` (including an explicit `default`) overrides this value — see below. See [Codex → Fast mode](/guides/codex/#fast-mode). | | `count_tokens` | `tiktoken` (default) \| `estimate` | `responses` and `cursor` providers: local tiktoken count vs. `501 not_supported` fallback ([details](/guides/effort-and-context/#token-counting-count_tokens)). | -| `classifier_model` | model id | `anthropic` providers only. Upstream model for Claude Code's auto-mode permission classifier request, identified by its request shape alone — every other request keeps the model it asked for. A remap **within this provider**, never a route to another one: the classifier carries `stop_sequences`, which the Responses translation drops. Unset by default. See [Anthropic → Auto-mode classifier](/providers/anthropic/#auto-mode-classifier). | +| `classifier_model` | model id | `anthropic` providers only. Upstream model for Claude Code's auto-mode permission classifier request, identified by its request shape alone — every other request keeps the model it asked for. A remap **within this provider**, never a route to another one — the key is accepted on `anthropic` upstreams only. Unset by default. See [Anthropic → Auto-mode classifier](/providers/anthropic/#auto-mode-classifier). | | `websocket` | `true` \| `false` (default) | Opt in to the Codex Responses WebSocket v2 transport (ChatGPT/Codex backend only; falls back to HTTP on any transport failure before the first event reaches the client, so it can never do worse than plain HTTP). | | `tool_search` | unset ("auto", default) \| `true` \| `false` | Use the native client-executed `tool_search` protocol for Claude Code's tool search on a GPT-5.4+ model, gated on flavor (non-xAI/Grok). Unset defaults to native only for known-good hosts — the ChatGPT/Codex backend and `api.openai.com` — and the text shim everywhere else, including custom OpenAI-compatible endpoints (LiteLLM, vLLM, OpenRouter, self-hosted). Set `true` to opt a verified custom endpoint into native, or `false` to always force the shim. See [Codex → Tool search](/guides/codex/#native-protocol). | | `request_compression` | `true` (default) \| `false` | zstd-compress the Responses **request** body (`content-encoding: zstd`, level 3), matching what the Codex CLI sends to the same backend. Effective only on the ChatGPT/Codex flavor (`auth = "chatgpt_oauth"`) — no other Responses upstream is verified to accept a compressed request body, so the flag is inert there. Set `false` to send plain JSON, e.g. behind a middlebox that mishandles compressed request bodies. | diff --git a/site/src/content/docs/zh-cn/providers/anthropic.mdx b/site/src/content/docs/zh-cn/providers/anthropic.mdx index f4ad5bdd1..218ca3928 100644 --- a/site/src/content/docs/zh-cn/providers/anthropic.mdx +++ b/site/src/content/docs/zh-cn/providers/anthropic.mdx @@ -146,9 +146,10 @@ classifier_model = "claude-sonnet-5" ``` 该键仅在 `kind = "anthropic"` 的提供方上被接受,且它是**该提供方内部的模型重映射**,而不是通往 -另一个提供方的路由。分类器请求会携带 `stop_sequences`,而 shunt 的 Responses 转换会丢掉这个字段: -经由 Sonnet 别名指向 Codex 上游后,分类器重试了它的第一阶段,每次权限检查多花 5–7 秒。不设置该键 -(默认)时,分类器仍使用客户端请求的模型。 +另一个提供方的路由。分类器请求依赖 `stop_sequences`。在 shunt 的 Responses 转换还会丢掉这个字段时, +经由 Sonnet 别名指向 Codex 上游后,分类器重试了它的第一阶段,每次权限检查多花 5–7 秒。现在 shunt 会 +[在 Responses 路径上模拟 `stop_sequences`](/zh-cn/providers/openai/#停止序列),因此该失败不再适用, +但该键仍然是提供方内部的重映射。不设置该键(默认)时,分类器仍使用客户端请求的模型。 在网关登录部署中,你也可以改为通过[托管设置](/zh-cn/guides/gateway-login/#托管设置与模型策略)把别名下发给客户端: diff --git a/site/src/content/docs/zh-cn/providers/openai.mdx b/site/src/content/docs/zh-cn/providers/openai.mdx index 55a38718d..7c243c11d 100644 --- a/site/src/content/docs/zh-cn/providers/openai.mdx +++ b/site/src/content/docs/zh-cn/providers/openai.mdx @@ -104,6 +104,25 @@ Responses 转换支持一个按提供方或按路由的推理力度旋钮 需要 `tool_search = true` 来选择加入。在这里设置 `tool_search = false` 可强制改用文本 垫片。见 [工具搜索](/zh-cn/guides/codex/#工具搜索)。 +## 停止序列 + +Responses API **没有 `stop` 参数** —— Chat Completions 有,Responses 没有 —— 所以客户端的 Anthropic +`stop_sequences` 无法转发给上游。shunt 改为在 Responses 转换中于网关侧模拟它,适用于所有 Responses +提供方和所有传输方式(HTTP 与 WebSocket、流式与非流式): + +- 助手文本会在任一停止序列首次出现处被截断,其后的文本不会到达客户端; +- 该轮以 `stop_reason: "stop_sequence"` 结束,`stop_sequence` 写明命中的那个字符串,与原生 Anthropic + 上游的上报方式完全一致; +- 命中时会中止上游请求,而不是任其继续 —— 这适用于流式 HTTP 和两条 WebSocket 路径。非流式 HTTP + 是例外:它会先把整个响应体读完再做转换,因此截断行为与其他路径一致,但上游会一直跑到结束。 + 上游在命中到中止之间生成的 token 仍由上游计费 —— 对停止序列通常服务的那种短回复来说可以忽略 + 不计。 + +有一点需要留意:停止会让该轮在上游的 `response.completed` 用量被采纳之前就结束,因此 shunt 改用 +自己的本地输入估算值,并报告 `output_tokens: 0` —— 流式轮次放在 `message_delta.usage` 里,非流式 +轮次放在响应的 `usage` 里。该估算值需要提供方开启 `count_tokens = "tiktoken"`;若未开启,被停止 +的轮次会报告 `input_tokens: 0`。 + ## 校验 ```bash diff --git a/site/src/content/docs/zh-cn/reference/configuration.md b/site/src/content/docs/zh-cn/reference/configuration.md index f8bcb7271..1d9991789 100644 --- a/site/src/content/docs/zh-cn/reference/configuration.md +++ b/site/src/content/docs/zh-cn/reference/configuration.md @@ -336,14 +336,14 @@ codex-fallback = "gpt-5.2" | 键 | 取值 | 含义 | | :-- | :-- | :-- | -| `kind` | `anthropic` \| `responses` \| `cursor` \| `gemini` \| `antigravity` \| `antigravity_cli` | 上游协议 / 适配器。`anthropic` = Messages API(透传,可选择重新设置密钥);`responses` = Anthropic Messages 转换为 OpenAI Responses API;`cursor` = 原生 Cursor ConnectRPC/protobuf AgentService 适配器;`gemini` = Anthropic Messages 转换为 Google Code Assist 后端的 Gemini `generateContent`/`streamGenerateContent`;`antigravity` = 通过 HTTP 连接 Google Antigravity 后端,与 `gemini` 使用相同的 Code Assist 协议,但以 Antigravity 订阅令牌认证,并在项目发现时以 `ideType: ANTIGRAVITY` 标识自身;`antigravity_cli` = **已弃用** —— 没有任何上游,以子进程方式运行本地 Antigravity CLI 二进制(`agy`)。由于 `agy` 自行解析工具调用,永远不会返回 `tool_use` 块,因此真正要求工具调用的请求——非空的 `tools` 数组,或值为 `any`、`tool` 的 `tool_choice`——会被 `400 invalid_request_error` 拒绝,而不是静默地以文本形式作答。`tool_choice: none`(即使与 `tools` 同时出现)、没有工具时的 `tool_choice: auto` 以及空的 `tools: []` 都不会强制工具调用,因此均被接受。 | +| `kind` | `anthropic` \| `responses` \| `cursor` \| `gemini` \| `antigravity` \| `antigravity_cli` | 上游协议 / 适配器。`anthropic` = Messages API(透传,可选择重新设置密钥);`responses` = Anthropic Messages 转换为 OpenAI Responses API(Responses API 没有 `stop` 参数,因此 `stop_sequences` 不会被静默丢弃,而是在转换中于网关侧模拟);`cursor` = 原生 Cursor ConnectRPC/protobuf AgentService 适配器;`gemini` = Anthropic Messages 转换为 Google Code Assist 后端的 Gemini `generateContent`/`streamGenerateContent`;`antigravity` = 通过 HTTP 连接 Google Antigravity 后端,与 `gemini` 使用相同的 Code Assist 协议,但以 Antigravity 订阅令牌认证,并在项目发现时以 `ideType: ANTIGRAVITY` 标识自身;`antigravity_cli` = **已弃用** —— 没有任何上游,以子进程方式运行本地 Antigravity CLI 二进制(`agy`)。由于 `agy` 自行解析工具调用,永远不会返回 `tool_use` 块,因此真正要求工具调用的请求——非空的 `tools` 数组,或值为 `any`、`tool` 的 `tool_choice`——会被 `400 invalid_request_error` 拒绝,而不是静默地以文本形式作答。`tool_choice: none`(即使与 `tools` 同时出现)、没有工具时的 `tool_choice: auto` 以及空的 `tools: []` 都不会强制工具调用,因此均被接受。 | | `base_url` | URL | 上游 base;shunt 追加端点路径。对于 `kind = "cursor"`,它仅用于登录/令牌刷新接口,不会选择代理/推理主机。 | | `auth` | `passthrough` \| `api_key` \| `chatgpt_oauth` \| `claude_oauth` \| `xai_oauth` \| `cursor_oauth` \| `google_oauth` \| `antigravity_oauth` \| `none` | `passthrough` 转发客户端自己的 credential;`api_key` 从 `api_key_env` 注入一个密钥;`chatgpt_oauth` 复用 `~/.codex/auth.json`;`claude_oauth` 从显式 Anthropic 账户中选择;`xai_oauth` 复用来自 `shunt login xai` 的 `~/.shunt/xai-auth.json`(仅经由 HTTPS 发送到 x.ai/grok.com 主机);`cursor_oauth` 复用 `~/.shunt/cursor-auth.json`(`shunt login cursor`);`google_oauth` 复用 gemini CLI 登录的 `~/.gemini/oauth_creds.json`,仅在 `kind = "gemini"` 下有效;`antigravity_oauth` 复用来自 `shunt login antigravity` 的 `~/.shunt/antigravity-auth.json`,仅在 `kind = "antigravity"` 下有效,且与 `google_oauth` **不可互换** —— Antigravity 会请求 Gemini CLI 令牌所没有的两个 scope(`cclog`、`experimentsandconfigs`);`none` 完全不发送 credential,用于没有上游需要认证的适配器(`kind = "antigravity_cli"`)。 | | `api_key_env` | 环境变量名 | 当 `auth = "api_key"` 时,从何处读取密钥。该值自身也可以写成 `${VAR}` / `${file:...}`(见 [Secret 引用](#secret-引用))。 | | `api_key_header` | `bearer`(默认) \| `x_api_key` | 注入的密钥在哪个头部中发送。 | | `effort` | `low` … `max` | 可选的默认推理力度(`responses` 提供方)。也适用于 `kind = "antigravity"`,会作为目录的 effort 后缀追加到不带后缀的 `gemini-*` `upstream_model` 上。 | | `count_tokens` | `tiktoken`(默认) \| `estimate` | `responses` 与 `cursor` provider:本地 tiktoken 计数 vs. `501 not_supported` 回退([详情](/zh-cn/guides/effort-and-context/#token-计数count_tokens))。 | -| `classifier_model` | 模型 id | 仅 `anthropic` provider。Claude Code 自动模式权限分类器请求所用的上游模型,仅凭请求形态识别 —— 其余请求一律保持客户端请求的模型。它是**该 provider 内部的**重映射,而不是通往另一个 provider 的路由:分类器请求携带 `stop_sequences`,而 Responses 转换会丢掉该字段。默认不设置。参见 [Anthropic → 自动模式分类器](/zh-cn/providers/anthropic/#自动模式分类器)。 | +| `classifier_model` | 模型 id | 仅 `anthropic` provider。Claude Code 自动模式权限分类器请求所用的上游模型,仅凭请求形态识别 —— 其余请求一律保持客户端请求的模型。它是**该 provider 内部的**重映射,而不是通往另一个 provider 的路由 —— 该键仅在 `anthropic` 上游上被接受。默认不设置。参见 [Anthropic → 自动模式分类器](/zh-cn/providers/anthropic/#自动模式分类器)。 | | `tool_search` | 未设置("auto",默认) \| `true` \| `false` | 在模型为 GPT-5.4+ 且风格不是 xAI/Grok 时,为 Claude Code 的工具搜索使用原生的客户端执行 `tool_search` 协议。未设置时仅对已验证支持的主机 —— ChatGPT/Codex 后端与 `api.openai.com` —— 默认使用原生协议,LiteLLM、vLLM、OpenRouter、自托管代理等其他所有 OpenAI 兼容端点都保留文本 shim。设为 `true` 可让已验证的自定义端点选择加入原生协议;设为 `false` 则始终强制使用 shim。见 [Codex → 工具搜索](/zh-cn/guides/codex/#原生协议)。 | 只带名称的条目读取 `~/.shunt/accounts/claude/.json`,该文件由 `shunt login claude --name --mode oauth|import|setup-token` 创建。交互式 CLI 会提示选择这三种 mode,并推荐可刷新的 OAuth。`--long-lived` 保留为 `--mode setup-token` 的 deprecated alias。`SHUNT_CLAUDE_ACCOUNTS_DIR` 可覆盖存储目录。可刷新的 OAuth/import 文件会在 provider 轮换 refresh token 时原地更新,因此每个文件只能有一个正在运行的 owner。不要在多个 shunt 进程之间共享或独立复制该文件。请为每个进程分别预配,或在适合时使用静态 setup token。 diff --git a/src/adapters/responses/context.rs b/src/adapters/responses/context.rs index baad3ecd0..bac6f4a75 100644 --- a/src/adapters/responses/context.rs +++ b/src/adapters/responses/context.rs @@ -28,6 +28,9 @@ pub(super) struct RelayOptions { pub model: String, pub thinking_enabled: bool, pub tool_search_native: bool, + /// The client's Anthropic `stop_sequences`, emulated gateway-side because + /// the Responses API has no `stop` parameter (issue #605). Usually empty. + pub stop_sequences: Vec, } impl RelayOptions { @@ -37,6 +40,7 @@ impl RelayOptions { /// no relay call site touches the options after building the machine). pub(super) fn machine(self) -> AnthropicSseMachine { AnthropicSseMachine::new(self.model, self.thinking_enabled, self.tool_search_native) + .with_stop_sequences(self.stop_sequences) } } @@ -44,7 +48,7 @@ impl RelayOptions { /// `forward` and threaded through each transport. `model` is intentionally /// absent — it is taken from the [`Route`] at relay time via /// [`TurnOptions::relay`], keeping these flags transport-agnostic. -#[derive(Debug, Clone, Copy)] +#[derive(Debug, Clone)] pub(super) struct TurnOptions { /// The client asked for a streaming (SSE) response. pub client_wants_stream: bool, @@ -52,6 +56,10 @@ pub(super) struct TurnOptions { pub thinking_enabled: bool, /// Native client-executed `tool_search` is enabled for this provider/model. pub tool_search_native: bool, + /// The client's Anthropic `stop_sequences` (issue #605), in request order. + /// Never forwarded upstream — the Responses API has no `stop` parameter — + /// but emulated by the SSE translation. + pub stop_sequences: Vec, } impl TurnOptions { @@ -62,6 +70,7 @@ impl TurnOptions { model: route.model.clone(), thinking_enabled: self.thinking_enabled, tool_search_native: self.tool_search_native, + stop_sequences: self.stop_sequences.clone(), } } } diff --git a/src/adapters/responses/early_stream.rs b/src/adapters/responses/early_stream.rs index d4d8883bd..53295d4a4 100644 --- a/src/adapters/responses/early_stream.rs +++ b/src/adapters/responses/early_stream.rs @@ -515,6 +515,7 @@ pub(super) fn relay_opts() -> super::context::RelayOptions { model: "gpt-5.2-codex".to_string(), thinking_enabled: false, tool_search_native: false, + stop_sequences: Vec::new(), } } diff --git a/src/adapters/responses/early_stream_tests.rs b/src/adapters/responses/early_stream_tests.rs index a8172774d..f89d8ae9f 100644 --- a/src/adapters/responses/early_stream_tests.rs +++ b/src/adapters/responses/early_stream_tests.rs @@ -1017,6 +1017,7 @@ async fn forward_http_commits_before_resolving_the_credential() { client_wants_stream: true, thinking_enabled: false, tool_search_native: false, + stop_sequences: Vec::new(), }, codex_quota_account: None, estimate_input: None, diff --git a/src/adapters/responses/http.rs b/src/adapters/responses/http.rs index 8167a11f8..978a525d5 100644 --- a/src/adapters/responses/http.rs +++ b/src/adapters/responses/http.rs @@ -19,8 +19,8 @@ use crate::{ use super::body::{prepare_body, PreparedBody}; use super::context::{CredentialSource, ForwardOptions, RelayOptions}; use super::early_stream::{ - early_streaming_response, estimated_machine_factory, http_events_stream, parsed_events, - translated_stream, HttpSendContext, + bounded_input_estimate, early_streaming_response, estimated_machine_factory, + http_events_stream, parsed_events, translated_stream, HttpSendContext, }; use super::error::{backend_error, mapped_upstream_error, own_error, transport_error}; use super::request::request_builder; @@ -128,6 +128,13 @@ pub(super) async fn forward_http( let codex_quota_account = codex_quota_account.or_else(|| super::codex_quota_account(&credential)); let body = prepare_body(state, route, upstream_body.as_ref()).await; + // Spawned before the send, like the pool loop's own estimate handle, so the + // CPU-bound tiktoken encode overlaps this request's connect/RTT instead of + // landing on the response's critical path. Only a non-streaming turn + // carrying `stop_sequences` reaches here with `Some` (see `forward`'s gate). + let estimate_handle = estimate_input.map(|request| { + tokio::task::spawn_blocking(move || crate::count_tokens::count_input_tokens_value(&request)) + }); let upstream = crate::retry::send_with_retry_with_safety( policy, &route.provider, @@ -147,11 +154,18 @@ pub(super) async fn forward_http( if !status.is_success() { return Err(mapped_upstream_error(status, upstream, auth).await); } + // Bounded like every other path so a saturated blocking pool cannot stall + // the response (see `bounded_input_estimate`); by now the encode has had the + // whole upstream round-trip to finish, so this normally resolves instantly. + let input_tokens_estimate = match estimate_handle { + Some(handle) => bounded_input_estimate(handle, std::time::Duration::from_secs(1)).await, + None => 0, + }; // Thread the real response status: `json_response` returns a `502` when // a backend error event surfaced via `backend_error` (issue #113), so // the proxy's access log (`upstream_status`) and `record_proxied_request` // metrics reflect the failure instead of a hardcoded `200`. - let response = json_response(upstream, turn.relay(route)).await?; + let response = json_response(upstream, turn.relay(route), input_tokens_estimate).await?; Ok((response.status(), response)) } @@ -184,15 +198,23 @@ pub(super) fn stream_response( /// a backend failure for a truncated-but-successful result (issue #113). This /// mirrors the streaming path, which emits the same error inline as an SSE /// `error` event. +/// +/// `input_tokens_estimate` seeds the same local prompt count the streaming path +/// puts in `message_start`. It matters here only for an emulated stop sequence: +/// the stop makes the upstream's `response.completed` usage a no-op, and +/// `final_json` falls back to the estimate when no usage was observed, so +/// without it a stopped turn would report `input_tokens: 0` (issue #605). On +/// every other turn the upstream's real usage arrives and overrides it. pub(super) async fn json_response( upstream: reqwest::Response, relay: RelayOptions, + input_tokens_estimate: u64, ) -> Result { let body = upstream .text() .await .map_err(|error| own_error(format!("failed to read Responses body: {error}")))?; - let mut machine = relay.machine(); + let mut machine = relay.machine().with_input_estimate(input_tokens_estimate); for event in parse_sse_events(&body) { let _ = machine.apply(event); } @@ -251,7 +273,7 @@ mod tests { "data: {\"type\":\"response.failed\",\"response\":{\"error\":{\"code\":\"server_error\",\"message\":\"Upstream failed\"}}}\n\n", ); let upstream = upstream_response(200, sse).await; - let error = json_response(upstream, relay_opts()) + let error = json_response(upstream, relay_opts(), 0) .await .expect_err("backend error event should stop failover"); @@ -276,7 +298,7 @@ mod tests { "data: {\"type\":\"response.failed\",\"response\":{\"error\":{\"code\":\"rate_limit_exceeded\",\"message\":\"Rate limit reached\"}}}\n\n", ); let upstream = upstream_response(200, sse).await; - let error = json_response(upstream, relay_opts()) + let error = json_response(upstream, relay_opts(), 0) .await .expect_err("in-stream rate limit is an error"); @@ -304,7 +326,7 @@ mod tests { "data: {\"response\":{\"usage\":{\"input_tokens\":3,\"output_tokens\":1}}}\n\n", ); let upstream = upstream_response(200, sse).await; - let response = json_response(upstream, relay_opts()) + let response = json_response(upstream, relay_opts(), 0) .await .expect("json_response builds a response"); @@ -314,6 +336,41 @@ mod tests { assert_eq!(body["content"][0]["text"], "hello"); } + /// A non-streaming turn cut short by an emulated stop sequence reports the + /// seeded local input estimate. The stop makes the upstream's own + /// `response.completed` usage a no-op, so without the seed this turn would + /// serialize `input_tokens: 0` for a non-empty prompt (issue #605). + #[tokio::test] + async fn json_response_reports_the_input_estimate_for_a_stopped_turn() { + let sse = concat!( + "event: response.created\n", + "data: {\"response\":{\"id\":\"resp_1\"}}\n\n", + "event: response.output_item.added\n", + "data: {\"item\":{\"type\":\"message\"}}\n\n", + "event: response.output_text.delta\n", + "data: {\"delta\":\"keep<>drop\"}\n\n", + "event: response.output_text.done\n", + "data: {}\n\n", + "event: response.completed\n", + "data: {\"response\":{\"usage\":{\"input_tokens\":42,\"output_tokens\":7}}}\n\n", + ); + let relay = super::super::context::RelayOptions { + stop_sequences: vec!["<>".to_string()], + ..relay_opts() + }; + let upstream = upstream_response(200, sse).await; + let response = json_response(upstream, relay, 11) + .await + .expect("json_response builds a response"); + + let body = response_body_json(response).await; + assert_eq!(body["content"][0]["text"], "keep"); + assert_eq!(body["stop_reason"], "stop_sequence"); + assert_eq!(body["stop_sequence"], "<>"); + // The seed, not the upstream's 42: the stop made that usage a no-op. + assert_eq!(body["usage"]["input_tokens"], 11); + } + /// The streaming path prefixes a synthesized completion with /// `stream_metrics::UPSTREAM_TRUNCATED_MARKER` when the upstream /// connection ends before a real terminal event, so the observer can @@ -419,6 +476,7 @@ mod tests { client_wants_stream: true, thinking_enabled: false, tool_search_native: false, + stop_sequences: Vec::new(), }, codex_quota_account: None, estimate_input: None, @@ -483,6 +541,7 @@ mod tests { client_wants_stream: true, thinking_enabled: false, tool_search_native: false, + stop_sequences: Vec::new(), }, codex_quota_account: None, estimate_input: None, diff --git a/src/adapters/responses/mod.rs b/src/adapters/responses/mod.rs index 89e5827f0..3322da30f 100644 --- a/src/adapters/responses/mod.rs +++ b/src/adapters/responses/mod.rs @@ -124,22 +124,43 @@ async fn forward( tool_search_native, "resolved tool_search protocol" ); + // The Responses API has no `stop` parameter (Chat Completions does; Responses + // does not), so `stop_sequences` is emulated gateway-side in the + // Responses->Anthropic SSE translation rather than forwarded upstream + // (issue #605). Kept in request order: ties on the match position break by it. + let stop_sequences: Vec = request_json + .get("stop_sequences") + .and_then(Value::as_array) + .map(|values| { + values + .iter() + .filter_map(Value::as_str) + .filter(|sequence| !sequence.is_empty()) + .map(str::to_string) + .collect() + }) + .unwrap_or_default(); // Seed message_start's usage.input_tokens with a local tiktoken estimate of // the (already-parsed) request so Claude Code's per-subagent progress // tracker — which reads that first snapshot and never re-reads the merged // total — shows a live context figure for codex subagents instead of a stuck // 0. The Responses API only reports real usage at response.completed, by // which point message_start is long sent; the accurate total still lands in - // the terminal message_delta. Only streaming turns emit message_start, so - // non-streaming requests carry `None` and skip the work; gated on the - // provider's local-counting opt-in (the same CountTokens knob as the - // count_tokens endpoint). The CPU-bound tiktoken encode itself is deferred to + // the terminal message_delta. Gated on the provider's local-counting opt-in + // (the same CountTokens knob as the count_tokens endpoint). The CPU-bound tiktoken encode itself is deferred to // each transport, where it runs on the blocking pool overlapped with the // upstream round-trip rather than serially in front of it (see forward_http / // forward_websocket); the multi-upstream chain races it against each // attempt's dispatch (the pool's first poll via `pooled_first_poll`, the // non-pooled send via `send_classified_with_estimate`). See model/responses.rs. - let estimate_input = if client_wants_stream + // + // A non-streaming turn emits no `message_start` and would otherwise skip the + // work, but one carrying `stop_sequences` needs the estimate for a second + // reason: an emulated stop makes the upstream's own `response.completed` + // usage a no-op, so without a seed the final JSON reports `input_tokens: 0` + // for a non-empty prompt (issue #605). `final_json` falls back to the + // estimate exactly when no usage was observed, so seeding it here is enough. + let estimate_input = if (client_wants_stream || !stop_sequences.is_empty()) && matches!( state .config @@ -156,6 +177,7 @@ async fn forward( client_wants_stream, thinking_enabled, tool_search_native, + stop_sequences, }; let upstream_body = Arc::new(translate_request_value( request_json, @@ -308,7 +330,7 @@ async fn forward( let websocket_options = ForwardOptions { upstream_body: upstream_body.clone(), auth, - turn, + turn: turn.clone(), codex_quota_account: codex_quota_account.clone(), estimate_input: estimate_input.clone(), started_at: None, @@ -394,10 +416,26 @@ pub(crate) async fn chain_attempt( let tool_search_native = state .config .native_tool_search(&route.provider, &route.upstream_model); + // See `forward`'s matching extraction: the Responses API has no `stop` + // parameter, so `stop_sequences` is emulated gateway-side rather than + // forwarded upstream (issue #605). + let stop_sequences = request_json + .get("stop_sequences") + .and_then(Value::as_array) + .map(|values| { + values + .iter() + .filter_map(Value::as_str) + .filter(|sequence| !sequence.is_empty()) + .map(str::to_string) + .collect() + }) + .unwrap_or_default(); let turn = TurnOptions { client_wants_stream: true, thinking_enabled, tool_search_native, + stop_sequences, }; let upstream_body = Arc::new(translate_request_value( request_json, diff --git a/src/adapters/responses/pool.rs b/src/adapters/responses/pool.rs index a1b8c59b7..f507eee92 100644 --- a/src/adapters/responses/pool.rs +++ b/src/adapters/responses/pool.rs @@ -949,7 +949,7 @@ pub(super) async fn forward_chatgpt_oauth( ForwardOptions { upstream_body: upstream_body.clone(), auth, - turn, + turn: turn.clone(), codex_quota_account: Some(account.clone()), // Each account attempt gets its own cheap Arc clone; // forward_websocket spawns its own blocking encode from it, @@ -1195,7 +1195,7 @@ async fn relay_success( keepalive, )) } else { - json_response(upstream, relay).await + json_response(upstream, relay, input_tokens_estimate).await } } @@ -1204,10 +1204,11 @@ async fn relay_success( /// out because the pool loop has two success arms (first attempt and /// refresh retry) that must consume the same handle without awaiting it /// twice — `JoinHandle` is not `Clone`, so `.take()` leaves a torn-down `None` -/// behind for whichever arm does not run. Non-streaming turns never seed -/// `message_start`, so `estimate_handle` is always `None` here already -/// (`forward`'s gate only produces `estimate_input`, and thus a spawned -/// handle, for streaming turns), which naturally yields `0` below. +/// behind for whichever arm does not run. A non-streaming turn seeds no +/// `message_start`, so `forward`'s gate spawns a handle for one only when it +/// carries `stop_sequences` — there the estimate is what keeps a stopped turn's +/// final JSON from reporting `input_tokens: 0` (issue #605). Without either, the +/// handle is `None` and this naturally yields `0` below. async fn take_estimate(estimate_handle: &mut Option>) -> u64 { match estimate_handle.take() { // Bounded like `forward_http`: the committed `message_start` must not @@ -1577,6 +1578,7 @@ mod tests { client_wants_stream: stream, thinking_enabled: false, tool_search_native: false, + stop_sequences: Vec::new(), }, estimate_input: None, } diff --git a/src/adapters/responses/sse_parse.rs b/src/adapters/responses/sse_parse.rs index 52aad4f29..af23dbac2 100644 --- a/src/adapters/responses/sse_parse.rs +++ b/src/adapters/responses/sse_parse.rs @@ -426,7 +426,25 @@ where "a terminal item implies the producer is live" ); }; - spawn_terminal_drain(events); + // An emulated stop sequence (issue #605) + // means the upstream is still mid-turn and + // will keep generating text nobody will + // see: drop `events` outright so the byte + // stream is dropped and the upstream + // connection is aborted. Any other + // terminal (a real `response.completed`/ + // `done`/`incomplete`, or a backend + // `error`/`response.failed`) means the + // upstream already finished, so keep + // draining it to EOF here so the + // connection still pools — aborting on an + // ordinary terminal would cost every + // normal turn its pooled connection. + if active.hit_stop_sequence() { + drop(events); + } else { + spawn_terminal_drain(events); + } return Some(( Ok(Bytes::from(data)), (Producer::Done, Some(active), finished, map, None), @@ -440,11 +458,19 @@ where if active.is_stopped() { // Unreachable for the current maps (every // stopping event emits); drain for pooling - // and end, defensively. + // and end, defensively — except an emulated + // stop sequence, which drops the events the + // same way the reachable arm above does, so + // the upstream is still aborted rather than + // drained. let Producer::Live(events) = producer else { unreachable!("a live producer precedes its item"); }; - spawn_terminal_drain(events); + if active.hit_stop_sequence() { + drop(events); + } else { + spawn_terminal_drain(events); + } return None; } machine = Some(active); @@ -597,6 +623,7 @@ mod tests { model: "gpt-5.2-codex".to_string(), thinking_enabled: false, tool_search_native: false, + stop_sequences: Vec::new(), } .machine() .with_input_estimate(0) diff --git a/src/adapters/responses/websocket.rs b/src/adapters/responses/websocket.rs index b9c0c525f..fe65319b8 100644 --- a/src/adapters/responses/websocket.rs +++ b/src/adapters/responses/websocket.rs @@ -19,6 +19,7 @@ use crate::{ use super::codex_continuation; use super::codex_ws::{self, CodexWsError, CodexWsEvents}; use super::context::ForwardOptions; +use super::early_stream::bounded_input_estimate; use super::error::build_upstream_error; use super::request::{responses_url, routing_hint, CODEX_CLIENT_VERSION, CODEX_USER_AGENT}; use super::ws_stream::{json_events_response, stream_events_response}; @@ -71,11 +72,21 @@ pub(super) async fn forward_websocket( tokio::task::spawn_blocking(move || crate::count_tokens::count_input_tokens_value(&request)) }); let (buffered, events) = open_ws_turn(&ctx).await?; + // Both branches consume it: the streaming arm seeds `message_start`, and a + // non-streaming turn cut short by an emulated stop sequence needs it because + // the stop makes the upstream's own usage a no-op (issue #605). + // + // Bounded, unlike the bare `handle.await` this replaces: the turn is already + // open by now, so blocking here stops the collector consuming events and + // backpressures the bounded `CodexWsEvents` channel until tokenization ends. + // The encode has had the whole `open_ws_turn` to finish, so the bound only + // bites when the blocking pool is saturated — the same trade the HTTP and + // pooled paths already make. + let input_tokens_estimate = match estimate_handle { + Some(handle) => bounded_input_estimate(handle, std::time::Duration::from_secs(1)).await, + None => 0, + }; if turn.client_wants_stream { - let input_tokens_estimate = match estimate_handle { - Some(handle) => handle.await.unwrap_or(0), - None => 0, - }; let keepalive = std::time::Duration::from_secs(state.config.server.sse_keepalive_seconds); Ok(( StatusCode::OK, @@ -91,7 +102,9 @@ pub(super) async fn forward_websocket( // See `forward_http`: surface the real status (a `502` when a backend // error event fired, issue #113) to the access log and metrics rather // than a hardcoded `200`. - let response = json_events_response(buffered, events, turn.relay(route)).await?; + let response = + json_events_response(buffered, events, turn.relay(route), input_tokens_estimate) + .await?; Ok((response.status(), response)) } } diff --git a/src/adapters/responses/ws_stream.rs b/src/adapters/responses/ws_stream.rs index c69d32a14..94a31c6c4 100644 --- a/src/adapters/responses/ws_stream.rs +++ b/src/adapters/responses/ws_stream.rs @@ -36,6 +36,11 @@ pub(super) fn stream_events_response( .machine() .with_input_estimate(input_tokens_estimate) .without_content_accumulation(); + // The receiver is held in an `Option` so an emulated stop sequence can drop + // it *with* the final chunk (issue #605): the codex_ws reader sees the + // receiver closed, abandons the turn and evicts the socket, which is exactly + // right for a turn shunt cut short — a half-consumed turn must not be pooled. + let events = Some(events); let output = stream::unfold( (buffered, events, machine, false), |(mut buffered, mut events, mut machine, finished)| async move { @@ -45,15 +50,26 @@ pub(super) fn stream_events_response( loop { let item = match buffered.take() { Some(item) => Some(item), - None => events.recv().await, + None => match events.as_mut() { + Some(events) => events.recv().await, + None => return None, + }, }; match item { Some(Ok(event)) => { let data = machine.apply(event).into_iter().collect::(); if !data.is_empty() { + // Only a stop-sequence stop aborts the upstream: a + // normally completed turn must keep its receiver + // alive so codex_ws can pool the socket instead of + // reading a client cancellation into it. + let aborted = machine.hit_stop_sequence(); + if aborted { + events = None; + } return Some(( Ok::<_, std::convert::Infallible>(Bytes::from(data)), - (buffered, events, machine, false), + (buffered, events, machine, aborted), )); } } @@ -105,8 +121,13 @@ pub(super) async fn json_events_response( buffered: BufferedEvent, mut events: CodexWsEvents, relay: RelayOptions, + input_tokens_estimate: u64, ) -> Result { - let mut machine = relay.machine(); + // Seeded like every other path: an emulated stop sequence makes the + // upstream's `response.completed` usage a no-op, and `final_json` falls back + // to this estimate when no usage was observed, so without it a stopped turn + // reports `input_tokens: 0` for a non-empty prompt (issue #605). + let mut machine = relay.machine().with_input_estimate(input_tokens_estimate); let mut buffered = buffered; loop { let item = match buffered.take() { @@ -123,6 +144,13 @@ pub(super) async fn json_events_response( if let Some((status, error)) = machine.take_backend_error() { return Err(backend_error(status, error)); } + // An emulated stop sequence (issue #605) is terminal for the same + // reason, except the upstream is still mid-turn: return now and + // let the dropped receiver abort it rather than draining the rest + // of an answer this message no longer contains. + if machine.hit_stop_sequence() { + break; + } } Some(Err(error)) => { tracing::warn!(error = %error.message, "codex websocket stream error"); @@ -167,12 +195,18 @@ mod tests { use super::{json_events_response, stream_events_response, ws_error_sse, RelayOptions}; /// The default relay options for these tests: the `gpt-5.2-codex` model with - /// both protocol toggles off. + /// both protocol toggles off and no emulated stop sequences. fn relay_opts() -> RelayOptions { + relay_opts_with_stops(&[]) + } + + /// [`relay_opts`] plus emulated Anthropic `stop_sequences` (issue #605). + fn relay_opts_with_stops(sequences: &[&str]) -> RelayOptions { RelayOptions { model: "gpt-5.2-codex".to_string(), thinking_enabled: false, tool_search_native: false, + stop_sequences: sequences.iter().map(|s| (*s).to_string()).collect(), } } @@ -216,7 +250,7 @@ mod tests { .unwrap(); drop(tx); - let error = json_events_response(None, rx, relay_opts()) + let error = json_events_response(None, rx, relay_opts(), 0) .await .expect_err("mid-stream transport error should stop failover"); assert!(error.failure.is_none()); @@ -240,7 +274,7 @@ mod tests { .unwrap(); drop(tx); - let response = json_events_response(None, rx, relay_opts()) + let response = json_events_response(None, rx, relay_opts(), 0) .await .expect("clean events should build a response"); assert_eq!(response.status(), StatusCode::OK); @@ -269,7 +303,7 @@ mod tests { .unwrap(); drop(tx); - let error = json_events_response(None, rx, relay_opts()) + let error = json_events_response(None, rx, relay_opts(), 0) .await .expect_err("backend error event should stop failover"); @@ -300,7 +334,7 @@ mod tests { .unwrap(); drop(tx); - let error = json_events_response(None, rx, relay_opts()) + let error = json_events_response(None, rx, relay_opts(), 0) .await .expect_err("in-stream rate limit is an error"); @@ -335,7 +369,7 @@ mod tests { let error = tokio::time::timeout( std::time::Duration::from_secs(5), - json_events_response(None, rx, relay_opts()), + json_events_response(None, rx, relay_opts(), 0), ) .await .expect("collector returns without waiting for channel close") @@ -399,4 +433,100 @@ mod tests { // `response.created` opens the stream with `message_start`. assert!(text.contains("message_start")); } + + fn text_delta_event(delta: &str) -> ResponseEvent { + ResponseEvent { + event: Some("response.output_text.delta".to_string()), + data: json!({ "delta": delta }), + } + } + + /// An emulated stop sequence is terminal for the collector even though the + /// upstream is still mid-turn: it must return without waiting for the channel + /// to close, mirroring the backend-error early return (issue #605). + #[tokio::test] + async fn json_events_response_returns_on_a_stop_sequence_without_channel_close() { + let (tx, rx) = mpsc::channel(16); + tx.try_send(Ok(created_event())).unwrap(); + tx.try_send(Ok(text_delta_event("answergarbage"))) + .unwrap(); + // Deliberately keep `tx` alive: the upstream is still generating, so the + // collector must return on the stop rather than draining to close. + + let response = tokio::time::timeout( + std::time::Duration::from_secs(5), + json_events_response(None, rx, relay_opts_with_stops(&[""]), 0), + ) + .await + .expect("collector returns without waiting for channel close") + .expect("a stop sequence is a successful turn"); + + assert_eq!(response.status(), StatusCode::OK); + let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let body: Value = serde_json::from_slice(&bytes).unwrap(); + assert_eq!(body["content"][0]["text"], "answer"); + assert_eq!(body["stop_reason"], "stop_sequence"); + assert_eq!(body["stop_sequence"], ""); + + drop(tx); + } + + /// A non-streaming websocket turn cut short by a stop sequence reports the + /// seeded input estimate. The stop makes the upstream's own `response.completed` + /// usage a no-op, so without the seed this path returns `input_tokens: 0` for a + /// non-empty prompt — the websocket sibling of the HTTP case (issue #605). + #[tokio::test] + async fn json_events_response_reports_the_input_estimate_for_a_stopped_turn() { + let (tx, rx) = mpsc::channel(16); + tx.try_send(Ok(created_event())).unwrap(); + tx.try_send(Ok(text_delta_event("answergarbage"))) + .unwrap(); + + let response = tokio::time::timeout( + std::time::Duration::from_secs(5), + json_events_response(None, rx, relay_opts_with_stops(&[""]), 19), + ) + .await + .expect("collector returns without waiting for channel close") + .expect("a stop sequence is a successful turn"); + + let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let body: Value = serde_json::from_slice(&bytes).unwrap(); + assert_eq!(body["stop_reason"], "stop_sequence"); + assert_eq!(body["usage"]["input_tokens"], 19); + + drop(tx); + } + + /// The streaming path drops the event receiver the moment the stop fires, so + /// the codex_ws reader sees the turn abandoned and evicts the socket instead + /// of letting the upstream keep generating (issue #605). + #[tokio::test] + async fn stream_events_response_aborts_the_upstream_on_a_stop_sequence() { + let (tx, rx) = mpsc::channel(16); + tx.try_send(Ok(created_event())).unwrap(); + tx.try_send(Ok(text_delta_event("answergarbage"))) + .unwrap(); + + let response = stream_events_response( + None, + rx, + relay_opts_with_stops(&[""]), + 0, + std::time::Duration::from_secs(15), + ); + let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let text = String::from_utf8_lossy(&bytes); + + assert!(text.contains("\"text\":\"answer\""), "got: {text}"); + assert!(!text.contains("garbage"), "post-stop text leaked: {text}"); + assert!( + text.contains("\"stop_reason\":\"stop_sequence\""), + "got: {text}" + ); + assert!( + tx.is_closed(), + "the receiver must be dropped so the upstream turn is aborted" + ); + } } diff --git a/src/config.rs b/src/config.rs index c3e4999c3..bc647be5c 100644 --- a/src/config.rs +++ b/src/config.rs @@ -1503,12 +1503,14 @@ pub struct ProviderConfig { /// environment variable it moves the classifier without changing what /// `sonnet` resolves to for the rest of the session. /// - /// Deliberately a model *remap within this provider*, not a route: the - /// classifier request carries `stop_sequences`, which the Responses - /// translation drops, and a classifier pointed at a Responses upstream - /// retries its first stage and adds seconds to every permission check. Off - /// by default — an unset key leaves the classifier on whatever model the - /// client asked for. + /// Deliberately a model *remap within this provider*, not a route. The + /// classifier request depends on `stop_sequences`: while the Responses + /// translation still dropped that field, a classifier pointed at a + /// Responses upstream retried its first stage and added seconds to every + /// permission check. The `model::stop_sequences` scanner now emulates them + /// gateway-side (issue #605), so that failure no longer applies — the key + /// stays an in-provider remap regardless. Off by default — an unset key + /// leaves the classifier on whatever model the client asked for. #[serde(default, skip_serializing_if = "Option::is_none")] pub classifier_model: Option, /// How `POST /v1/messages/count_tokens` is answered for this provider. diff --git a/src/model/mod.rs b/src/model/mod.rs index b8c1dc864..c78caf477 100644 --- a/src/model/mod.rs +++ b/src/model/mod.rs @@ -5,4 +5,5 @@ pub mod inbound_responses; pub mod responses; pub mod responses_request; pub mod responses_schema; +pub(crate) mod stop_sequences; pub mod thinking_signature; diff --git a/src/model/responses.rs b/src/model/responses.rs index 7d5fe4470..455648cea 100644 --- a/src/model/responses.rs +++ b/src/model/responses.rs @@ -7,6 +7,7 @@ use crate::model::responses_request::TOOL_SEARCH_NAME; pub use crate::model::responses_request::{ encode_reasoning_signature, translate_request, translate_request_value, }; +use crate::model::stop_sequences::StopScanner; #[derive(Debug, Clone)] pub struct ResponseEvent { @@ -88,6 +89,10 @@ pub struct AnthropicSseMachine { /// ([`backend_error_status`]): `429` for an in-stream `rate_limit_exceeded`, /// else `502`. backend_error: Option<(StatusCode, Value)>, + /// The client's Anthropic `stop_sequences`, emulated gateway-side because the + /// Responses API has no `stop` parameter (issue #605). Empty — the common + /// case — leaves every text path byte-identical to before the feature. + stop: StopScanner, } #[derive(Debug, Clone)] @@ -134,6 +139,7 @@ impl AnthropicSseMachine { web_search_indexes: HashMap::new(), tool_search_native, backend_error: None, + stop: StopScanner::default(), } } @@ -146,6 +152,33 @@ impl AnthropicSseMachine { self } + /// Emulate the client's Anthropic `stop_sequences` on this Responses stream + /// (issue #605). Assistant text is truncated at the first occurrence of any + /// of them, the turn ends with `stop_reason: "stop_sequence"`, and the caller + /// aborts the upstream. An empty list is a no-op. + #[must_use] + pub fn with_stop_sequences(mut self, sequences: Vec) -> Self { + self.stop = StopScanner::new(sequences); + self + } + + /// Whether the machine has reached a terminal state — a real upstream + /// terminal, a backend error event, or an emulated stop sequence. Transports + /// poll this right after [`Self::apply`] to drop the upstream stream instead + /// of reading a response no one will see. + pub fn is_stopped(&self) -> bool { + self.stopped + } + + /// Whether an emulated stop sequence fired (issue #605). Distinct from + /// [`Self::is_stopped`]: this is the only case where the *upstream* is still + /// mid-turn, so it is what the websocket transport keys its abort on — a + /// receiver dropped on a normally completed turn would evict a healthy + /// pooled socket (see `adapters::responses::codex_ws`). + pub fn hit_stop_sequence(&self) -> bool { + self.stop.matched().is_some() + } + /// Disable final-message reconstruction for a streaming relay. Incremental SSE /// output and usage tracking are unchanged, while text, tool, and reasoning /// payloads are no longer retained after being emitted. @@ -186,6 +219,12 @@ impl AnthropicSseMachine { self.complete(&event.data) } "error" | "response.failed" => { + // Release the stop-sequence holdback first: it is ordinary output + // the client would already have received had `stop_sequences` been + // unset, and once `stopped` is set nothing else can emit it. The + // open block is deliberately *not* closed here — this arm has + // always left it open, with or without stop sequences. + let mut out = self.flush_stop_holdback(); self.stopped = true; let status = backend_error_status(&event.data); let value = map_error_value(&event.data, status); @@ -193,7 +232,8 @@ impl AnthropicSseMachine { // ownership into `backend_error` — avoids cloning the envelope. let sse_event = sse("error", &value); self.backend_error = Some((status, value)); - vec![sse_event] + out.push(sse_event); + out } _ => Vec::new(), } @@ -208,14 +248,6 @@ impl AnthropicSseMachine { self.backend_error.take() } - /// Whether a terminal event (`response.completed`/`response.done`/ - /// `response.incomplete` or an error frame) has already been applied, so a - /// producer error after that point must not append an `error` event to a - /// completed turn. - pub fn is_stopped(&self) -> bool { - self.stopped - } - pub fn finish(&mut self) -> Vec { if self.stopped { return Vec::new(); @@ -229,14 +261,23 @@ impl AnthropicSseMachine { if !self.stopped { let _ = self.finish(); } + // An emulated stop sequence (issue #605) takes precedence over the + // tool_use/end_turn choice: the turn ended because shunt cut it, and the + // accumulated `content` is already truncated at the match. + let matched = self.stop.matched().map(str::to_string); + let stop_reason = match (&matched, self.saw_tool) { + (Some(_), _) => "stop_sequence", + (None, true) => "tool_use", + (None, false) => "end_turn", + }; json!({ "id": self.id, "type": "message", "role": "assistant", "model": self.model, "content": self.content, - "stop_reason": if self.saw_tool { "tool_use" } else { "end_turn" }, - "stop_sequence": null, + "stop_reason": stop_reason, + "stop_sequence": matched.map_or(Value::Null, Value::from), "usage": self.usage_value(), }) } @@ -569,19 +610,55 @@ impl AnthropicSseMachine { if delta.is_empty() { return Vec::new(); } + if self.stop.is_empty() { + return self.emit_text(delta); + } + // Emulated `stop_sequences` (issue #605): the scanner emits everything up + // to a match (holding back only a partial stop that may still complete in + // the next delta) and reports the match, at which point the whole message + // ends here — text after the stop is neither emitted nor accumulated, and + // `stopped` makes every later event, `response.completed` included, a + // no-op. + let mut out = Vec::new(); + // Open the text block on the delta itself, exactly as the unconfigured + // path does — not on the first byte that survives the scan. A delta that + // is held back in its entirety (the whole message so far is a proper + // prefix of a stop sequence) would otherwise leave no block open, and + // `close_any`'s flush — which is gated on an open text block, and cannot + // open one itself without recursing back through `open_text` — would + // never release it. + if self.open.as_ref().map(|block| block.kind) != Some(BlockKind::Text) { + out.extend(self.open_text()); + } + let scan = self.stop.push(delta); + if !scan.emit.is_empty() { + out.extend(self.emit_text(&scan.emit)); + } + if scan.matched.is_some() { + out.extend(self.close_any()); + out.extend(self.stop_events("stop_sequence")); + } + out + } + + /// Stream one run of assistant text: open the text block if this is the first + /// of it, record it for the non-streaming reconstruction, and emit the + /// `text_delta`. Only text that actually reaches the client passes through + /// here, so `text_buffer` and the streamed output can never disagree. + fn emit_text(&mut self, text: &str) -> Vec { let mut out = Vec::new(); if self.open.as_ref().map(|block| block.kind) != Some(BlockKind::Text) { out.extend(self.open_text()); } if self.accumulate_content { - self.text_buffer.push_str(delta); + self.text_buffer.push_str(text); } out.push(sse( "content_block_delta", &json!({ "type": "content_block_delta", "index": self.open_index(), - "delta": {"type": "text_delta", "text": delta} + "delta": {"type": "text_delta", "text": text} }), )); out @@ -734,9 +811,35 @@ impl AnthropicSseMachine { self.close_any() } + /// Release whatever the stop-sequence scanner is still holding back as an + /// ordinary `text_delta`. A holdback is only ever a *partial* stop that never + /// completed, so it is output the client is owed — and it is owed it on every + /// terminal, not only on the ones that close the block ([`Self::close_any`]). + /// Empty unless stop sequences are configured, something is actually held, + /// and the open block is text. + fn flush_stop_holdback(&mut self) -> Vec { + let mut out = Vec::new(); + if !self.stop.is_empty() + && self.open.as_ref().map(|block| block.kind) == Some(BlockKind::Text) + { + let held = self.stop.flush(); + if !held.is_empty() { + out.extend(self.emit_text(&held)); + } + } + out + } + fn close_any(&mut self) -> Vec { + // Text the stop-sequence scanner is still holding back is a partial stop + // that never completed — ordinary output. Flush it here, before this + // block's `content_block_stop` and before the accumulation below reads + // `text_buffer`, so every path that closes a text block (`close_current`, + // `open_tool`, `open_reasoning`, `output_item_done`, `complete`, `finish`) + // releases it exactly once. + let mut out = self.flush_stop_holdback(); let Some(open) = self.open.take() else { - return Vec::new(); + return out; }; match open.kind { BlockKind::Text => { @@ -782,10 +885,11 @@ impl AnthropicSseMachine { } } self.index += 1; - vec![sse( + out.push(sse( "content_block_stop", &json!({"type": "content_block_stop", "index": open.index}), - )] + )); + out } fn complete(&mut self, data: &Value) -> Vec { @@ -802,12 +906,15 @@ impl AnthropicSseMachine { fn stop_events(&mut self, stop_reason: &str) -> Vec { self.stopped = true; + // `stop_sequence` names the emulated stop that fired (issue #605), and is + // `null` on every other terminal — exactly as before the feature. + let stop_sequence = self.stop.matched().map_or(Value::Null, Value::from); vec![ sse( "message_delta", &json!({ "type": "message_delta", - "delta": {"stop_reason": stop_reason, "stop_sequence": null}, + "delta": {"stop_reason": stop_reason, "stop_sequence": stop_sequence}, // Carry input_tokens here (not message_start): the Responses // API only reports usage at response.completed, so this is the // first point shunt knows the prompt size. The Anthropic SDK @@ -1540,4 +1647,334 @@ mod tests { "pre-delta citation must be streamed as a citations_delta frame" ); } + // ---- emulated `stop_sequences` (issue #605) ---------------------------- + + /// A streaming machine with `stop_sequences` configured, matching what the + /// HTTP/websocket transports build. + fn stop_machine(sequences: &[&str]) -> AnthropicSseMachine { + AnthropicSseMachine::new("test", true, false) + .with_stop_sequences(sequences.iter().map(|s| (*s).to_string()).collect()) + } + + /// The `delta` of every `text_delta` frame in `frames`, concatenated. + fn streamed_text(frames: &[String]) -> String { + frames + .iter() + .filter(|frame| frame.contains("\"type\":\"text_delta\"")) + .filter_map(|frame| { + let data = frame.split("data: ").nth(1)?; + let value: Value = serde_json::from_str(data.trim_end()).ok()?; + value["delta"]["text"].as_str().map(str::to_string) + }) + .collect() + } + + /// The terminal `message_delta` payload, or `None` if the machine never + /// emitted one. + fn message_delta(frames: &[String]) -> Option { + frames + .iter() + .find(|frame| frame.contains("message_delta")) + .and_then(|frame| frame.split("data: ").nth(1)) + .and_then(|data| serde_json::from_str(data.trim_end()).ok()) + } + + /// A stop sequence wholly inside one delta truncates the message there: the + /// preceding text is emitted, the turn ends with `stop_reason: + /// "stop_sequence"`, and every later event is a no-op. + #[test] + fn stop_sequence_inside_one_delta_ends_the_message() { + let mut machine = stop_machine(&[""]); + let mut frames = machine.apply(event("response.created", json!({"id": "resp_1"}))); + frames.extend(machine.apply(event( + "response.output_text.delta", + json!({"delta": "allowtrailing junk"}), + ))); + + assert_eq!(streamed_text(&frames), "allow"); + assert!(frames + .iter() + .any(|frame| frame.contains("content_block_stop"))); + assert_eq!( + message_delta(&frames).expect("a stop sequence emits message_delta")["delta"], + json!({"stop_reason": "stop_sequence", "stop_sequence": ""}) + ); + assert!(frames.iter().any(|frame| frame.contains("message_stop"))); + + // Everything after the stop is ignored, `response.completed` included. + assert!(machine + .apply(event( + "response.output_text.delta", + json!({"delta": "more"}) + )) + .is_empty()); + assert!(machine + .apply(event("response.completed", json!({"response": {}}))) + .is_empty()); + } + + /// A stop sequence split across two deltas is still caught, and the partial + /// stop is never streamed as text. + #[test] + fn stop_sequence_split_across_deltas_is_caught() { + let mut machine = stop_machine(&[""]); + let first = machine.apply(event( + "response.output_text.delta", + json!({"delta": "allow"}))); + assert_eq!(streamed_text(&second), ""); + assert_eq!( + message_delta(&second).expect("the completed stop emits message_delta")["delta"], + json!({"stop_reason": "stop_sequence", "stop_sequence": ""}) + ); + } + + /// A held-back prefix that never completes is ordinary output: it is flushed + /// before the text block closes, and the turn ends normally. + #[test] + fn an_incomplete_stop_prefix_is_flushed_before_the_block_closes() { + let mut machine = stop_machine(&[""]); + let mut frames = machine.apply(event( + "response.output_text.delta", + json!({"delta": "abc "]); + let mut frames = machine.apply(event("response.created", json!({"id": "resp_1"}))); + frames.extend(machine.apply(event( + "response.output_text.delta", + json!({"delta": ""]); + let reasoning = machine.apply(event( + "response.reasoning_summary_text.delta", + json!({"delta": "thinking still"}), + )); + assert!(reasoning + .iter() + .any(|frame| frame.contains("thinking_delta"))); + assert!(message_delta(&reasoning).is_none()); + + let mut machine = stop_machine(&[""]); + machine.apply(event( + "response.output_item.added", + json!({"item": {"type": "function_call", "call_id": "call_1", "name": "run"}}), + )); + let arguments = machine.apply(event( + "response.function_call_arguments.delta", + json!({"delta": "{\"cmd\":\"\"}"}), + )); + assert!(arguments + .iter() + .any(|frame| frame.contains("input_json_delta"))); + assert!(message_delta(&arguments).is_none()); + } + + /// Every event of a full turn, as a machine consumes it. + fn mixed_turn() -> Vec { + vec![ + event("response.created", json!({"id": "resp_1"})), + event( + "response.reasoning_summary_text.delta", + json!({"delta": "pondering"}), + ), + event( + "response.output_item.added", + json!({"item": {"type": "message"}}), + ), + event("response.output_text.delta", json!({"delta": "hello "})), + event("response.output_text.delta", json!({"delta": "world"})), + event("response.output_text.done", json!({})), + event( + "response.output_item.added", + json!({"item": {"type": "function_call", "call_id": "call_1", "name": "run"}}), + ), + event( + "response.function_call_arguments.delta", + json!({"delta": "{\"cmd\":\"ls\"}"}), + ), + event("response.function_call_arguments.done", json!({})), + event( + "response.completed", + json!({"response": {"usage": {"input_tokens": 7, "output_tokens": 3}}}), + ), + ] + } + + fn drive(machine: &mut AnthropicSseMachine) -> Vec { + mixed_turn() + .into_iter() + .flat_map(|event| machine.apply(event)) + .collect() + } + + /// With no stop sequence configured the translation must be byte-identical to + /// a machine that never heard of the feature — the fast path stays untouched. + #[test] + fn an_empty_stop_sequence_list_is_byte_identical() { + let mut baseline = AnthropicSseMachine::new("test", true, false); + let mut configured = + AnthropicSseMachine::new("test", true, false).with_stop_sequences(Vec::new()); + + assert_eq!(drive(&mut configured), drive(&mut baseline)); + } + + /// Stop sequences that never match change only how text is chunked, never the + /// text itself or the terminal events. + #[test] + fn unmatched_stop_sequences_preserve_the_turn() { + let mut baseline = AnthropicSseMachine::new("test", true, false); + let mut configured = stop_machine(&["", ""]); + let baseline_frames = drive(&mut baseline); + let configured_frames = drive(&mut configured); + + assert_eq!( + streamed_text(&configured_frames), + streamed_text(&baseline_frames) + ); + assert_eq!( + message_delta(&configured_frames), + message_delta(&baseline_frames) + ); + assert_eq!(configured.final_json(), baseline.final_json()); + } + + /// The non-streaming reconstruction is truncated at the stop too, and reports + /// the stop in `stop_reason` / `stop_sequence`. + #[test] + fn final_json_is_truncated_at_the_stop_sequence() { + let mut machine = stop_machine(&[""]); + machine.apply(event("response.created", json!({"id": "resp_1"}))); + machine.apply(event( + "response.output_text.delta", + json!({"delta": "answergarbage"}), + )); + machine.apply(event("response.completed", json!({"response": {}}))); + + let final_json = machine.final_json(); + assert_eq!(final_json["content"][0]["text"], "answer"); + assert_eq!(final_json["content"][1], Value::Null); + assert_eq!(final_json["stop_reason"], "stop_sequence"); + assert_eq!(final_json["stop_sequence"], ""); + } + + /// The holdback is sliced on char boundaries, so a multi-byte code point that + /// precedes a split stop sequence is emitted whole. + #[test] + fn a_multibyte_code_point_before_the_stop_survives() { + let mut machine = stop_machine(&[""]); + let first = machine.apply(event( + "response.output_text.delta", + json!({"delta": "한글"}), + )); + + assert_eq!(streamed_text(&first), "한글"); + assert_eq!(streamed_text(&second), ""); + assert_eq!( + message_delta(&second).expect("the stop emits message_delta")["delta"]["stop_sequence"], + "" + ); + } + + /// With two stop sequences configured and both present, the one that starts + /// earliest wins regardless of the order they were configured in. + #[test] + fn the_earliest_stop_sequence_wins() { + let mut machine = stop_machine(&["", ""]); + let frames = machine.apply(event( + "response.output_text.delta", + json!({"delta": "abc"}), + )); + + assert_eq!(streamed_text(&frames), "a"); + assert_eq!( + message_delta(&frames).expect("the stop emits message_delta")["delta"]["stop_sequence"], + "" + ); + } + + /// A backend `error` / `response.failed` event is terminal, so it is the last + /// chance to release the holdback: the partial stop never completed, which + /// makes it ordinary output the client would already have seen had + /// `stop_sequences` been unset. It must reach the client *before* the `error` + /// frame. + #[test] + fn a_backend_error_releases_the_held_back_stop_prefix() { + let mut machine = stop_machine(&[""]); + let mut frames = machine.apply(event( + "response.output_text.delta", + json!({"delta": "answer"]`, `[""]`), and the trailing text past +//! the stop breaks its parser. +//! +//! [`StopScanner`] is the piece the Responses→Anthropic SSE translation +//! ([`crate::model::responses::AnthropicSseMachine`]) drives over *assistant text* +//! only: reasoning summaries and tool-call arguments are never scanned. It holds +//! back at most `max_len - 1` bytes — the longest suffix that could still grow +//! into a stop sequence — so a stop split across two upstream deltas is still +//! caught, and everything else is emitted immediately (streaming is preserved). + +/// What one scanned delta yields. +pub(crate) struct StopScan { + /// Text to emit to the client. May be empty (everything was held back). + pub emit: String, + /// The stop sequence that matched, if this delta completed one. Text after + /// the match is discarded and the scanner is terminal from here on. + pub matched: Option, +} + +/// Scans assistant text for the client's `stop_sequences`, holding back only the +/// bytes that could still be the start of one. +#[derive(Debug, Clone, Default)] +pub(crate) struct StopScanner { + /// The configured stop strings, in the client's order (ties on the match + /// position are broken by this order). Never contains an empty string. + sequences: Vec, + /// The longest configured sequence in bytes; the holdback never exceeds + /// `max_len - 1`. + max_len: usize, + /// Text seen but not yet emitted, because it is a proper prefix of some + /// stop sequence. + holdback: String, + /// The sequence that matched, once one has. + matched: Option, +} + +impl StopScanner { + /// A scanner for `sequences`. Empty strings are dropped (they would match + /// everywhere); an empty list disables the feature entirely. + pub(crate) fn new(sequences: Vec) -> Self { + let sequences: Vec = sequences.into_iter().filter(|s| !s.is_empty()).collect(); + let max_len = sequences.iter().map(String::len).max().unwrap_or(0); + Self { + sequences, + max_len, + holdback: String::new(), + matched: None, + } + } + + /// Whether no stop sequence is configured — the caller's fast path, which + /// must stay byte-identical to the pre-#605 translation. + pub(crate) fn is_empty(&self) -> bool { + self.sequences.is_empty() + } + + /// The stop sequence that matched, if any. + pub(crate) fn matched(&self) -> Option<&str> { + self.matched.as_deref() + } + + /// Feed one assistant-text delta. Never call this once [`Self::matched`] is + /// `Some` — the translation stops the whole message at that point. + pub(crate) fn push(&mut self, delta: &str) -> StopScan { + self.holdback.push_str(delta); + if let Some((start, sequence)) = self.earliest_match() { + let emit = self.holdback[..start].to_string(); + self.holdback.clear(); + self.matched = Some(sequence.clone()); + return StopScan { + emit, + matched: Some(sequence), + }; + } + let keep = self.pending_prefix_len(); + let split = self.holdback.len() - keep; + let emit = self.holdback[..split].to_string(); + self.holdback.drain(..split); + StopScan { + emit, + matched: None, + } + } + + /// Release the held-back text. Called when the open text block closes for any + /// reason other than a match — a prefix that never completed is ordinary + /// output and the client must still receive it. + pub(crate) fn flush(&mut self) -> String { + std::mem::take(&mut self.holdback) + } + + /// The earliest occurrence of any stop sequence in the holdback, as a byte + /// offset. Earliest start wins; equal starts are broken by the configured + /// order (so `find`'s first hit at that offset is kept). + fn earliest_match(&self) -> Option<(usize, String)> { + let mut best: Option<(usize, &String)> = None; + for sequence in &self.sequences { + let Some(start) = self.holdback.find(sequence.as_str()) else { + continue; + }; + if best.is_none_or(|(best_start, _)| start < best_start) { + best = Some((start, sequence)); + } + } + best.map(|(start, sequence)| (start, sequence.clone())) + } + + /// Length in bytes of the longest holdback suffix that is a *proper* prefix + /// of some stop sequence, i.e. how much must stay buffered. Only char + /// boundaries are considered, so a multi-byte code point is never split. + fn pending_prefix_len(&self) -> usize { + let len = self.holdback.len(); + // A proper prefix is shorter than the sequence itself, so never more + // than `max_len - 1` bytes need holding back. + let earliest = len.saturating_sub(self.max_len.saturating_sub(1)); + for start in earliest..len { + if !self.holdback.is_char_boundary(start) { + continue; + } + let suffix = &self.holdback[start..]; + if self + .sequences + .iter() + .any(|sequence| sequence.len() > suffix.len() && sequence.starts_with(suffix)) + { + return len - start; + } + } + 0 + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn scanner(sequences: &[&str]) -> StopScanner { + StopScanner::new(sequences.iter().map(|s| (*s).to_string()).collect()) + } + + #[test] + fn an_empty_list_disables_the_scanner() { + assert!(scanner(&[]).is_empty()); + // An empty string would match everywhere; it is dropped, not honoured. + assert!(scanner(&[""]).is_empty()); + } + + #[test] + fn emits_text_before_a_match_and_discards_the_rest() { + let mut scanner = scanner(&[""]); + let scan = scanner.push("answergarbage"); + assert_eq!(scan.emit, "answer"); + assert_eq!(scan.matched.as_deref(), Some("")); + assert_eq!(scanner.matched(), Some("")); + } + + #[test] + fn holds_back_only_a_proper_prefix_across_deltas() { + let mut scanner = scanner(&[""]); + let first = scanner.push("hello tail"); + assert_eq!(second.emit, ""); + assert_eq!(second.matched.as_deref(), Some("")); + } + + #[test] + fn flush_releases_a_prefix_that_never_completed() { + let mut scanner = scanner(&[""]); + assert_eq!(scanner.push("abc "]); + let first = scanner.push("한글"); + assert_eq!(second.emit, ""); + assert_eq!(second.matched.as_deref(), Some("")); + } + + #[test] + fn the_earliest_match_wins_over_the_configured_order() { + // `` is listed first but occurs later, so `` wins. + let mut scanner = scanner(&["", ""]); + let scan = scanner.push("abc"); + assert_eq!(scan.emit, "a"); + assert_eq!(scan.matched.as_deref(), Some("")); + } + + #[test] + fn a_tie_on_position_is_broken_by_the_configured_order() { + let mut scanner = scanner(&[""]); + let scan = scanner.push("x"); + assert_eq!(scan.emit, "x"); + assert_eq!(scan.matched.as_deref(), Some(", +} + +impl Drop for TestGateway { + fn drop(&mut self) { + self.task.abort(); + } +} + +async fn start_gateway(config: Config) -> TestGateway { + let mut config = config; + config.server.bind = "127.0.0.1:0".to_string(); + let listener = TcpListener::bind(config.server.bind_addr().unwrap()) + .await + .unwrap(); + let addr: SocketAddr = listener.local_addr().unwrap(); + let (app, _shared, _state) = server::build_router(config).unwrap(); + let task = tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + TestGateway { + base_url: format!("http://{addr}"), + task, + } +} + +fn can_bind_loopback() -> bool { + match std::net::TcpListener::bind("127.0.0.1:0") { + Ok(listener) => { + drop(listener); + true + } + Err(error) if error.kind() == ErrorKind::PermissionDenied => { + eprintln!("skipping network integration test: loopback bind is not permitted"); + false + } + Err(error) => panic!("unexpected loopback bind failure: {error}"), + } +} + +/// A hand-rolled Responses upstream that answers with `text/event-stream`, emits +/// the stop sequence mid-text, and then *keeps writing* deltas until the socket +/// is torn down. Returns its base URL and the flag it sets once a write fails — +/// i.e. once shunt dropped the connection. +/// +/// A mock-server fixture cannot express this: it serves a finite body and the +/// abort has nothing to fail against. The write loop is bounded by a generous +/// deadline, which is the cost the *green* path pays (it exits on the first +/// failed write), not a window a racing failure has to beat. +async fn spawn_streaming_upstream() -> (String, Arc) { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let aborted = Arc::new(AtomicBool::new(false)); + let flag = Arc::clone(&aborted); + + tokio::spawn(async move { + let Ok((mut socket, _)) = listener.accept().await else { + return; + }; + // Read whatever of the request arrives; the body is irrelevant here. + let mut scratch = [0_u8; 4096]; + let _ = socket.read(&mut scratch).await; + + let head = concat!( + "HTTP/1.1 200 OK\r\n", + "content-type: text/event-stream\r\n", + "cache-control: no-cache\r\n", + "connection: close\r\n", + "\r\n", + ); + let opening = concat!( + "event: response.created\n", + "data: {\"response\":{\"id\":\"resp_1\"}}\n\n", + "event: response.output_item.added\n", + "data: {\"item\":{\"type\":\"message\"}}\n\n", + "event: response.output_text.delta\n", + "data: {\"delta\":\"answergarbage\"}\n\n", + ); + if socket.write_all(head.as_bytes()).await.is_err() + || socket.write_all(opening.as_bytes()).await.is_err() + || socket.flush().await.is_err() + { + return; + } + + // Keep generating past the stop. A live upstream would; the point of the + // abort is that shunt stops paying for it. + let more = concat!( + "event: response.output_text.delta\n", + "data: {\"delta\":\"still talking\"}\n\n", + ); + for _ in 0..200 { + if socket.write_all(more.as_bytes()).await.is_err() || socket.flush().await.is_err() { + flag.store(true, Ordering::SeqCst); + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + }); + + (format!("http://{addr}"), aborted) +} + +/// The counterpart upstream: no stop string in its text, a real +/// `response.completed`, and then a short tail of further events before it ends +/// the body itself. Returns its base URL, the flag it sets if a write fails — +/// i.e. if shunt tore the connection down — and the flag it sets once it reached +/// that natural end instead. +/// +/// The tail is what makes the two outcomes distinguishable: without bytes after +/// the terminal there is nothing for an abort to fail against. Its length is the +/// cost the *green* path pays, so it is short. +async fn spawn_completing_upstream() -> (String, Arc, Arc) { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let aborted = Arc::new(AtomicBool::new(false)); + let finished = Arc::new(AtomicBool::new(false)); + let abort_flag = Arc::clone(&aborted); + let finish_flag = Arc::clone(&finished); + + tokio::spawn(async move { + let Ok((mut socket, _)) = listener.accept().await else { + return; + }; + let mut scratch = [0_u8; 4096]; + let _ = socket.read(&mut scratch).await; + + let head = concat!( + "HTTP/1.1 200 OK\r\n", + "content-type: text/event-stream\r\n", + "cache-control: no-cache\r\n", + "connection: close\r\n", + "\r\n", + ); + let turn = concat!( + "event: response.created\n", + "data: {\"response\":{\"id\":\"resp_1\"}}\n\n", + "event: response.output_item.added\n", + "data: {\"item\":{\"type\":\"message\"}}\n\n", + "event: response.output_text.delta\n", + "data: {\"delta\":\"answer\"}\n\n", + "event: response.output_text.done\n", + "data: {}\n\n", + "event: response.completed\n", + "data: {\"response\":{\"usage\":{\"input_tokens\":3,\"output_tokens\":1}}}\n\n", + ); + if socket.write_all(head.as_bytes()).await.is_err() + || socket.write_all(turn.as_bytes()).await.is_err() + || socket.flush().await.is_err() + { + return; + } + + // Trailing SSE comments: the translation ignores them, so they only ever + // exercise the transport's read side. + for _ in 0..20 { + if socket.write_all(b": keep-alive\n\n").await.is_err() || socket.flush().await.is_err() + { + abort_flag.store(true, Ordering::SeqCst); + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + let _ = socket.shutdown().await; + finish_flag.store(true, Ordering::SeqCst); + }); + + (format!("http://{addr}"), aborted, finished) +} + +/// Collect the `text` of every `text_delta` frame in an Anthropic SSE stream. +fn streamed_text(sse: &str) -> String { + sse.lines() + .filter_map(|line| line.strip_prefix("data: ")) + .filter_map(|data| serde_json::from_str::(data).ok()) + .filter(|value| value["delta"]["type"] == "text_delta") + .filter_map(|value| value["delta"]["text"].as_str().map(str::to_string)) + .collect() +} + +/// The last non-empty `event:` name in an Anthropic SSE stream. +fn last_event(sse: &str) -> Option { + sse.lines() + .filter_map(|line| line.strip_prefix("event: ")) + .next_back() + .map(str::to_string) +} + +#[tokio::test] +async fn responses_stop_sequence_truncates_the_client_stream_and_aborts_the_upstream() { + if !can_bind_loopback() { + return; + } + let mut vars = common::env_lock().await; + vars.set("SHUNT_TEST_STOP_SEQUENCES_KEY", "sk-test"); + + let (upstream_url, upstream_aborted) = spawn_streaming_upstream().await; + + let mut config = Config::default(); + { + let openai = config.providers.get_mut("openai").unwrap(); + openai.base_url = upstream_url; + openai.api_key_env = Some("SHUNT_TEST_STOP_SEQUENCES_KEY".to_string()); + } + config.routes.push(RouteConfig { + model: "stop-sequence-model".to_string(), + provider: "openai".to_string(), + upstream_model: None, + effort: None, + service_tier: None, + }); + let gateway = start_gateway(config).await; + + let response = reqwest::Client::new() + .post(format!("{}/v1/messages", gateway.base_url)) + .header("content-type", "application/json") + .body( + serde_json::json!({ + "model": "stop-sequence-model", + "max_tokens": 64, + "stream": true, + "stop_sequences": [STOP], + "messages": [{"role": "user", "content": "classify"}] + }) + .to_string(), + ) + .send() + .await + .unwrap(); + assert_eq!(response.status(), reqwest::StatusCode::OK); + + let sse = tokio::time::timeout(Duration::from_secs(15), response.text()) + .await + .expect("the client stream must end at the stop, not at the upstream's") + .unwrap(); + + assert_eq!( + streamed_text(&sse), + "answer", + "only the text before the stop reaches the client; got: {sse}" + ); + assert!( + !sse.contains("garbage") && !sse.contains("still talking"), + "post-stop text leaked to the client: {sse}" + ); + assert!( + sse.contains("\"stop_reason\":\"stop_sequence\""), + "got: {sse}" + ); + assert!(sse.contains("\"stop_sequence\":\"\""), "got: {sse}"); + assert_eq!( + last_event(&sse).as_deref(), + Some("message_stop"), + "message_stop must be the last event; got: {sse}" + ); + + // The upstream's next write fails once shunt drops the connection. Poll for + // it rather than sleeping a fixed amount: the deadline is generous because + // it is only ever paid in full by a *failing* run. + let deadline = std::time::Instant::now() + Duration::from_secs(10); + while !upstream_aborted.load(Ordering::SeqCst) && std::time::Instant::now() < deadline { + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert!( + upstream_aborted.load(Ordering::SeqCst), + "shunt must drop the upstream connection at the stop instead of letting it keep generating" + ); +} + +/// The positive twin of the abort test: with no `stop_sequences` configured, a +/// turn that ends on a real `response.completed` must *not* have its upstream +/// torn down. Keying the abort on "the machine is stopped" rather than on the +/// emulated stop would drop the byte stream on every ordinary terminal, and a +/// reqwest body that never reaches EOF costs the connection its place in the +/// idle pool — on the overwhelming majority of turns, since `stop_sequences` is +/// usually unset. +#[tokio::test] +async fn responses_completed_turn_without_stop_sequences_keeps_the_upstream_connection() { + if !can_bind_loopback() { + return; + } + let mut vars = common::env_lock().await; + vars.set("SHUNT_TEST_STOP_SEQUENCES_KEY", "sk-test"); + + let (upstream_url, upstream_aborted, upstream_finished) = spawn_completing_upstream().await; + + let mut config = Config::default(); + { + let openai = config.providers.get_mut("openai").unwrap(); + openai.base_url = upstream_url; + openai.api_key_env = Some("SHUNT_TEST_STOP_SEQUENCES_KEY".to_string()); + } + config.routes.push(RouteConfig { + model: "stop-sequence-model".to_string(), + provider: "openai".to_string(), + upstream_model: None, + effort: None, + service_tier: None, + }); + let gateway = start_gateway(config).await; + + let response = reqwest::Client::new() + .post(format!("{}/v1/messages", gateway.base_url)) + .header("content-type", "application/json") + .body( + serde_json::json!({ + "model": "stop-sequence-model", + "max_tokens": 64, + "stream": true, + "messages": [{"role": "user", "content": "classify"}] + }) + .to_string(), + ) + .send() + .await + .unwrap(); + assert_eq!(response.status(), reqwest::StatusCode::OK); + + // The client stream ends at the terminal frame rather than waiting on the + // upstream drain (`translated_core` ends the outward stream immediately and + // hands the remaining bytes to a detached `spawn_terminal_drain`, bounded by + // `TERMINAL_DRAIN_BUDGET`, purely so the connection can still be pooled — see + // its own `a_keepalive_ping_never_follows_the_terminal_frame` test). So + // `response.text()` returning here proves nothing about the upstream socket + // yet; poll for it to reach its own natural end within that same budget + // instead of asserting on it synchronously. + let sse = tokio::time::timeout(Duration::from_secs(15), response.text()) + .await + .expect("the client stream must end at the terminal frame") + .unwrap(); + + assert_eq!(streamed_text(&sse), "answer", "got: {sse}"); + assert!(sse.contains("\"stop_reason\":\"end_turn\""), "got: {sse}"); + assert_eq!( + last_event(&sse).as_deref(), + Some("message_stop"), + "message_stop must be the last event; got: {sse}" + ); + + let drained = tokio::time::timeout(Duration::from_secs(5), async { + while !upstream_finished.load(Ordering::SeqCst) && !upstream_aborted.load(Ordering::SeqCst) + { + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await; + assert!( + drained.is_ok(), + "the detached drain must resolve the upstream to a terminal state within its budget" + ); + assert!( + !upstream_aborted.load(Ordering::SeqCst), + "an ordinary terminal must not abort the upstream: the body has to drain to EOF for the \ + connection to stay poolable" + ); + assert!( + upstream_finished.load(Ordering::SeqCst), + "the upstream must have reached its own end of body" + ); +}