API - Translation issues / Round-Robin between models from different Upstream API Styles

[weselben]

Summary

When a Virtual Model in GoModel is configured to round-robin across upstreams that expose different API protocols (OpenAI Chat Completions vs. OpenAI Responses API), a chat session cannot be continued once the upstream swap crosses a protocol boundary.

• ✅ Requests that land on the same protocol type work fine.
• ❌ As soon as round-robin switches from one protocol to another, continuing the conversation fails — the previous conversation state is not portable between the two APIs.

Use Case / Motivation

Virtual Models are a great way to combine capacity from multiple providers behind a single model name. A typical setup:

Upstream	Protocol	Example model
kimicode	Chat Completion	k3
chatgpt	Responses API	gpt-5.6-terra
zai	Chat Completion	glm-5.3-flash

Scenario: A client (e.g. Kimi Code CLI) sends an OpenAI Chat Completion request to the Virtual Model. GoModel round-robins between the three upstreams above. Requests hitting kimi-code or zai work as expected — but as soon as the router lands on chatgpt (Responses API), the conversation can no longer be continued.

Observed behavior

1. Send request N → routed to Chat Completion upstream → response OK.
2. Send request N+1 → routed to Responses API upstream → failure / broken context.
3. From then on, the session is effectively unusable for that virtual model.

Root cause (assumption): Chat Completions and the Responses API represent conversation state differently (messages array vs. response/previous_response_id chain). When GoModel swaps upstreams mid-conversation, state from one protocol cannot be replayed into the other, and the client has no way to know a protocol switch occurred.

Expected behavior (options)

• 🅰️ GoModel translates conversation state between protocols on upstream swap (messages ↔ response chain), so mixed-protocol round-robin works transparently.
• 🅱️ GoModel detects protocol incompatibility and pins a conversation (by session/thread identifier) to a protocol-compatible upstream subset.
• 🅲️ Document this as a known limitation and allow Virtual Models to be restricted to a single protocol type across all upstreams (validation/warning at config time).

Test plan

Detailed test matrix will follow later today — mapping client type × client protocol × upstream protocol and whether the conversation survives upstream swaps. Planned coverage:

• Kimi Code CLI → Virtual Model (mixed round-robin)
• Raw OpenAI Chat Completions client → Virtual Model
• Responses API client → Virtual Model
• Behavior per load-balancing strategy (round_robin, least_latency, …)

Environment

• GoModel version: (to be added)
• Deployment: (to be added)
• Config snippet (virtual model + upstreams): (to be added)

---

Root cause (confirmed by code walk on main @ 2c0f662)

The router is protocol-blind: viableTargets (internal/virtualmodels/chain.go:49) only checks catalog availability, never the upstream's API protocol. Session affinity is default-on (internal/virtualmodels/types.go:65-67), but only engages if a session id is detected (internal/virtualmodels/balancer.go:119-121) — Kimi Code CLI sends none of the built-in session headers (internal/session/session.go:47), and content auto-detection (internal/session/detect.go:59,128, default on per config/config.go:212) needs a request body ≤ 1 MiB (internal/auditlog/constants.go:7). No detected id → per-request rotation crosses the protocol boundary.

The boundary fails in both directions:

• Chat client → chatgpt: no Chat→Responses request translator exists anywhere in the codebase. chatgpt.ChatCompletion/StreamChatCompletion return 501 unsupported_provider_operation (internal/providers/chatgpt/chatgpt.go:159-166).
• Responses client with previous_response_id → chat-translated upstream: hard 400 in the shared adapter (internal/providers/responses_adapter.go:91-93). The local response store is never consulted on the request path to expand previous_response_id into replayable input.
• Failover worsens it: default retry statuses 429,5xx (internal/gateway/failover_policy.go:11) — the 501 above is a 5xx, so ShouldRetry sweeps the failover chain into the incompatible target.

The only portable state model is the gateway-managed conversation field (internal/server/conversation_responses.go:15-21), which can't combine with previous_response_id (:53). Full write-up and option analysis in the investigation comment below.

Suggested direction (from investigation)

Option B (pin conversations to a protocol-compatible subset) is the most contained fix in this codebase: the pinning machinery (stickySessions) and provider-protocol info (catalog.GetProviderType) already exist; the gap is precisely that viableTargets/balancedResolution ignore protocol — filtering the viable pool by client endpoint protocol before strategy selection is a contained change at internal/virtualmodels/chain.go:49 / internal/virtualmodels/balancer.go:56. Option A (state translation on swap) is feasible but requires a brand-new Chat→Responses request translator plus wiring the response store into the request path — much larger surface. Option C is cheap and worth shipping alongside B as a config-time warning.

---

Wayfinder map (the wayfinding view of this issue)

Destination

Any client protocol (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages) can reach any upstream protocol through GoModel, so virtual-model round-robin and failover work across protocol boundaries. This issue's symptom — a conversation breaking when the router crosses the Chat Completions / Responses boundary — disappears because the protocol matrix is complete and translation is loss-bounded and explicit.

Notes

• Domain: protocol translation providers (internal/providers), the dispatch path (internal/gateway/inference_execute.go), and the virtual-model balancer (internal/virtualmodels).
• Every session consults the existing translator inventory in internal/providers/responses_adapter.go and the streaming converters.
• Architecture decisions are settled; open decisions are ticketed below as child issues.
• Work only in this repo (weselben/GoModel); no PRs to any other upstream.

Decisions so far

• Investigation: confirmed root cause (protocol-blind router + missing chat→responses translation + previous_response_id rejection) and the failure shapes per direction — see the investigation comment for the full trace and file:line citations.
• Architecture: extend the existing provider-adapts pattern rather than add a central gateway gate. Providers already translate their own dialect; the chat↔responses pair is the only incomplete pair. No new dispatch layer.
• Topology: chat completions is the hub (star). Not a canonical internal message representation.
• Slice shape: translators first (pure additions), then chatgpt wiring (replaces its 501s), then previous_response_id expansion from the local response store (orthogonal, separate decision).
• Verify chatgpt stream-parser reuse and lock translator signatures — collapseResponsesStream not reusable for streaming (non-streaming collapse); five new functions locked with exact signatures (ConvertChatRequestToResponses, ConvertResponsesResponseToChat, ChatResponsesStreamConverter, ChatViaResponses, StreamChatViaResponses); responses→chat needs index-tracked call_id patching (id arrives late on response.output_item.done).
• Decide follow-up scope: protocol-aware pool filtering and config-time warning — defer same-protocol filtering (pure optimization after the matrix); no config-time mixed-protocol warning, warn only on missing directed edges.
• Lock the chat↔responses translation-loss policy — the ADR-0011 field-forwarding ladder already locks it; initial per-field rows for the chat↔responses pair recorded in the ticket.
• kimicode speaks the Responses API natively (draft PR ENTERPILOT/GoModel#916); live probe confirms kimi's native /responses is stateless (store:true and previous_response_id both 400), so gateway-managed state is the only chaining path even against responses-speaking upstreams.

Not yet specified

• Documentation updates (docs/features/virtual-models.mdx, docs/features/failover.mdx) once the loss policy is locked.

Out of scope

• A canonical internal message model / full protocol IR redesign.
• Migrating the ~10 provider-local ResponsesViaChat wrappers onto shared translators (cleanup only; behavior-preserving, deferred).
• Decide scope: session-detection gap for bodies over 1 MiB — affinity-only concern once the matrix is complete; separate effort.

[weselben]

🔬 Test Results: Client ↔ Protocol ↔ Upstream compatibility

Tested Kimi Code CLI against a GoModel Virtual Model, manually pinning different upstreams to isolate the behavior per combination.

✅ Test 1 — Chat Completion → Chat Completion

KimiCLI (Chat Completion) → GoModel → ZAI (Chat Completion, glm5.3-flash)

✔️ Works fine. Requests and conversation flow without issues.

---

❌ Test 2 — Responses API client → Chat Completion upstream

KimiCLI (Responses API) → GoModel → ZAI (Chat Completion)

✖️ Fails with:

The max_tokens parameter is illegal.：限制数值范围[1,131072]

It looks like GoModel passes through a max_tokens value that ZAI's Chat Completion endpoint rejects (out of range [1, 131072]). Possibly a parameter translation issue between the Responses API request and the Chat Completion upstream — e.g. max_output_tokens not being clamped/remapped when converting to max_tokens.

---

❌ Test 3 — Responses API client → Responses API upstream

KimiCLI (Responses API) → GoModel → ChatGPT (Responses API, gpt-5.6-terra)

✖️ Fails with:

invalid responses input item: role is required

All protocols match here, so GoModel should be able to forward as-is. The error suggests the input items KimiCLI sends don't carry a role field in the shape the upstream expects. This may actually be a KimiCLI issue rather than GoModel — needs verification with a direct client (curl) against the same upstream to isolate.

---

❌ Test 4 — Chat Completion client → Responses API upstream

KimiCLI (Chat Completion) → GoModel → ChatGPT (Responses API)

✖️ Fails with:

chatgpt serves only the Responses API; chat completions are not available on a ChatGPT subscription

Expected — the ChatGPT upstream doesn't expose Chat Completions at all, so a Chat Completion client can never reach it. GoModel correctly surfaces the upstream's refusal rather than translating.

---

📋 Summary table

#	Client	Client protocol	GoModel	Upstream	Upstream protocol	Result
1	KimiCLI	Chat Completion	→	ZAI (glm5.3-flash)	Chat Completion	✅ OK
2	KimiCLI	Responses API	→	ZAI	Chat Completion	❌ max_tokens illegal (range [1,131072])
3	KimiCLI	Responses API	→	ChatGPT (gpt-5.6-terra)	Responses API	❌ role is required (possibly KimiCLI bug)
4	KimiCLI	Chat Completion	→	ChatGPT	Responses API	❌ upstream only serves Responses API

Key takeaways

• Only same-protocol combinations work today (Test 1). Cross-protocol combinations fail either at the upstream (Test 4) or during parameter/message translation (Test 2).
• Test 2 points to a fixable GoModel translation bug: max_tokens/max_output_tokens handling between Responses and Chat Completion requests.
• Test 3 needs a control test (direct Responses API call via curl) to determine whether it's KimiCLI or GoModel.
• This confirms the main issue: with a mixed-protocol round-robin Virtual Model, only the upstreams matching the client's protocol can ever succeed — the rest will always error on at least one hop.

[weselben]

Root cause (code walk on main @ 2c0f662)

Three independent facts in the codebase combine to produce this. The issue's hypothesis is correct, and the code shows exactly where each half breaks.

1. Target selection is protocol-blind. internal/virtualmodels/chain.go:49 (viableTargets) filters a redirect's targets only by catalog availability (catalog.ModelAvailable, internal/virtualmodels/chain.go:67). The API protocol a target speaks — Chat Completions vs. Responses — is never consulted at any point in balancedResolution (internal/virtualmodels/balancer.go:56). A round-robin pool mixing chatgpt (Responses-only) with chat-completions providers is treated as one homogeneous pool.

2. Session affinity exists but cannot prevent the crossing. Affinity is on by default (internal/virtualmodels/types.go:65-67) and pins a detected session to the target that first served it (internal/virtualmodels/sticky.go, 6h TTL), with the first request's strategy pick deciding the pin (internal/virtualmodels/balancer.go:163-171). But it only engages when a session id is detected (internal/virtualmodels/balancer.go:119-121). Detection runs built-in header/body rules (internal/session/session.go:47) — Kimi Code CLI sends none of the covered headers — then falls back to content auto-detection (internal/session/detect.go:59, contentSessionID at internal/session/detect.go:128, default on per config/config.go:212), which needs the captured request body and is capped at 1 MiB (internal/auditlog/constants.go:7). No detected id means sessionID == "" and every request rotates through the full pool (internal/virtualmodels/balancer.go:176-178), crossing the protocol boundary. Even with a pin, the capacity fallback and re-pin paths (internal/virtualmodels/balancer.go:63-70) re-enter strategy selection without any protocol notion.

3. The boundary itself fails in both directions, for different reasons:

• Chat client → Responses-only upstream: there is no Chat→Responses request translation anywhere in the codebase. The chatgpt provider serves only the Responses surface; ChatCompletion/StreamChatCompletion return 501 unsupported_provider_operation (internal/providers/chatgpt/chatgpt.go:159-166). So a /v1/chat/completions request that round-robins onto the chatgpt leg fails outright rather than being adapted.
• Responses client with previous_response_id → chat-translated upstream: the shared adapter hard-rejects the field with a 400 (internal/providers/responses_adapter.go:91-93); conversation is rejected the same way (:95-97). The gateway's local response store is never consulted to expand previous_response_id into replayable input — it only persists snapshots asynchronously for the /v1/responses/{id} lifecycle endpoints (internal/server/translated_inference_service.go:430). The state chain therefore lives upstream and cannot follow a rotation.
• The one portable state model is the gateway-managed conversation field: resolved locally, merged into input, "works uniformly for native Responses providers and chat-translated ones" (internal/server/conversation_responses.go:15-21) — but it cannot be combined with previous_response_id (internal/server/conversation_responses.go:53), and it is gateway-local state, not client-carried state.
• Note also that the chatgpt leg itself (Codex backend) rejects previous_response_id outright and pins store: false (internal/providers/chatgpt/request.go:9-11, :43-44), so even native Responses chaining cannot ride that upstream.

Failover compounds it: the default retry statuses are 429,5xx (internal/gateway/failover_policy.go:11), the 501 above is a 5xx, so ShouldRetry (internal/gateway/failover_policy.go:51-86) sweeps the failover chain straight into the protocol-incompatible target instead of stopping.

Most feasible option: B (pin conversations to a protocol-compatible subset). The pinning machinery already exists (stickySessions), the protocol information already exists (catalog.GetProviderType and provider registration know exactly which surfaces each upstream serves), and the gap is precisely that viableTargets/balancedResolution ignore it — filtering the viable pool by the client endpoint's protocol before strategy selection is a contained change at internal/virtualmodels/chain.go:49 / internal/virtualmodels/balancer.go:56. Option A (translate state on swap) would need a brand-new Chat→Responses request translator plus wiring the response store into the request path to expand previous_response_id — the streaming half of that translation exists (internal/providers/responses_converter.go), the request half does not — so it is a much larger surface. Option C is cheap and worth shipping alongside B as a config-time warning, but on its own it forecloses the mixed-capacity use case this issue is about.

What I could not confirm: I traced these paths in code but did not run a live reproduction against real upstreams, so the 501/400 failure shapes are from code reading, not an executed test matrix. I also don't know the reporter's exact config (strategy, session_affinity, session-detection outcome for Kimi Code CLI specifically), so I can't say which affinity-bypass path applied in their case — no detected session id, disabled affinity, or the capacity/re-pin fallback all end at the same protocol-blind rotation.

[weselben]

Session intel: kimicode + protocol matrix

Upstream change (draft PR: ENTERPILOT#916): kimicode now speaks the Responses API natively (override of Responses/StreamResponses on the provider; chat path untouched). Live probe of https://api.kimi.com/coding/v1/responses (2026-09-08):

• Non-streaming and streaming Responses both work upstream.
• store:true → 400; previous_response_id → 400. The upstream is stateless even on the native Responses endpoint.

Impact on this issue's model table: the kimicode row moves from Chat Completion-only to dual-protocol. A Responses client can now hit kimicode natively; the remaining failure is still chat client → Responses-only upstream (chatgpt), which is exactly the chat→responses edge in #87.

Intel for the state question: both protocol styles among current targets (chat upstreams, and now kimicode's native Responses) retain no server-side state. Gateway-managed state (conversation field, conversationstore) is therefore the only path for previous_response_id chaining — confirms the map's orthogonal slice, and it stays required even after the matrix is complete.

Discussion decisions this session (not code):

1. The chat→responses pair (Verify chatgpt stream-parser reuse and lock translator signatures #87) is the correctness fix; the router stays protocol-blind on purpose once the edge exists.
2. Protocol-aware pool filtering is not needed for correctness. Defer as optional same-protocol preference (tracked in Decide follow-up scope: protocol-aware pool filtering and config-time warning #88).
3. Failover must not sweep a 501 unsupported_provider_operation into protocol-incompatible hops: retry statuses (429,5xx) currently do. This remains open and is not ticketed yet.
4. Loss budget stays explicit per edge (logprobs on chat→responses; state fields on any down-convert). No god-IR; translators are pure functions per directed edge.

Nothing here changes the map topology: provider-adapts, star on chat, translators as pure additions.