[gdevenyi]

Problem and result

When the engine keeps refusing a slot after a streamed request has got its 200, the server writes a second HTTP response into the SSE body. I reproduced it with the repo's fake engine process, nothing patched: the BUSYALL prompt with 2 slots, the trigger of test_bgen_refused_for_good_is_a_retryable_503_and_frees_the_slots.

Before, the raw chat body ended like this (line endings shown as line breaks):

data: {... "delta": {"role": "assistant", "content": ""}, "finish_reason": null}]}

HTTP/1.0 503 Service Unavailable
Server: BaseHTTP/0.6 Python/3.12.3
Date: Sat, 31 Oct 2026 12:47:12 GMT
Content-Type: application/json
Retry-After: 5
Content-Length: 120
Connection: close

{"error": {"type": "server_error", "message": "the engine's batch slots are not free right now; try again in a moment"}}

The messages and responses bodies ended with the same block. The openai and anthropic SDKs raised nothing, and the caller saw an empty turn.

Any other exception in a started stream ended the body with no last frame at all. I showed this only by injecting a RuntimeError into MockEngine.

After this change, both cases end with the frame the stream loops already use for other errors:

• Chat: an error chunk, then [DONE].
• Messages: an error event, overloaded_error for GpuBusy and api_error for the others.
• Responses: response.failed.

The SDKs now see the failure. openai chat raises APIError, anthropic raises APIStatusError, and the openai responses stream ends on response.failed.

What the specs and this repo say:

• OpenAI openapi.yaml, chat streaming: "If a failure occurs after streaming has started, a data frame may contain a JSON object with an error field instead of a completion chunk."
• Anthropic, platform.claude.com/docs/en/api/errors: "When receiving a streaming response over server-sent events (SSE), an error can occur after the API returns a 200 response. In that case, error handling doesn't follow these standard mechanisms."
• The _responses docstring in this repo: "once the stream has started they arrive as a response.failed event."

Related discussion

No scope agreement was sought: this is a bug fix and no default changes.

• [Engine] answer an unexpected handler error with a 500, not a dropped connection #4757 (open) handles a handler error only when answer_started is false and re-raises otherwise, so a started stream still gets no frame.
• [Other] add legacy /v1/completions for raw-prompt clients (#2277) #3332 (open) adds EngineStuck to the chat stream's except list, but not GpuBusy, and adds nothing to messages or responses. It touches the same lines, so one of the two may need a rebase.
• [Engine] serve: a failed engine start no longer crashes when another thread closed the process #4792, [Engine] serve: a stalled peer does not hold a handler or the engine's turn #4759 and [Engine] serve: an engine program that cannot be started is a 503, not a dropped connection #2497 cover restart cleanup, a socket deadline and a spawn OSError before the stream. None covers this.

Approach

Only the stream loops of _openai, _anthropic and _responses change, plus three small helpers. The new except branches come after the existing ones. The _post handlers, the non-streaming paths and errors before the stream are not touched, and a stream that raises nothing never reaches the new code.

• GpuBusy, EngineStuck, EngineStarting and ModelBusy send their own message in the frame.
• Any other exception sends a plain sentence that names no exception type, and its traceback goes to the server log.
• ClientGone is re-raised, because it means the client left. OSError keeps its existing branch.
• If the frame cannot be written on chat or messages, the request's outcome is disconnected. Its state stays error, because a server error did happen. The responses path keeps _responses_failed, which ignores that write failure.

Validation

Revision: c62d5e3 plus this one commit. Linux, Python 3.12.3, openai 3.42.0, anthropic 2.3.0. The unit tests, the differential and the fake-engine SDK checks use MockEngine and the repo's fake engine process, with no GPU. The real-engine runs are marked as such below.

Passed:

• New tests, with this change:
  python -m unittest serve.test_stream_errors serve.test_parallel.ParallelService.test_bgen_refused_for_good_in_a_stream_is_each_apis_error_frame
  ran 6 tests, OK.

• The same command on c62d5e3, with serve/test_stream_errors.py and serve/test_parallel.py copied in: FAILED (failures=10, errors=1), counting subtests. The error is one test calling a helper that does not exist there. The two tests that pass there are guards: a client that left gets no frame, and a clean stream ends as before.

• Test set:
  python -m unittest serve.test_server serve.test_responses serve.test_anthropic_base_url serve.test_chat_tool_results serve.test_context_overflow serve.test_images serve.test_stream_errors serve.test_parallel serve.test_quiet_cancel serve.test_request_accounting serve.test_fatal_recovery serve.test_queue_keepalive
  ran 506 tests in 188 s: FAILED (failures=1, skipped=1). The one failure is listed below.

• Differential against c62d5e3, a local script that is not in this PR. It sends 13 requests through MockEngine: 10 normal ones across the three APIs, stream and non-stream, and 3 streamed ones refused with a 400 before the stream. With ids, timestamps and timings normalized and Date, Server and Content-Length left out, status, headers and bodies were byte-identical.

• SDK checks, local scripts that are not in this PR, on both trees:

  • Fake engine BUSYALL, nothing patched. Before: no exception from any SDK. After: APIError, APIStatusError with overloaded_error, and events response.created, response.in_progress, response.failed.
  • Injected RuntimeError. Before: no exception from any SDK. After: APIError, APIStatusError with api_error, and a stream ending on response.failed.

• Real engine, nothing patched. RTX 5080 16 GB, engine 0.1.55 from release v0.2.9 (the tarball's sha256 matches the release's SHA256SUMS.txt), Qwen3.8-Flash-Next IQ3_S, context 65536. Before is c62d5e3, after is 6f201ab, on the same day.

  The trigger: the config sets min_free_vram_mib to 15800 and another program holds 300 MiB of VRAM. Request A is running and request B is queued with its 200 already sent when the engine gets kill -9. On B's turn ensure_loaded finds too little free VRAM and raises GpuBusy. Each API ran once with a raw client and once with its SDK, so 6 kills on each side:

  B on	Raw body before	Raw body after	SDK before	SDK after
  Chat	ends with an HTTP/1.0 503 block	error chunk, [DONE], no HTTP block	no exception	APIError with the GPU message
  Messages	ends with an HTTP/1.0 503 block	error event, overloaded_error, no HTTP block	no exception	APIStatusError, type overloaded_error
  Responses	ends with an HTTP/1.0 503 block	response.failed, no HTTP block	no exception, events stop at response.in_progress	no exception, stream ends on response.failed with code server_error and the GPU message

  The Responses stream does not raise: the SDK hands response.failed to the caller as an event.

  On both sides, all 6 times: A ended with its existing error frame, and a normal request right after reloaded the engine and returned 200 with finish reason stop. With this change the server log has no traceback from these runs.

  Normal streams on both sides, before the kills and after them: the 3 SDKs and 3 raw clients, 12 streams per side. Chat ended on stop and [DONE], Messages on message_stop, Responses on response.completed, and no raw body held an HTTP block.

• Real engine, --parallel 2, after only: 488 streams with half or more dropped mid-body (details under Not run). Of the 182 streams read to the end, none held an HTTP block or a refusal message, and the 120 short chat ones ended on [DONE]. I did not check the last frame of the 62 long ones. 6 normal streams after each of the 2 runs ended properly, 12 in all.

Failed:

• serve.test_server.GpuChoice.test_hip_ordinal: AssertionError: '0,1' != '1,0'. It fails the same way on c62d5e3, so this change does not cause it.

Not run:

• The slot refusal on the real engine. I found no real trigger, so the BUSYALL result above comes from the fake engine only. On 6f201ab with --parallel 2, I sent 240 short streams with half dropped, then 248 long streams on all three APIs with 186 dropped mid-generation. The log shows 0 refused slots. It takes more than 6 refusals for one request to reach this path. These runs were not repeated on c62d5e3, since nothing was refused to compare.
• EngineStuck on the real engine. It has no real trigger. EngineStarting and ModelBusy were not reached on the real engine either.
• Any other exception on the real engine. It takes fault injection, which is a patch.
• More than one real-engine kill per API and client, on either side.
• Windows and macOS.
• CUDA, HIP, SYCL and Metal builds. This change is Python only.

Limits

• Both triggers I reproduced are uncommon. The slot refusal, on the fake engine only, needs --parallel of 2 or more and more than 6 refusals for one request. The real-engine one needs an engine crash with a request queued behind it, and too little free VRAM for the restart.
• GpuBusy from ensure_loaded inside Service.run is confirmed by the real-engine runs. Unconfirmed static analysis: EngineStuck can come from there too.
• Unconfirmed static analysis: I found no raise of EngineStarting or ModelBusy after the stream starts. Listing them is defensive only.
• The "any other exception" half was shown only by fault injection. For chat it is a weaker spec claim, because the same OpenAI passage says "An interrupted stream may end without the marker."
• Retry-After cannot follow a 200, so that wait does not reach a streaming client. The slot refusal's message does say "try again in a moment".
• On Messages, GpuBusy is now overloaded_error inside a stream. Before the stream it is still a 503 with type server_error, as it was. This PR does not change that, so the two disagree.
• This does not claim to fix [Engine] serve: a new agent request answers empty in 0.2 s (200, 1 chunk, no finish_reason) while the engine never logs it; /health stays ok — Linux Docker, 0.2.3, engine 0.1.48 #3961. That report has 40 bytes and a clean server log, while the injected path prints a traceback and writes more bytes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W53iURzvDQMjnYemag8uvv