fix(buzz-agent): escalate LLM timeouts per retry and log per-call latency - #5130
Open
wpfleger96 wants to merge 1 commit into
Open
fix(buzz-agent): escalate LLM timeouts per retry and log per-call latency#5130wpfleger96 wants to merge 1 commit into
wpfleger96 wants to merge 1 commit into
Conversation
…ency Non-streaming calls to slow models (claude-fable via the Databricks gateway) routinely exceed the fixed 240s read timeout before the first response byte arrives, so every retry re-ran an identical losing bet and turns black-holed for 30+ minutes. Escalate the per-request budget 2x after each timeout failure (capped at max(1200s, base)), default unset timeouts to 600s for known slow-generation models, and emit one INFO line per completed LLM call (duration + token usage incl. cache reads) so slowness is visible before it becomes failure. Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Non-streaming LLM calls (
"stream": false) through slow model/provider combinations — Claude Fable 5 via the Databricks AI gateway in particular — routinely take longer than the fixed 240 s client timeout to return their first response byte. The retry loop then re-ran the identical 240 s bet three times, failed the turn, and the ACP harness requeued the whole turn from scratch: agents spent 30+ minutes producing nothing while every attempt died at the same wall. And because the LLM path only logged WARN lines on failure, a healthy-but-slow call was indistinguishable from a wedged one.Timeout handling
base × 2^n, capped atmax(1200 s, base)—escalated_timeout()inllm.rs), shared by the mainpost()loop andopenrouter_post(). Non-timeout retryables (429/5xx/connect) do not escalate. A call that needs six minutes now succeeds on the second attempt instead of never.BUZZ_AGENT_LLM_TIMEOUT_SECSis unset, known slow-generation models (currently theclaude-fablefamily) default to 600 s instead of 240 s (Config::effective_llm_timeout()). An explicit env value remains authoritative for every model. Handoff summarization stays on the generic default — small responses — with escalation as the backstop.read_timeouttoRequestBuilder::timeout()on each LLM request, so escalated budgets aren't silently floored by the shared client and each attempt's bound covers connect through body completion. Timeout error messages were updated to match the new semantics and still point atBUZZ_AGENT_LLM_TIMEOUT_SECS.Observability
duration_ms,input_tokens,cached_input_tokens,output_tokens. Slowness and prompt-cache effectiveness are now visible in harness logs without waiting for a failure, andNonevs0token reports stay distinguishable.session_idtracing span so each line is attributable to a session.