Skip to content

fix(buzz-agent): escalate LLM timeouts per retry and log per-call latency - #5130

Open
wpfleger96 wants to merge 1 commit into
mainfrom
wpfleger/llm-timeout-escalation-logging
Open

fix(buzz-agent): escalate LLM timeouts per retry and log per-call latency#5130
wpfleger96 wants to merge 1 commit into
mainfrom
wpfleger/llm-timeout-escalation-logging

Conversation

@wpfleger96

Copy link
Copy Markdown
Member

Non-streaming LLM calls ("stream": false) through slow model/provider combinations — Claude Fable 5 via the Databricks AI gateway in particular — routinely take longer than the fixed 240 s client timeout to return their first response byte. The retry loop then re-ran the identical 240 s bet three times, failed the turn, and the ACP harness requeued the whole turn from scratch: agents spent 30+ minutes producing nothing while every attempt died at the same wall. And because the LLM path only logged WARN lines on failure, a healthy-but-slow call was indistinguishable from a wedged one.

Timeout handling

  • Per-attempt escalation: the per-request budget doubles after each timeout failure (base × 2^n, capped at max(1200 s, base)escalated_timeout() in llm.rs), shared by the main post() loop and openrouter_post(). Non-timeout retryables (429/5xx/connect) do not escalate. A call that needs six minutes now succeeds on the second attempt instead of never.
  • Model-aware default: when BUZZ_AGENT_LLM_TIMEOUT_SECS is unset, known slow-generation models (currently the claude-fable family) default to 600 s instead of 240 s (Config::effective_llm_timeout()). An explicit env value remains authoritative for every model. Handoff summarization stays on the generic default — small responses — with escalation as the backstop.
  • Per-request total timeouts: enforcement moved from the client-level read_timeout to RequestBuilder::timeout() on each LLM request, so escalated budgets aren't silently floored by the shared client and each attempt's bound covers connect through body completion. Timeout error messages were updated to match the new semantics and still point at BUZZ_AGENT_LLM_TIMEOUT_SECS.

Observability

  • One INFO line per completed LLM call: model, provider, duration_ms, input_tokens, cached_input_tokens, output_tokens. Slowness and prompt-cache effectiveness are now visible in harness logs without waiting for a failure, and None vs 0 token reports stay distinguishable.
  • The agent main loop wraps the call in a session_id tracing span so each line is attributable to a session.

…ency

Non-streaming calls to slow models (claude-fable via the Databricks
gateway) routinely exceed the fixed 240s read timeout before the first
response byte arrives, so every retry re-ran an identical losing bet and
turns black-holed for 30+ minutes. Escalate the per-request budget 2x
after each timeout failure (capped at max(1200s, base)), default unset
timeouts to 600s for known slow-generation models, and emit one INFO
line per completed LLM call (duration + token usage incl. cache reads)
so slowness is visible before it becomes failure.

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 requested a review from a team as a code owner August 7, 2026 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant