Skip to content

Retry builder chunk-store timeouts as infrastructure failures - #67

Merged
rlrs merged 2 commits into
mainfrom
fix/builder-infrastructure-retries
Nov 7, 2026
Merged

rlrs merged 2 commits into
mainfrom
fix/builder-infrastructure-retries

Conversation

@rlrs

@rlrs rlrs commented Nov 7, 2026

Copy link
Copy Markdown
Owner

On 2026-11-07, five builders converting at once kept the store node's chunk index past the builders' 60 s client timeout. 50 builds failed with chunk store request failed: ReadTimeoutError. The gateway counted each one as a recipe failure: 10 minutes between attempts, and a recipe failed for good after 3. Nothing was wrong with any of those recipes.

  • Gateway (image_recipes): errors that start with the chunk store client's own transport-failure text (chunk store request failed: or chunk store request deadline exceeded, now constants in chunk_index) are infrastructure failures, like sandbox builds' sandbox build infrastructure:. They retry after 30 s, doubling, up to 6 attempts. The index summary counts them the same way. Only the gateway changes; builders keep reporting what they already report.
  • Chunk index server: a client that hung up before its reply used to surface as a broken pipe inside the 502 path and was logged as "object store unavailable". Now that case is logged as the client leaving, with the endpoint and the duration. Requests slower than 10 s are logged with their endpoint, which we need before we can make the index itself faster.
  • Test: the infrastructure-retry test now runs for both builder error forms. It fails without the fix, because the second attempt is never sent.
  • Docs: docs/images.md retry rules and a CHANGELOG.md entry.

The broken-pipe and slow-request changes only affect logging, so there's no test for them.

Production: separately from this PR, builder.max_finishing_builds is now 1 (was 2), which halves concurrent conversions per builder. It takes effect as builders are replaced.

Checks: full suite with PostgreSQL (2441 tests: 2426 pass, 15 skipped), ruff, and the test-rule check all pass.

🤖 Generated with Claude Code

rlrs and others added 2 commits November 7, 2026 17:18
A builder build whose conversion timed out reaching the chunk index (or
S3) counted as a recipe failure: 10 minutes between attempts and failed
for good after 3. The gateway now retries the chunk store client's own
transport failures like sandbox builds' infrastructure failures.

The chunk index logs requests slower than 10 s, and a client that left
before its reply is no longer logged as "object store unavailable".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rlrs
rlrs merged commit 29b17b4 into main Nov 7, 2026
4 checks passed
@rlrs rlrs mentioned this pull request Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant