Repository navigation
Retry builder chunk-store timeouts as infrastructure failures - #67
Merged
Merged
Conversation
A builder build whose conversion timed out reaching the chunk index (or S3) counted as a recipe failure: 10 minutes between attempts and failed for good after 3. The gateway now retries the chunk store client's own transport failures like sandbox builds' infrastructure failures. The chunk index logs requests slower than 10 s, and a client that left before its reply is no longer logged as "object store unavailable". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cture-retries # Conflicts: # CHANGELOG.md
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On 2026-11-07, five builders converting at once kept the store node's chunk index past the builders' 60 s client timeout. 50 builds failed with
chunk store request failed: ReadTimeoutError. The gateway counted each one as a recipe failure: 10 minutes between attempts, and a recipe failed for good after 3. Nothing was wrong with any of those recipes.image_recipes): errors that start with the chunk store client's own transport-failure text (chunk store request failed:orchunk store request deadline exceeded, now constants inchunk_index) are infrastructure failures, like sandbox builds'sandbox build infrastructure:. They retry after 30 s, doubling, up to 6 attempts. The index summary counts them the same way. Only the gateway changes; builders keep reporting what they already report.docs/images.mdretry rules and aCHANGELOG.mdentry.The broken-pipe and slow-request changes only affect logging, so there's no test for them.
Production: separately from this PR,
builder.max_finishing_buildsis now 1 (was 2), which halves concurrent conversions per builder. It takes effect as builders are replaced.Checks: full suite with PostgreSQL (2441 tests: 2426 pass, 15 skipped), ruff, and the test-rule check all pass.
🤖 Generated with Claude Code