evm-rpc: tolerate null from trace_replayBlockTransactions (fixes tempo hotblocks crash-loop)#497
Merged
Conversation
A provider can transiently return `result: null` for a freshly produced (tip) block whose trace index has not caught up yet. `addTraceTxReplays` validated the result with `array(...)`, so a null response threw a fatal, non-retryable `DataValidationError: null is not an array`, terminating block ingestion and forcing a restart loop. The sibling debug-trace paths (`addDebugStateDiffs`, `addDebugFrames`) already handle this case: they accept a null result and mark the block `_isInvalid` so `getBlocks` retries it (up to 5x). Apply the same pattern to the trace-replay path: wrap the validator in `nullable(...)` and flag the block for retry on null instead of crashing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tmcgroul
approved these changes
Jun 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident
Hotblocks data-lag alert on tempo-mainnet (correlated providers dwellir/uniblock), onset ~13:31 UTC on 2026-06-17, ~9 min after a fleet-wide deploy of
evm-data-service:e56f20a9(image bump commitd409edbat 13:20 UTC).Root cause
The new image introduced a crash-loop on tempo:
trace_replayBlockTransactionstransiently returnsresult: nullfor a freshly-produced tip block whose trace index isn't ready yet. InRpc.addTraceTxReplays, the result is validated witharray(...), so anullthrows a fatal, non-retryableDataValidationError: null is not an array, terminating ingestion repeatedly.Evidence:
56617aaf) had 0 terminations over 3h before cutover; crash-looping began only withe56f20a9.nullduring the incident now return a valid empty array[]→ confirms thenullis a transient tip-race, not a permanent capability gap.debug_*paths (addDebugStateDiffs/addDebugFrames, andeth_getBlockReceipts) already toleratenullby flagging the block for retry.addTraceTxReplayswas the lone path missing this guard.(Note: the cronos-mainnet error
could not find results for heightis pre-existing on both images and is a separate issue, not introduced by this bump.)Fix
Mirror the existing debug-path pattern in
addTraceTxReplays:nullable(array(...))null, setblock._isInvalid = true+_errorMessageandcontinue, so the batch is retried instead of crashing.Structurally identical to the already-shipping
addDebugStateDiffs/addDebugFramesnull handling.🤖 Generated with Claude Code