Kill stalled tier2 segments instead of slow ones

[maoueh]

Context

A customer's map_token_balances substreams on eth-mainnet was reconnecting in a loop for hours while delivering ~0 blocks. Every attempt died with:

error during init_stores_and_backprocess: run_parallel_process failed: parallel processing run:
scheduler run: segment [25652000-25653000] failed while streaming data:
rpc error: code = DeadlineExceeded desc = deadline_exceeded: request active for too long

Traced end to end:

1. service/tier2.go watchdog fires once a segment exceeds segmentExecutionTimeout (60min), cancelling with ErrRequestActiveForTooLong.
2. Tier1's worker maps codes.DeadlineExceeded to a RetryableErr.
3. But orchestrator/work/worker.go:161 intercepts first: streamOutput && upstream.DataSent() makes it fatal with zero retries — correctly so, since those blocks already reached the client and retrying would duplicate them.

The module makes roughly a thousand eth_call per block (the rpc:eth_call metric counts batches, not individual calls). Segments were landing right at the edge of the 60 minute budget. A segment that misses is never cached, so the retry redoes all of it and races the same clock again — the request could never complete regardless of how many times the client reconnected.

Change

Tier2 now aborts a segment when it stops making progress rather than when it merely takes a long time.

• New stall timeout, 10min by default (WithSegmentStallTimeout), reset on every block processed.
• segmentExecutionTimeout demoted to an absolute backstop, default raised 60min → 4h.
• Stall wins when both deadlines blow — it is the more actionable diagnosis.
• Decision extracted into segmentWatchdog (service/segment_watchdog.go) so it is testable against wall-clock scenarios rather than only inside the ProcessRange goroutine.

A wedged segment is still killed promptly. Because a single block is already bounded by blockExecutionTimeout (3min), the stall timeout cannot be tripped by one legitimately slow block.

Observability: request active for a long time gained a since_last_progress field, and a stall-kill reports request stalled, no block progress, distinct from request active for too long.

Also in here

Two smaller fixes found while tracing:

• Stats.processedBlocks is now atomic. It is written by the block-processing goroutine and read by the watchdog goroutine. That cross-goroutine read already existed for the slow-query log, but control flow now depends on it.
• Tier1 shutdown reported as Internal instead of Unavailable. The endpoint is shutting down, please reconnect error is wrapped several times on its way up from the scheduler and was matched with a pointer comparison (err == errShuttingDown), so it never took the Unavailable path. Clients lost the reconnect signal during every tier1 rollout.

Testing

• service/segment_watchdog_test.go — covers the customer scenario (one block per 5min for 3h survives, where the old budget killed it at 60min), stall detection, deadline reset on progress, the absolute backstop, precedence, and disabled timeouts.
• service/error_shutting_down_test.go — asserts Unavailable for both the bare and the four-deep-wrapped shutdown error.
• go build ./... clean; service, metrics, orchestrator, pipeline, app pass; -race clean on service, metrics, pipeline.

Reviewer notes

• The 4h backstop is a judgement call, not a derived number. It is what makes this workload survivable. A segment can now hold a tier2 worker slot up to 4h instead of 1h — a real fleet-wide capacity change.
• worker.go:161 is deliberately untouched. This change avoids tripping the no-retry path rather than softening it.
• This is a resilience fix, not a cure. ~1000 eth_call/block remains the root cause on the module side; caching token metadata in a store would be the real fix there.

[sduchesneau]

🔍 Vulnerabilities of ghcr.io/streamingfast/substreams:854d0df

📦 Image Reference ghcr.io/streamingfast/substreams:854d0df

digest	sha256:01d21563577b4df6a435a8ddb59e85f571af444055c57fbf4b1a7fb1f406d98f
vulnerabilities
platform	linux/amd64
size	120 MB
packages	381

📦 Base Image ubuntu:24.04

also known as	noble noble-20260610
digest	sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf
vulnerabilities