[NET-528] [Alert iPNiow] ethereum-sepolia_Ingest_Behind_Head

[elina-chertova]

Automated fix proposal for alert iPNiow.

• Alert: ethereum-sepolia_Ingest_Behind_Head
• Base branch: open-beta
• Investigation: /root/alert/incident-agent/agent-system/data/investigations/iPNiow
• Report: /root/alert/incident-agent/agent-system/data/investigations/iPNiow/report.html

Reviewer quick view

• Scope: 1 file(s) in evm

• Root cause (agent): not explicitly captured

• Summary: Verdict: accept_with_changes

  Root cause confirmed [evidence]: A code bug in squid-sdk:open-beta
  evm/evm-normalization/src/mapping.ts line 114 groups the EVM INVALID opcode
  with real CALL variants under assertNotNull(frame.to). Alchemy returns to:
  null for INVALID frames. The assertion throws. Ingest loops forever every
  ~8s on block 0xa5d8bc. No crash, no OOM, no 429.

  Two tracks:

  Track A (now): Deploy debug_api_for_statediffs: false in infra/master YAML →
  bypasses mapDebugFrame, ingest resumes within minutes. Proposed patch
  already on disk at fixes/proposed/.../ethereum-sepolia.yaml.

  Track B (days): SDK PR on open-beta — extract INVALID into its own case
  (proposed patch at fixes/proposed/evm/evm-normalization/src/mapping.ts). One
  issue before merging: (frame.to ?? '') as Bytes20 emits an empty string
  which is not a valid Bytes20 — the team needs to decide whether
  TraceCallAction.to should be optional for callType: 'invalid'.

  All original hypotheses (429, OOM, shared rate-limit) ruled out by evidence.

Fix metadata

• Fix class: rca_fix (mapping.ts) + mitigation (infra YAML)
• Confidence: high
• Evidence basis: logs, code, rpc
• Falsification: if block 0xa5d8bc's Alchemy debug trace does NOT contain an
• Follow-up: unit test for mapDebugFrame with INVALID frame; verify frame.input/
  (Generated by the terminal-debate agent — values reflect the agent's self-assessment, not a verified verdict. Use them as a starting point for review.)

Summary

Verdict: accept_with_changes

Root cause confirmed [evidence]: A code bug in squid-sdk:open-beta
evm/evm-normalization/src/mapping.ts line 114 groups the EVM INVALID opcode
with real CALL variants under assertNotNull(frame.to). Alchemy returns to:
null for INVALID frames. The assertion throws. Ingest loops forever every
~8s on block 0xa5d8bc. No crash, no OOM, no 429.

Two tracks:

Track A (now): Deploy debug_api_for_statediffs: false in infra/master YAML →
bypasses mapDebugFrame, ingest resumes within minutes. Proposed patch
already on disk at fixes/proposed/.../ethereum-sepolia.yaml.

Track B (days): SDK PR on open-beta — extract INVALID into its own case
(proposed patch at fixes/proposed/evm/evm-normalization/src/mapping.ts). One
issue before merging: (frame.to ?? '') as Bytes20 emits an empty string
which is not a valid Bytes20 — the team needs to decide whether
TraceCallAction.to should be optional for callType: 'invalid'.

All original hypotheses (429, OOM, shared rate-limit) ruled out by evidence.

Risk & rollout

• Suggested rollout: canary / one-network-first, then broader rollout after signal is stable.
• Rollback: revert this PR (or restore previous config values/files) if the incident signal worsens.

Reproduction status

Incident behavior was reproduced or corroborated strongly enough for a non-hypothesis fix proposal.

Validation checklist

• Verify the original incident signal improves (logs/metrics/alerts) after deploy.
• Verify no regression on sibling networks/providers/services touched by this change.
• Confirm queue / delivery pipeline status returns to expected steady state.

Changed files

• evm/evm-normalization/src/mapping.ts

Notify

cc @tmcgroul (automation opened this PR.)