[NET-526] [Alert vrjLJH] uniblock_hyperliquid-mainnet_Hotblocks_Critical_Lag

[elina-chertova]

Automated fix proposal for alert vrjLJH.

• Alert: uniblock_hyperliquid-mainnet_Hotblocks_Critical_Lag
• Base branch: open-beta
• Investigation: /root/alert/incident-agent/agent-system/data/investigations/vrjLJH
• Report: /root/alert/incident-agent/agent-system/data/investigations/vrjLJH/report.html

Reviewer quick view

• Scope: 2 file(s) in evm

• Root cause (agent): not explicitly captured

• Summary: Three confirmed errors, all with proposed fixes already in fixes/proposed/:

  Error A (primary root cause, SDK fix): AssertionError at
  chain-utils.ts:calculateLogsBloom — same unfixed bug as human-confirmed
  incident 8LVGbY. A receipt log references a system tx hash absent from
  block.transactions; assertNotNull throws. Fix: null-guard before the assert.
  Infra-only patches won't help (per @tmcgroul, PR [SCAM ALERT] Something goes wrong on your discord verification #408).

  Error B (config fix, deploy now): debug_traceBlockByHash fails 100% during
  alert window (Alchemy 429 + Dwellir -32001). Fix:
  use_debug_api_for_statediffs: false in uniblock.yaml. Live probe confirms
  trace_replayBlockTransactions works on uniblock/Hyperliquid — safe to deploy
  immediately.

  Error C (secondary SDK fix): eth_getBlockReceipts returns system receipts
  (cumulativeGasUsed=0x0) not in block.transactions; count check fails after 5
  retries. Fix: filter system receipts in rpc.ts before the count check.
  Critic noted the filter also strips those receipts from bloom verification —
  this is correct behavior for Hyperliquid, and the proposed patch already
  documents it.

  One change from critic: The use_debug_api_for_statediffs: false gate
  (conditional on probe) is removed — the probe confirmed
  trace_replayBlockTransactions works, so the config fix is unconditional.

Fix metadata

• Fix class: rca_fix (Error A, C) + mitigation (Error B)
• Confidence: high (A), medium (B), medium (C)
• Evidence basis: logs, code
• Falsification: Error A: only fails if block.transactions always contains every tx referenced by logs — the pod logs prove it does not at block 35335416.
• Follow-up: Probe trace_replayBlockTransactions support on uniblock for Hyperliquid. If it also fails, set diffs: false.
  (Generated by the terminal-debate agent — values reflect the agent's self-assessment, not a verified verdict. Use them as a starting point for review.)

Summary

Three confirmed errors, all with proposed fixes already in fixes/proposed/:

Error A (primary root cause, SDK fix): AssertionError at
chain-utils.ts:calculateLogsBloom — same unfixed bug as human-confirmed
incident 8LVGbY. A receipt log references a system tx hash absent from
block.transactions; assertNotNull throws. Fix: null-guard before the assert.
Infra-only patches won't help (per @tmcgroul, PR #408).

Error B (config fix, deploy now): debug_traceBlockByHash fails 100% during
alert window (Alchemy 429 + Dwellir -32001). Fix:
use_debug_api_for_statediffs: false in uniblock.yaml. Live probe confirms
trace_replayBlockTransactions works on uniblock/Hyperliquid — safe to deploy
immediately.

Error C (secondary SDK fix): eth_getBlockReceipts returns system receipts
(cumulativeGasUsed=0x0) not in block.transactions; count check fails after 5
retries. Fix: filter system receipts in rpc.ts before the count check.
Critic noted the filter also strips those receipts from bloom verification —
this is correct behavior for Hyperliquid, and the proposed patch already
documents it.

One change from critic: The use_debug_api_for_statediffs: false gate
(conditional on probe) is removed — the probe confirmed
trace_replayBlockTransactions works, so the config fix is unconditional.

Risk & rollout

• Suggested rollout: canary / one-network-first, then broader rollout after signal is stable.
• Rollback: revert this PR (or restore previous config values/files) if the incident signal worsens.

Reproduction status

Incident behavior was reproduced or corroborated strongly enough for a non-hypothesis fix proposal.

Validation checklist

• Verify the original incident signal improves (logs/metrics/alerts) after deploy.
• Verify no regression on sibling networks/providers/services touched by this change.
• Confirm queue / delivery pipeline status returns to expected steady state.

Changed files

• evm/evm-rpc/src/chain-utils.ts
• evm/evm-rpc/src/rpc.ts

Notify

cc @tmcgroul (automation opened this PR.)