Skip to content

feat: pipelined fetch/write for indexer #55

@pthmas

Description

@pthmas

Summary

Currently the indexer fetches a batch of blocks, then writes them to the database, then fetches the next batch. The RPC is idle during writes and the DB is idle during fetches. Pipelining would overlap these operations: fetch batch N+1 while writing batch N.

Motivation

This is the single biggest throughput improvement available for indexing speed. Based on analysis of tidx, which implements this pattern, estimated 30-50% faster sync during both initial sync and steady-state head tracking.

Design

  1. After fetching batch N, spawn the DB write as a background task
  2. Immediately begin fetching batch N+1
  3. Before writing batch N+1, await completion of the batch N write
  4. Errors in either fetch or write should still propagate correctly

Key considerations:

  • The reorder buffer (BTreeMap) already collects results in sequence — pipelining fits naturally after the reorder step
  • The binary COPY connection is exclusive during writes, so only one write can be in-flight at a time
  • SSE publishing (via HeadTracker) currently happens before the DB write, so pipelining doesn't affect SSE timing

References

  • tidx implementation: sync/engine.rstick_realtime fetches batch N+1 while writing batch N

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions