Skip to content

feat(storage): add orchestrator.read.* OTEL metrics and chunk.fetch span - #16

Closed
levb wants to merge 3 commits into
lev-storage-rationalizefrom
lev-gcs-read-ttfb-metric
Closed

levb wants to merge 3 commits into
lev-storage-rationalizefrom
lev-gcs-read-ttfb-metric

Conversation

@levb

@levb levb commented May 27, 2026

Copy link
Copy Markdown
Owner

Wire read-stage instrumentation across the read path with file_type/source/ codec attributes.

Metrics:

  • orchestrator.read.open OpenRangeReader latency
  • orchestrator.read.read caller-side read wall time
  • orchestrator.read.decompress decompression time + bytes
  • orchestrator.read.fetch backend fetch total latency
  • orchestrator.read.writeback NFS cache writeback latency
  • orchestrator.read.pipeline.efficiency pipeline saturation ratio
  • orchestrator.read.cache NFS cache hit/miss/writeback events
  • orchestrator.read.inflight concurrently-open readers (gauge)

Spans:

  • chunk.fetch

@levb
levb force-pushed the lev-storage-rationalize branch 3 times, most recently from 53e81af to 042a224 Compare June 1, 2026 20:23
Lev Brouk and others added 3 commits June 1, 2026 13:47
Adds an orchestrator.read.* metric family with consistent attributes
(file_type/source/codec/outcome) covering each stage of a read, plus
per-layer chunker and build-file timers, so dashboards can attribute
latency end-to-end from sandbox-visible read to backend fetch.

Does not remove any of the prior metrics, this will be done separately
after the dashboards are updated.

Metrics:
  - orchestrator.file.read_at               build.File.ReadAt — per-fault
                                            unit, aggregates all underlying
                                            mappings into one record
  - orchestrator.chunk.slice                Chunker.Slice — per per-mapping
                                            unit, source=mmap on cache hit
                                            else the backend that served
  - orchestrator.read.open                  OpenRangeReader (open / TTFB)
  - orchestrator.read.read                  source-read wall, compressed bytes
  - orchestrator.read.decompress            decompress CPU + uncompressed bytes
  - orchestrator.read.fetch                 total fetch wall + bytes delivered
  - orchestrator.read.writeback             NFS cache writeback wall + bytes
  - orchestrator.read.pipeline.efficiency   fetch / (open+read+decompress)
  - orchestrator.read.cache                 NFS hit/miss/writeback events
  - orchestrator.read.inflight              concurrent fetches gauge

Spans:
  - chunk.fetch                             runFetch goroutine span
@levb
levb force-pushed the lev-gcs-read-ttfb-metric branch from f75e37b to 579c5ea Compare June 1, 2026 20:47
@levb levb closed this Jun 8, 2026
@levb
levb deleted the lev-gcs-read-ttfb-metric branch June 8, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant