Skip to content

docs: record dashboard metrics aggregation decisions - #587

Open
amondnet wants to merge 7 commits into
mainfrom
amondnet/monitoring
Open

amondnet wants to merge 7 commits into
mainfrom
amondnet/monitoring

Conversation

@amondnet

@amondnet amondnet commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Document how the admin dashboard can display the 18 metric series that shunt already records.
The decision record explains why the current OpenTelemetry instrument handles cannot serve as a
read path and why tying dashboard visibility to an [otel] endpoint would invert the intended
use case.

The record adopts three decisions:

  • shunt owns the metric aggregate, while Sentry and OpenTelemetry read from it;
  • dashboard history uses a bounded in-memory ring rather than a durable store;
  • metric label cardinality must be bounded before retaining history by label set.

This adopts no store and changes no code, behavior, configuration, endpoint, or CLI surface. It
removes the dashboard from the list of work blocked on the storage decision. The accompanying
docs/storage.md scope note preserves the durable-storage requirement for history that survives
restarts and for seven-day per-account retention.

Issue #296 remains open and is referenced only as context. Its title scopes the unbounded model
label to the Codex endpoint, but the /v1/messages default-provider fallback has the same
property. A Codex-only fix could therefore close #296 while leaving the Anthropic path unbounded.

Milestone / spec

Documentation-only design record: docs/dashboard-metrics.md, aligned with docs/storage.md and
docs/admin-ui-delivery.md.

Checklist

  • cargo build passes — not run; no code changed
  • cargo test passes (new behavior is covered; tests run without network/loopback where possible) — not run; no behavior changed
  • cargo clippy --all-targets -- -D warnings clean — not run; no Rust changed
  • cargo fmt --all --check clean — not run; no Rust changed
  • Source files stay under 500 lines
  • English only; matches surrounding style
  • Frozen spec in docs/ updated if this change deviates from it
  • User-facing docs updated for behavior/config/endpoint/CLI/provider/model changes — no user-facing behavior changed
  • Any new GitHub Action is pinned to a full commit SHA — no GitHub Action added

Notes for reviewers

This PR is a decision record, not an implementation. No tests apply because no code changed.

Every src/<file>:<line> reference in the new document was mechanically verified to resolve to
the claimed line. Internal document links and the admin-ui-delivery.md anchor were also verified
to resolve.


Summary by cubic

Adds docs/dashboard-metrics.md, a documentation-only design record for how the admin dashboard can display the 18 metric series shunt already records, without adopting a store or changing code. It unblocks monitoring for a recent window; restart-surviving history and seven-day per-account retention still require durable storage.

  • Records three decisions: shunt owns the metric aggregate, dashboard history uses a bounded in-memory ring, and metric label cardinality is bounded before retention. The aggregate is an additional read model written alongside the existing Sentry and OTel emissions, so counters and histograms keep exporting and are not accumulated twice.
  • Documents that the ring is process-local and does not survive restarts, so the UI must label it as this instance's view rather than the fleet's.
  • Notes the ring's fixed window is itself a retention policy, and since model is client-controlled and passed through verbatim, the label set needs an explicit allowlist/redact/hash policy; the choice of transform is left as an open question.
  • Records opentelemetry_sdk 0.32.1 feature-gating as an independent reason to reject the ManualReader read path: both ManualReader and InMemoryMetricExporter are gated behind features this crate does not enable.
  • Updates docs/admin-ui-delivery.md and docs/storage.md to reconcile them with the new record; access-permission management remains blocked on the storage decision.
  • Notes that issue fix(codex): bound the inbound model metrics label length #296 is scoped to the Codex endpoint, but the /v1/messages default-provider fallback has the same unbounded model label property.

Written for commit 1961798. Summary will update on new commits.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a design document, docs/dashboard-metrics.md, which outlines the strategy for enabling the admin UI to read metrics without requiring a durable store. The proposed approach involves shunt owning the aggregate in-memory, using a bounded in-memory ring for history, and enforcing label cardinality limits. Additionally, docs/storage.md is updated to clarify that these changes satisfy the dashboard's need for recent history, decoupling it from the requirement for durable storage. I have no feedback to provide on these changes.

@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed

codspeed Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will degrade performance by 13.05%

❌ 1 regressed benchmark
✅ 100 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
parse_body_to_value[200] 2.9 ms 3.4 ms -13.05%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing amondnet/monitoring (1961798) with main (b4fa5fd)

Open in CodSpeed

@amondnet amondnet closed this Sep 18, 2026
@amondnet amondnet reopened this Sep 18, 2026
@amondnet
amondnet marked this pull request as ready for review September 18, 2026 03:24

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9720c2b859

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/dashboard-metrics.md Outdated
@greptile-apps

greptile-apps Bot commented Sep 18, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The documentation-only PR appears safe to merge, with no outstanding correctness or repository-rule violations.

Summary

Documents the intended architecture for exposing existing metrics to the admin dashboard without adopting durable storage.

  • Establishes a shunt-owned aggregate as an additional read model while preserving individual Sentry and OpenTelemetry emissions.
  • Selects bounded, process-local in-memory history and documents its restart and multi-replica limitations.
  • Requires bounded label cardinality before retention while leaving the unmatched-label transformation as an explicit open question.
  • Reconciles the dashboard delivery and storage records with these decisions.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  O[Metric observation] --> A[Shunt-owned aggregate]
  O --> S[Sentry]
  O --> T[OpenTelemetry]
  A --> R[Bounded in-memory ring]
  R --> API[Admin read API]
  API --> UI[Dashboard]
Loading

Reviews (6) · Last reviewed commit: "chore(docs): reconcile Alternative C wit..."

Comment thread docs/dashboard-metrics.md Outdated
Alternative A stated that `ManualReader` and `InMemoryMetricExporter` are
"neither feature-gated" and "re-exported unconditionally" from
`opentelemetry_sdk::metrics`. Both are gated in 0.32.1: `ManualReader` behind
`experimental_metrics_custom_reader` (metrics/mod.rs:50-51, 75-76) and
`InMemoryMetricExporter` behind `testing` (metrics/mod.rs:67-72). The SDK's
default feature set carries neither, and this crate takes those defaults
(Cargo.toml:73), so neither type is reachable in the current build.

The decision is unchanged — Alternative A stays rejected — but the gating is a
second blocker independent of the conditional-provider constraint, so it is
recorded alongside it rather than left implicit.
greptile-apps[bot]
greptile-apps Bot previously approved these changes Sep 18, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5b971eb920

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/dashboard-metrics.md Outdated
Two bot review findings on #587, both cross-document drift:

- The ring's stated limitation named only "does not survive a restart". It is
  also process-local, so across several processes each ring sees one replica's
  traffic. Decision 2 now carries that, pointing at the caveat
  admin-ui-delivery.md already puts on `/admin/api/pool` — this instance's view,
  not the fleet's.
- admin-ui-delivery.md still said a dashboard worth building needs durable
  history and that monitoring is blocked on storage.md, which contradicts
  dashboard-metrics.md Decision 2. Its storage section and its
  "What this does not decide" list now separate a recent in-memory window, which
  needs no store, from restart-surviving and per-account history, which does.

The blocked-feature count in that list drops from two to one accordingly:
access-permission management stays blocked, monitoring becomes partly unblocked.
@greptile-apps
greptile-apps Bot dismissed their stale review September 18, 2026 03:56

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

Decision 2 listed "no retention policy, no PII decision" among the questions a
ring buffer disposes of. Both overclaim, and Decision 3 in the same document is
what contradicts them: `model` is client-controlled and passed through verbatim,
so whatever a client puts there is retained for the ring's window and served
over the admin API, and a fixed window is itself a retention policy. Capping
cardinality bounds how many distinct labels are kept, not what one contains.

The claim is now scoped to what a ring genuinely avoids — schema, migration,
replicas — with storage.md's fourth question recorded as inherited rather than
escaped, and the label set pointed at an allowlist/redact/hash policy alongside
Decision 3's bounding.
Comment thread docs/dashboard-metrics.md Outdated
greptile-apps[bot]
greptile-apps Bot previously approved these changes Sep 18, 2026
The retention scoping said the admitted label set needs an allowlist, redaction,
or hash policy without choosing among them, and the choice appeared nowhere in
the open-questions list — so an implementer could not tell what transform an
unmatched `model` value gets before the ring retains it.

Decision 3 deliberately bounds cardinality rather than content, so the transform
is a separate decision this record does not settle. It is now stated as such:
open question 5, with a pointer to it from the retention paragraph.
@greptile-apps
greptile-apps Bot dismissed their stale review September 18, 2026 04:01

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

greptile-apps[bot]
greptile-apps Bot previously approved these changes Sep 18, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 97d4f727e3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/dashboard-metrics.md Outdated
Decision 1 said Sentry and OTel "become readers of that structure rather than
its only home". Taken literally that breaks the sixteen counters and histograms:
they push once per observation (Sentry `capture()`, OTel `add()`/`record()`,
src/metrics.rs:423-443), and only the two observable gauges are read by a
callback. Feeding a running aggregate into `add()` would accumulate it twice,
and replacing the per-observation writes with a callback would stop those series
exporting.

The aggregate is an additional read model written alongside the existing
emissions, which both sinks keep unchanged — the gauge pattern is the model for
reading a value in-process, not for how a counter reaches a sink.
@greptile-apps
greptile-apps Bot dismissed their stale review September 18, 2026 04:07

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

Comment thread docs/dashboard-metrics.md
greptile-apps[bot]
greptile-apps Bot previously approved these changes Sep 18, 2026
Alternative C was still headed "both sinks read from it" while Decision 1 now
says the aggregate is an additional read model and the sinks keep receiving
individual observations. The adopted alternative and the decision it adopts
described different data flows, so an implementer could not tell whether
exporters consume observations or aggregate values.

C now names what is applied uniformly — the gauge pattern's ownership, not its
emission — and points at Decision 1 for why.
@greptile-apps
greptile-apps Bot dismissed their stale review September 18, 2026 04:12

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(codex): bound the inbound model metrics label length

1 participant