Aggregate prom-client metrics across cluster workers for /metrics#667
Draft
silvadenisaraujo wants to merge 2 commits into
Draft
Aggregate prom-client metrics across cluster workers for /metrics#667silvadenisaraujo wants to merge 2 commits into
silvadenisaraujo wants to merge 2 commits into
Conversation
Specification only (no implementation) for fixing per-worker Prometheus registry interleaving on /metrics via prom-client AggregatorRegistry. Internal task id: task-58ee1acb (bean)
Multi-worker runtimes kept a per-worker prom-client registry, so /metrics answered from whichever worker the shared-port round-robin picked. Because keep-alive (5s) is shorter than the scrape interval, Prometheus stored an interleaved braid of independent counters under one instance label, treating every downward sample as a counter reset and inflating rate()/increase() on runtime_* counters by orders of magnitude. Aggregate across workers with prom-client's AggregatorRegistry: the master builds a single merged, monotonic view over the existing cluster IPC and the worker answering the scrape serves that aggregate. Single-worker / LINKED mode keeps serving the local default registry unchanged. - src/service/metrics/clusterMetricsAggregator.ts: IPC request/response protocol, master aggregator, worker responder, timeout fallback to local registry. - master.ts / worker/index.ts: wire the tagged IPC messages and set up the aggregator only when workers > 1. - builtIn/middlewares.ts: prometheusLoggerMiddleware serves the aggregate in multi-worker mode, local registry otherwise; /metrics still excluded from request counters. Includes unit tests proving cross-worker sum, monotonicity, single-worker behavior, and default-metric presence. Co-authored-by: Cursor <cursoragent@cursor.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VTEX IO runtimes run as a Node.js cluster of
min(cpus, 4)workers, each keeping its own prom-client registry, and/metricsis answered by whichever worker the shared-port round-robin hands the scrape to. Because keep-alive (5s) is shorter than the scrape interval, nearly every scrape hits a different worker, so Prometheus stores an interleaved braid of 4 independent counters under one instance label — treating every downward sample as a counter reset and inflatingincrease()/rate()onruntime_*counters by orders of magnitude (~×40 measured live). This work will fix that by aggregating across workers with prom-client'sAggregatorRegistry(pinned^14.2.0): the master builds a single merged, monotonic view over the existing cluster IPC and the worker answering the scrape serves that aggregate. Single-worker / LINKED mode stays unchanged. Full analysis:metrics-discrepancy-report.html§3 and §9.1 (recommendation 1).Spec
⏳ awaiting approval — implementation will follow in a second phase on this same PR
See
specs/aggregate-prom-client-metrics-across-cluster-wor.md.This PR currently contains only the specification file — no implementation, no refactors.
Internal task id: task-58ee1acb (bean)