Skip to content

Drop negative-bucket histogram _sum only in OpenMetrics 1.0 output - #1254

Open
haeganm wants to merge 2 commits into
prometheus:masterfrom
haeganm:negative-bucket-histogram-count
Open

haeganm wants to merge 2 commits into
prometheus:masterfrom
haeganm:negative-bucket-histogram-count

Conversation

@haeganm

@haeganm haeganm commented Nov 10, 2026 •

Copy link
Copy Markdown

Histogram leaves out _sum when its lowest bucket is negative, since OpenMetrics 1.0 says a histogram with negative buckets MUST NOT contain a sum. MultiProcessCollector never did that check, so in multiprocess mode the same histogram still had a _sum, OM 1.0 output included:

# Histogram('temp_delta', 'help', buckets=[-1, 0, 1]).observe(-0.5), multiprocess
temp_delta_sum -0.5
temp_delta_bucket{le="-1.0"} 0.0
temp_delta_bucket{le="0.0"} 1.0
temp_delta_bucket{le="1.0"} 1.0
temp_delta_bucket{le="+Inf"} 1.0
temp_delta_count 1.0

OpenMetrics 2.0 goes the other way (every histogram has count and sum), so per review this keeps _sum in the samples everywhere and filters it at exposition instead:

  • Histogram always emits _sum
  • openmetrics.exposition.generate_latest drops _sum for histograms with a negative bucket when version is below 2.0.0
  • the histogram docs say which output leaves it out

side effect: the classic text format now shows _sum for these histograms in single process too.

test_histogram_negative_buckets in tests/test_multiprocess.py fails on master (b'h_sum' unexpectedly found in the OM 1.0 output) and passes here, and the OM exposition test now checks that 1.0 leaves _sum out and 2.0 keeps it. full suite passes on 3.12 (Windows), and flake8, isort and mypy are clean.

@csmarchbanks

Histogram leaves out _sum when its lowest bucket is negative, since
OpenMetrics 1.0 doesn't allow a sum there. MultiProcessCollector builds
its own histogram samples and kept the _sum from the mmap files, so the
same histogram exposed an extra series in multiprocess mode.

Drop _sum on the accumulate path when the lowest bucket bound is
negative. merge(accumulate=False) still returns the raw value.

Signed-off-by: Haegan McGarry <haeganm@gmail.com>

@csmarchbanks csmarchbanks left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since OM 2.0 is adding sum back in everywhere I would learn towards at lest keeping it in the samples. That said, we should probably filter it out in exposition for at least OM 1.0 users.

OpenMetrics 2.0 wants a sum on every histogram, so Histogram now keeps
_sum even when the lowest bucket is negative, and multiprocess goes back
to keeping it too. OpenMetrics 1.0 doesn't allow a sum with negative
buckets, so generate_latest drops _sum for those histograms when the
version is below 2.0.0.

Signed-off-by: Haegan McGarry <haeganm@gmail.com>
@haeganm haeganm changed the title Omit _sum for negative-bucket histograms in multiprocess mode Drop negative-bucket histogram _sum only in OpenMetrics 1.0 output Nov 13, 2026
@haeganm

haeganm commented Nov 13, 2026 •

Copy link
Copy Markdown
Author

makes sense, flipped it in ca715c8. Histogram now always keeps _sum (multiprocess already did, so that part of the diff is gone), and openmetrics.exposition.generate_latest drops _sum for histograms with a negative bucket when version is below 2.0.0. so 1.0 output has no sum for these and 2.0 gets it.

one side effect, the classic text format now shows _sum for these in single process too, since that format never had the rule. happy to filter it there as well if you'd rather.

one more thing on 1.0: its text format says _count is there if and only if there's a sum, so a fully valid 1.0 output would drop _count for these too. that's what the OM parser trips on (_sum/_gsum must be present if _count is present), same on master. want me to drop it here? the +Inf bucket keeps the same value. the parser also rejects a negative _sum in 2.0 output (Counter-like samples cannot be negative), but 2.0 only treats sum as a counter when nothing negative was observed. left that one out of this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants