Repository navigation
Conversation
Add STALE_NAN_BITS, STALE_NAN, DATA_POINT_FLAG_NO_RECORDED_VALUE, is_stale_nan, lazy flags column materialization on NumberDataPoints, SummaryDataPoints, and HistogramDataPoints, and zero-steady-state-allocation per-target StalenessTracker support in ArrowDecoder. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka
force-pushed
the
bwplotka/staleness-detection
branch
from
November 12, 2026 07:30
099e550 to
2b0161b
Compare
Explain that OTAP treats data-point flags as default_values_optional so omitting all-zero flags columns preserves schemas and avoids UInt32 allocations on live-only batches. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…or StalenessTracker Fix the module rustdoc describing StaleNaN as quiet, note bitwise-only handling for signaling NaN bits on STALE_NAN and is_stale_nan, and implement Debug on StalenessTracker. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Use NONE_IDX sentinels in batch_key_to_tracker and batch_val_to_tracker instead of sequential catch-up loops so ephemeral exemplar label keys and values in batch interners never leak into StalenessTracker. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Normalize counter family names by stripping trailing _total suffixes and include MetricType and saw_total_suffix in TrackedFamily keys so # TYPE changes never attach data points to mismatched UnivariateMetrics rows and # TYPE req counter vs # TYPE req_total counter share one family. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
… scrape completion Stage per-scrape family metadata updates and preserve committed active flags until commit_scrape runs after final RecordBatch construction so mid-scrape errors and early DecodeStream drops roll back cleanly. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…d lazy allocation Make StalenessTracker::new() zero-allocating until first use, shrink TrackedSeries layout, clear arenas when active_count drops to zero, compact dead series, families, and interned strings under label churn, and document the opt-in memory model. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
… OTAP batches Emit NULL for NumberDataPoints.double_value and HistogramDataPoints.sum (and 0.0 for SummaryDataPoints.sum with empty buckets, quantiles, and exemplars) when FLAG_NO_RECORDED_VALUE is set, matching OTLP and OTAP semantics without poisoning Arrow float aggregations. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Add DecodeOptions::track_timestamps_staleness (default false, matching Go Prometheus scrape/scrape.go) so client-timestamped samples are excluded from StalenessTracker unless explicitly enabled. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…ith_staleness Enforce entry and duration DecodeBudget limits inside StalenessTracker::finish_scrape_step so cooperative streams yield during large target drops, and add step_decode_with_staleness and finish_decode_with_staleness for lifetime-free DecodeCursor callers. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…ess in SPEC.md Drop ArrowDecoder::encode_stale_target in favor of flush_target_stale to keep the public API surface minimal, and document StalenessTracker, FLAG_NO_RECORDED_VALUE, and track_timestamps_staleness in specs/SPEC.md. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…talenessTracker loop Keep the stateless ArrowDecoder::decode OpenMetrics bounded-memory check intact and add separate opt-in StalenessTracker checks covering steady-state scrapes and alternating full/subset disappearing-series cycles. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…and budgets Add tests covering flushed metadata rollback, zero-capacity rehashing, key-order resorting, exemplar interner isolation, churn compaction, wire NaN vs StaleNaN, # TYPE transitions, UntypedPolicy::Drop, track_timestamps_staleness, and budgeted cooperative stale draining. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Monomorphize ingest_entries and step_cursor over const TRACKED: bool so untracked decode, decode_stream, and step_decode paths compile out StalenessTracker branches and register state. Split OptionalValueCol::push_value and SparseFlagsCol::push_zero from #[cold] null/flag slow paths so live scalar samples avoid Option<f64> construction and inlined Vec growth paths. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka
commented
Nov 12, 2026
| /// `flags = DATA_POINT_FLAG_NO_RECORDED_VALUE` (`1`), and then removed from the active set so | ||
| /// subsequent scrapes do not re-emit it. | ||
| /// | ||
| /// # Memory model and optional usage |
Member
Author
There was a problem hiding this comment.
NOTE: I tried to optimize it hard. I think there's a room for a 3rd alternative (currently it's no staleness or staleness per series) -- staleness per target as we discussed in https://cloud-native.slack.com/archives/C0CFYDE79D5/p1794385004482139
bwplotka
marked this pull request as ready for review
November 12, 2026 09:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Prometheus series staleness detection and optional cross-scrape disappeared series tracking to
expfmt::arrow::ArrowDecoder.STALE_NAN_BITS,STALE_NAN,DATA_POINT_FLAG_NO_RECORDED_VALUE, andis_stale_nan(f64)inexpfmt::arrow.flagscolumn onNumberDataPoints,SummaryDataPoints, andHistogramDataPointsonly when a batch contains a non-zero flag, and omits stale data point values per OTLPFLAG_NO_RECORDED_VALUEsemantics rather than writingSTALE_NANinto OTAP payloads.StalenessTrackeralong withArrowDecoder::decode_with_staleness,decode_stream_with_staleness,step_decode_with_staleness,finish_decode_with_staleness, andflush_target_stale, withDecodeOptions::track_timestamps_stalenessdefaulting tofalse, transactional rollback on failed or aborted scrapes, and dead-series arena compaction under series churn.prombenchyotap-rustworkload is0 BwhenStalenessTrackeris not used, or17.1 MiBin initial steady state and29.1 MiBat the2xcompaction watermark under 1h full series churn when enabled across all 16 targets.Static live-heap estimate (
prombenchyotap-rust)Workload from
avalanche.exampletarget.yamlandmanifests/scenarios/otap-rust/1_collector.yaml:--num-cores=1andmax_data_points_per_batch: 200scraping 16avalanchepods every 15s.158gauges,280counters,28classic 10-bucket histograms,482-objective summaries),--series-count=10, 12 labels per series (series_id,cycle_id, and 10label_key_kkkkk_*labels), and--series-interval=3600rotatingcycle_idevery hour.StalenessTrackertracks assembledHistogramDataPointsandSummaryDataPointsrows rather than per-bucket or per-quantile lines.Stateless mode without
StalenessTracker(decode,decode_stream,step_decode):0 Badditional heap when no wireStaleNaNis present becauseSparseFlagsColandOptionalValueCol::validstay at capacity0(104 Binline struct fields on the sharedArrowDecoder).Opt-in
StalenessTrackerenabled (1 tracker per target, 64-bitVecdoubling capacities and 70% load-factor hash tables):Veccapacity (0 < t < 1h)2xchurn watermark (1h <= t < 2h)compact()(t >= 2h)series_attrs((u32, u32)@ 8 B, 61,680 active pairs)family_bytes(514 names + ~361 B# HELPtext)series(TrackedSeries@ 16 B, 5,140 active)series_buckets(SeriesBucket@ 16 B)families(TrackedFamily@ 52 B) +family_bucketskey_interner+val_interner+ remap scratch + struct5,140OTAP /9,940Prom series)82,240OTAP /159,040Prom series)Notes on churn and label cardinality:
commit_scrape()runscompact()onceseries.len() > 2 * active_count, so the first 1h churn (5,140active +5,140dead =10,280) retains one dead generation (29.10 MiBacross 16 targets) and the second 1h churn (15,420 > 10,280) compacts back to20.00 MiB, with a ~2.3 MiB single-target transient spike while scraping and compacting one target at a time (~31.4 MiB collector peak).avalanchereuses 21 distinct label values across all 5,140 series per target; if a non-synthetic target instead carried 5,140 distinct 16-byte label values per target,val_internerwould add ~368 KiB per target (~5.8 MiB across 16 targets).Commits in order
2b0161barrow: add Prometheus series staleness detection and StalenessTrackera92e491arrow: document why SparseFlagsCol omits all-zero flags columns9d0f192arrow: clarify signaling StaleNaN bit semantics and implement Debug for StalenessTracker31b5eb4arrow: lazily intern only series attributes in StalenessTrackerbc491a0arrow: key StalenessTracker families by canonical name and MetricType6cff462arrow: defer StalenessTracker active and family metadata commit until scrape completiona076c06arrow: bound StalenessTracker live heap via dead-series compaction and lazy allocation00bde12arrow: omit stale data point values instead of writing STALE_NAN into OTAP batchese65fe06arrow: skip staleness tracking for explicit wire timestamps by default87d4b18arrow: honor DecodeBudget during stale draining and add step_decode_with_staleness03ab0a3arrow: remove duplicate encode_stale_target alias and document staleness in SPEC.md2ee1003tests: preserve stateless OpenMetrics dhat check alongside optional StalenessTracker loop37b3679arrow: add unit and OTAP tests for staleness edge cases, compaction, and budgets0d63e9carrow: specialize untracked decode loop and fast-path live scalar valuesBenchmarks (
benchstat before.txt after.txt)Comparing
fd6c3c1(before.txt) vs0d63e9c(after.txt):