Add UnivMon-Q and correct UnivMon

[zaoxing]

Fixed UnivMon bugs

This PR first corrects several correctness and merge issues in the existing UnivMon implementations:

• Missing per-layer L2 updates: Standard insert did not correctly maintain CountSketch/L2 state at every sampled logical layer.

• Incomplete Joltik update path: fast_insert did not fully implement terminal-stratum storage with logical-layer reconstruction at query time.

• Unsafe update-mode mixing: A sketch could mix standard and terminal-only updates, producing state that neither query algorithm could interpret correctly.

• Negative terminal updates: Terminal-only reconstruction assumes insertion-only streams, but negative weights were previously accepted.

• Stale candidate frequencies: Queries used counts stored in heavy-hitter heaps instead of re-estimating candidates from current CountSketch counters.

• Incorrect merge candidate counts: Merges combined candidate metadata without consistently re-estimating frequencies from the merged counter state.

• Stale merged L2 norms: CountL2HH cached row norms were not recomputed after counter arrays were merged.

• Incorrect stream-weight accounting: UnivMon merge and experimental window paths could omit or double-count aggregate stream weight.

• Fixed cardinality cutoff: Cardinality used a fixed heavy-item cutoff instead of a threshold derived from L2 and candidate capacity.

• Unknown candidate completeness: The implementation did not track when bounded heavy-hitter storage had evicted identities, making zero-threshold estimation unsafe.

• Empty entropy behavior: Entropy on an empty sketch did not consistently return zero.

• Legacy-state handling: Older MessagePack states lacked update-mode and candidate-completeness fields and required normalization during decoding.

The corrected implementation now maintains distinct standard and terminal update modes, reconstructs logical terminal layers at query time, refreshes candidate estimates from counters, and performs merge reconstruction from the combined state.

Summary

This PR also adds UnivMonQ, a mergeable universal sketch that combines frequency-vector metrics with ordered queries.

UnivMon-Q supports:

• Exact observation count, minimum, and maximum

• Point-frequency estimation

• F0, F2, and F3

• Generic compatible g-sum queries

• Shannon entropy

• Heavy hitters

• Rank, CDF, and quantile queries

• Compatible merges and tumbling windows

• Native MessagePack serialization

• Pluggable hashing

UnivMon-Q construction

Updates use a Joltik-style terminal-stratum layout:

• One 128-bit hash is split into CountSketch bucket/sign fields, terminal-level selection, and ordered-sample priority.

• Each observation updates only one physical CountSketch layer.

• Logical UnivMon levels are reconstructed during queries.

• A coordinated bottom-k residual sample provides ordered distribution coverage.

• Bounded candidate summaries recover heavy values.

This keeps the update and merge paths substantially cheaper than maintaining every logical layer.

Query API

The direct API includes:

sketch.estimate_frequency(value);
sketch.estimate_distinct();
sketch.estimate_f2();
sketch.estimate_f3();
sketch.estimate_g_sum(|frequency| frequency.powi(4));
sketch.estimate_entropy();
sketch.heavy_hitters(k);
sketch.rank(value);

sketch.quantile(q);

sketch.quantiles(&[0.50, 0.90, 0.99]);

sketch.cdf();

For multi-metric query batches, callers can reconstruct query state once:

let query = sketch.prepare_queries();

let f0 = query.estimate_distinct();

let f2 = query.estimate_f2();

let f3 = query.estimate_f3();

let entropy = query.estimate_entropy();

let quantiles = query.quantiles(&[0.50, 0.90, 0.99]);

let cdf = query.cdf();

The prepared view shares candidate recovery, logical hierarchy reconstruction, F2 thresholds, and CDF construction across queries.

Query-performance improvements

The optimized reconstruction:

• Hashes and estimates each retained candidate once.

• Scans each physical CountSketch once for F2 thresholds.

• Constructs suffix top-k sets incrementally.

• Reuses recovered frequencies across arbitrary g-sums.

• Builds one CDF for batched rank and quantile queries.

• Avoids rebuilding the complete hierarchy for every metric in a query batch.

On the included 250,000-element synthetic benchmark with 64 candidates:

Query	Previous	Optimized direct	Prepared view
F0	1.58 ms	144 µs	70 ns
F2	1.33 ms	144 µs	92 ns
F3	Not exposed	143 µs	103 ns
Entropy	1.79 ms	145 µs	458 ns
Top-4	714 µs	175 µs	76 ns
Rank	371 µs	327 µs	20 ns
p50/p90/p99	1.29 ms	337 µs	90 ns
Full CDF	312 µs	215 µs	Cached slice

Results are workload- and machine-dependent; the benchmark is included for reproducibility rather than as a universal performance guarantee.

Accuracy snapshot

For the same synthetic workload:

• All constructions recovered the four true heavy hitters.

• UnivMon-Q F2 relative error was below 0.01%.

• UnivMon-Q F3 relative error was approximately 0.02%.

• The memory-matched UnivMon-Q configuration achieved:

  • F0 error: 8.73%

  • Entropy error: 1.51%

  • p50 error: 0.32%

  • p90 error: 0.32%

  • p99 error: 0.33%

Compatibility

• UnivMon-Q is currently marked Unstable.

• It does not yet have an ASAPv1 cross-language kind identifier.

• Its native serialization uses a validated MessagePack DTO.

• Compatible merges require identical dimensions, seed, counter width, ordered-sample configuration, and hasher type.

• Existing UnivMon decoding normalizes legacy states without the new update-mode and candidate-completeness fields.

Validation

• 550 library tests passed.

• All integration-test suites passed.

• 20 documentation tests passed.

• cargo clippy --all-targets --all-features -- -D warnings passed.

• Formatting and diff checks passed.

Included examples

• examples/quantile_univmon_q.rs

• examples/compare_univmon_q.rs

[zaoxing]

Performance versus UnivMon

Quick benchmark: 250K observations, 12 levels, width 2,048, depth 5.

Construction	Updates	Memory	Merge
UnivMon standard	0.312 M/s	≥960 KiB + heaps	—
UnivMon terminal	0.505 M/s	≥960 KiB + heaps	2.16 ms
UnivMon-Q	9.29 M/s	548 KiB	123 µs
UnivMon-Q memory-matched	10.10 M/s	944 KiB	—

UnivMon-Q was approximately 18× faster to update and merge than terminal UnivMon. At equal candidate capacity, it used about 43% less memory.

Query latency

Query	UnivMon standard	UnivMon terminal	Q direct	Q prepared
Count	33.3 µs	246.1 µs	<1 ns	<1 ns
Frequency	45 ns	50 ns	24 ns	24 ns
F0	48.4 µs	165.6 µs	143.9 µs	70 ns
F2	75.0 µs	260.9 µs	143.5 µs	92 ns
F3/g-sum	96.9 µs	162.4 µs	143.3 µs	103 ns
Entropy	65.9 µs	172.0 µs	144.9 µs	458 ns
Top-4	6.0 µs	101.3 µs	174.6 µs	76 ns
Rank	Unsupported	Unsupported	326.7 µs	20 ns
p50/p90/p99	Unsupported	Unsupported	337.2 µs	90 ns
CDF	Unsupported	Unsupported	214.6 µs	Cached

Preparing the reusable query view costs 388 µs. A complete multi-metric batch takes approximately 389 µs, compared with 292 µs for standard UnivMon and 862 µs for terminal UnivMon.

Standard UnivMon remains best for isolated universal queries. UnivMon-Q provides much faster ingestion and merging, while prepared views amortize reconstruction across F0/F2/F3/entropy/heavy-hitter/rank/quantile/CDF queries.

Accuracy snapshot

• All constructions recovered the true top four heavy hitters.
• F2 error: UnivMon 0.06%, UnivMon-Q below 0.01%.
• F3 error: UnivMon 0.04%, UnivMon-Q 0.02%.
• Memory-matched UnivMon-Q: F0 8.73%, entropy 1.51%.
• Memory-matched quantiles: p50 0.32%, p90 0.32%, p99 0.33%.

Results are workload- and machine-dependent; the benchmark is included for reproducibility.