feat: mark UnivMon-Q experimental and add evaluation - #86
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed UnivMon issues
fast_insert.ever_evictedstate through updates, merges, and native MessagePack.Experimental UnivMon-Q
estimate_l1,estimate_entropy_universal, andestimate_entropy_occurrence.UnivMon comparison
Release-scale sweep: 1M observations, 100K domain, seven Zipf skews, five trials, eight-way merges.
The 3.05% entropy result precedes the final heavy/residual variance refinement and is conservative for the current estimator.
Compact UnivMon-Q uses 0.707 MiB estimated memory versus at least 2.188 MiB of UnivMon counters before heap metadata. It is 8–204x faster on updates, 16–52x faster on eight-way merges, and 2.6–4.4x faster for all-metric query batches.
The conservative equal-memory profile fits all UnivMon-Q state within UnivMon's counter bytes alone. It remains 7.4–199x faster on updates, 13.2–18.3x faster on merge, and 1.4–2.2x faster on query batches.
Entropy stress evaluation
The final assisted estimator was evaluated on 144 datasets / 576 sketch runs: four uniform supports, nine Zipf exponents, five 50%–99.99% heavy-head/diffuse-tail mixtures, eight trials, and eight-way merges.
A raw occurrence estimator exposed 22.76% p95 error for a 99% heavy head. Computing recovered-heavy entropy directly and sampling only conditional residual entropy reduced it to 0.32%. Entropy was bit-identical across left-fold and balanced merge trees in all 576 evaluations.
Validation
cargo test --all-targets: 505 library tests plus all integration/example targets passcargo test --doc: 20 passcargo clippy --all-targets -- -D warnings: passdocs/univmon_q_evaluation.mdanddocs/api/api_univmon_q.md