Repository navigation
Count-Min: write serialization version 2, keep reading version 1 - #576
Merged
Merged
Conversation
Implements version 2 of the Count-Min format specification in datasketches-tck (docs/count-min-format.md), so that images give the same results in every language and with every C++ standard library. - Row seeds are XXH64(LE64(i), seed) instead of values drawn from std::default_random_engine, which differs between standard libraries. - Items are hashed with XXH64, and bucket = (hash >> 1) mod numBuckets. - New sketches write serial version 2. Version 1 images are still read; such sketches keep the version 1 hashing and are written back as version 1. Sketches of different versions can't be merged. - numHashes must be in [1, 127]; version 1 images may have up to 255. suggest_num_hashes() returns at most 127. - Empty items are ignored: an update with size 0 does nothing, and every query returns 0, as the string overloads already did. - New get_serial_version(). Tests check the specification's test vectors, the empty-item rule, the numHashes range, version 1 images, and the cross-language files of section 8 (generated with GENERATE, and compared byte for byte with Java's with SERDE_COMPAT). The tests no longer write the version 1 file names count_min_empty_cpp.sk and count_min_non_empty_cpp.sk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage Report for CI Build 43950927667Warning Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes. Warning No base build found for commit Coverage: 96.911%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsRequires a base build to compare against. How to fix this → Coverage Stats
💛 - Coveralls |
proost
reviewed
Nov 14, 2026
The header check compared a weighted sum of the preamble longs, serial version, family ID and empty flag against a list of valid values, so some wrong combinations passed, for example serial version 0 with family ID 19. Check each field separately, as Java does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements version 2 of the Count-Min format, as specified in docs/count-min-format.md in datasketches-tck. The spec was adopted by lazy consensus in apache/datasketches-tck#18 and merged in apache/datasketches-tck#21.
Closes #555.
Changes
rowSeed[i] = XXH64(LE64(i), seed)(§3). Version 1 drew the row seeds fromstd::default_random_engineandstd::uniform_int_distribution, whose results differ between libstdc++, libc++ and the MSVC STL. Version 1 images were therefore read with the wrong buckets on another standard library, with no error (count_min_sketch hash seeds depend on the C++ standard library, so serialized sketches are not portable #555).bucket = (hash >> 1) mod numBuckets(§5). Integers are hashed as 8 little-endian bytes, and strings as their bytes, as before (§4).get_serial_version()reports the version.numHashes: must be in [1, 127] (§1). Before, 0 was accepted, and estimates then returned the maximum value ofW. Version 1 images written by C++ may have up to 255 rows, so they are still read.suggest_num_hashes()now returns at most 127.(nullptr, 0), does nothing. Every query (estimate, lower bound and upper bound) returns 0 for an empty item. Thestd::stringoverloads already behaved this way.W = int64_timages are readable in other languages, and states the empty-item rule.New sketches hash items differently from before. Estimates from the same inputs can therefore change, within the usual error bounds. Version 1 images keep their behavior.
Tests
count_min_v2_test.cpp:numHashesrange, in the constructor and in version 2 images;Cross-language files (§8):
GENERATE,count_min_serialize_for_java.cppwrites the sevencount_min_v2_<case>_cpp.skfiles;SERDE_COMPAT,count_min_deserialize_from_java_test.cppruns the §8.2 checks on the Java files and requires each one to be byte-identical to the C++ sketch for the same case.All seven cases are byte-identical to the files generated by datasketches-java main.
CM frequency cancellation: this test assumed that"x"and"y"land in different buckets of a 1 × 5 sketch. Under version 2 hashing they share a bucket, so the test now uses 6 buckets.Old file names: the tests no longer write
count_min_empty_cpp.skandcount_min_non_empty_cpp.sk. §8.3 reserves those names for version 1 images, and C++ no longer writes version 1. The next TCK snapshot update will show them removed.The Count-Min tests pass under ASan and UBSan, with and without
SERDE_COMPAT. The integration test and the Count-Min benchmark build. This PR touches onlycount/, and it merges cleanly with all open PRs.🤖 Generated with Claude Code