Skip to content

Count-Min: write serialization version 2, keep reading version 1 - #576

Merged
proost merged 2 commits into
masterfrom
count-min-v2
Nov 17, 2026
Merged

proost merged 2 commits into
masterfrom
count-min-v2

Conversation

@leerho

@leerho leerho commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

This implements version 2 of the Count-Min format, as specified in docs/count-min-format.md in datasketches-tck. The spec was adopted by lazy consensus in apache/datasketches-tck#18 and merged in apache/datasketches-tck#21.

Closes #555.

Changes

  • Row seeds: rowSeed[i] = XXH64(LE64(i), seed) (§3). Version 1 drew the row seeds from std::default_random_engine and std::uniform_int_distribution, whose results differ between libstdc++, libc++ and the MSVC STL. Version 1 images were therefore read with the wrong buckets on another standard library, with no error (count_min_sketch hash seeds depend on the C++ standard library, so serialized sketches are not portable #555).
  • Hashing: items are hashed with XXH64, and bucket = (hash >> 1) mod numBuckets (§5). Integers are hashed as 8 little-endian bytes, and strings as their bytes, as before (§4).
  • Versions:
    • New sketches write serial version 2.
    • Version 1 images are still read. A sketch read from one keeps the version 1 hashing and is written back as version 1 (§6).
    • Sketches of different versions can't be merged.
    • Any other serial version is rejected.
    • The new get_serial_version() reports the version.
  • numHashes: must be in [1, 127] (§1). Before, 0 was accepted, and estimates then returned the maximum value of W. Version 1 images written by C++ may have up to 255 rows, so they are still read. suggest_num_hashes() now returns at most 127.
  • Empty items: ignored (§4.4). An update with a size of 0, including (nullptr, 0), does nothing. Every query (estimate, lower bound and upper bound) returns 0 for an empty item. The std::string overloads already behaved this way.
  • Header check: preamble longs, serial version and family ID are each checked on their own, as in Java. Before, only a weighted sum of them was checked, so some wrong headers passed.
  • Docs: the class documentation describes the two versions, notes that only W = int64_t images are readable in other languages, and states the empty-item rule.

New sketches hash items differently from before. Estimates from the same inputs can therefore change, within the usual error bounds. Version 1 images keep their behavior.

Tests

  • count_min_v2_test.cpp:

    • the spec's test vectors: XXH64 (§7.1), row seeds (§7.2), buckets for seeds 9001 and 42 (§7.3), and both images, byte for byte (§7.4);
    • the empty-item rule, checked on a non-empty sketch so that the upper bound is covered;
    • the numHashes range, in the constructor and in version 2 images;
    • version 1 images: read, written back unchanged, still usable, and not mergeable with version 2. This uses a version 1 image written by the current code;
    • rejection of other serial versions;
    • rejection of headers with a wrong preamble longs, serial version or family ID, including combinations the old weighted-sum check accepted;
    • copies keeping their version.
  • Cross-language files (§8):

    • with GENERATE, count_min_serialize_for_java.cpp writes the seven count_min_v2_<case>_cpp.sk files;
    • with SERDE_COMPAT, count_min_deserialize_from_java_test.cpp runs the §8.2 checks on the Java files and requires each one to be byte-identical to the C++ sketch for the same case.

    All seven cases are byte-identical to the files generated by datasketches-java main.

  • CM frequency cancellation: this test assumed that "x" and "y" land in different buckets of a 1 × 5 sketch. Under version 2 hashing they share a bucket, so the test now uses 6 buckets.

  • Old file names: the tests no longer write count_min_empty_cpp.sk and count_min_non_empty_cpp.sk. §8.3 reserves those names for version 1 images, and C++ no longer writes version 1. The next TCK snapshot update will show them removed.

The Count-Min tests pass under ASan and UBSan, with and without SERDE_COMPAT. The integration test and the Count-Min benchmark build. This PR touches only count/, and it merges cleanly with all open PRs.

🤖 Generated with Claude Code

Implements version 2 of the Count-Min format specification in
datasketches-tck (docs/count-min-format.md), so that images give the same
results in every language and with every C++ standard library.

- Row seeds are XXH64(LE64(i), seed) instead of values drawn from
  std::default_random_engine, which differs between standard libraries.
- Items are hashed with XXH64, and bucket = (hash >> 1) mod numBuckets.
- New sketches write serial version 2. Version 1 images are still read;
  such sketches keep the version 1 hashing and are written back as
  version 1. Sketches of different versions can't be merged.
- numHashes must be in [1, 127]; version 1 images may have up to 255.
  suggest_num_hashes() returns at most 127.
- Empty items are ignored: an update with size 0 does nothing, and every
  query returns 0, as the string overloads already did.
- New get_serial_version().

Tests check the specification's test vectors, the empty-item rule, the
numHashes range, version 1 images, and the cross-language files of
section 8 (generated with GENERATE, and compared byte for byte with
Java's with SERDE_COMPAT). The tests no longer write the version 1 file
names count_min_empty_cpp.sk and count_min_non_empty_cpp.sk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested review from proost and tisonkun November 12, 2026 22:05
@coveralls

coveralls commented Nov 12, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 43950927667

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Warning

No base build found for commit 386f335 on master.
Coverage changes can't be calculated without a base build.
If a base build is processing, this comment will update automatically when it completes.

Coverage: 96.911%

Details

  • Patch coverage: 46 of 46 lines across 1 file are fully covered (100%).

Uncovered Changes

No uncovered changes found.

Coverage Regressions

Requires a base build to compare against. How to fix this →


Coverage Stats

Coverage Status
Relevant Lines: 16866
Covered Lines: 16345
Line Coverage: 96.91%
Coverage Strength: 1643867.03 hits per line

💛 - Coveralls

Comment thread count/include/count_min_impl.hpp Resolved
The header check compared a weighted sum of the preamble longs, serial
version, family ID and empty flag against a list of valid values, so some
wrong combinations passed, for example serial version 0 with family ID 19.
Check each field separately, as Java does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested a review from proost November 16, 2026 21:50

@proost proost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@proost
proost merged commit f7937b4 into master Nov 17, 2026
35 checks passed
@proost
proost deleted the count-min-v2 branch November 17, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

count_min_sketch hash seeds depend on the C++ standard library, so serialized sketches are not portable

3 participants