Skip to content

feat: AoS sketch - #550

Merged
proost merged 4 commits into
apache:masterfrom
proost:feat-aos-sketch
Nov 19, 2026
Merged

proost merged 4 commits into
apache:masterfrom
proost:feat-aos-sketch

Conversation

@proost

@proost proost commented Oct 17, 2026

Copy link
Copy Markdown
Member

issue: apache/datasketches-go#191

revert from reverted PR and I added fixed encoding logic following go.

@proost
proost requested a review from leerho October 17, 2026 07:44
@proost proost self-assigned this Oct 17, 2026

@leerho leerho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Hyeonho, and thanks for porting the Go fixes. The key hashing now matches Java: joined with ,, XXH64 with seed 0x7A3CCA71 over UTF-16LE code units, and invalid UTF-8 replaced by the same rules as apache/datasketches-go#197. I merged this into current master locally and ran the tuple tests with SERDE_COMPAT against the Java-generated aos_* files, under ASan and UBSan. All 15 AoS test cases pass, including aos_unicode and the Java hash test.

A few requests:

  1. total_bytes isn't checked against the bytes actually read. The bytes overload only checks that it fits in the buffer, and the stream overload ignores it. Java's ArrayOfStringsSummary asserts pos == totBytes. Could both overloads throw if the bytes read for an item differ from total_bytes?
  2. The class comment says other implementations enforce UTF-8 and that non-UTF-8 strings will fail to deserialize there. Java decodes values with new String(bytes, UTF_8), which replaces invalid bytes with U+FFFD instead of failing, and Go doesn't validate. Could you reword it to say values are stored as raw bytes, and that Java replaces invalid UTF-8 with U+FFFD when it reads them?
  3. Minor: rethrowing based on the text "size exceeds 127" in the exception message is fragile. Checking the node count before the try, or using a dedicated exception, would be more robust.

Not for this PR: the stream overload allocates length bytes before reading them, as serde<std::string> already does. We should bound that across the library separately.

5.3.1 is in its vote now, so this will be merged after it closes.

@leerho
leerho requested a review from tisonkun November 12, 2026 00:14
@proost

proost commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

@leerho
Those points makes sense to me.

2486103

@proost
proost requested a review from leerho November 13, 2026 07:01

@leerho leerho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Hyeonho. All three points are addressed:

  1. Both deserialize() overloads now check total_bytes against the bytes actually read for each item, and throw std::invalid_argument on a mismatch, as Java's ArrayOfStringsSummary does.
  2. The comments now say that strings are stored as raw bytes, and that Java replaces invalid UTF-8 with U+FFFD when it reads them.
  3. check_num_nodes() throws std::invalid_argument, and the handlers catch that type instead of matching the message text.

I merged this into current master locally and ran tuple_test with SERDE_COMPAT and the Java-generated files, under ASan and UBSan: all 72 test cases pass, including the 15 AoS cases. Against the previous commit's code, the new "total_bytes mismatch rejected" and "too many nodes rejected on deserialize" sections fail, as they should.

One thing to remove before this merges: the last commit adds tuple/test/java, a symlink to /Users/lani/workspace/datasketches-cpp/java_generated_files on your machine. The tests don't use it (they read from serialization_test_data/java_generated_files), so it looks like it was committed by accident. Once it's gone, I'll approve.

@proost
proost requested a review from leerho November 18, 2026 05:49

@leerho leerho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Hyeonho. The symlink is gone, and the code is the same as the version I reviewed, apart from the merge of master. CI is green, so approving.

@proost
proost merged commit 5f446e1 into apache:master Nov 19, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants