Repository navigation
feat: AoS sketch - #550
feat: AoS sketch #550
Conversation
leerho
left a comment
There was a problem hiding this comment.
Thanks, Hyeonho, and thanks for porting the Go fixes. The key hashing now matches Java: joined with ,, XXH64 with seed 0x7A3CCA71 over UTF-16LE code units, and invalid UTF-8 replaced by the same rules as apache/datasketches-go#197. I merged this into current master locally and ran the tuple tests with SERDE_COMPAT against the Java-generated aos_* files, under ASan and UBSan. All 15 AoS test cases pass, including aos_unicode and the Java hash test.
A few requests:
total_bytesisn't checked against the bytes actually read. The bytes overload only checks that it fits in the buffer, and the stream overload ignores it. Java'sArrayOfStringsSummaryassertspos == totBytes. Could both overloads throw if the bytes read for an item differ fromtotal_bytes?- The class comment says other implementations enforce UTF-8 and that non-UTF-8 strings will fail to deserialize there. Java decodes values with
new String(bytes, UTF_8), which replaces invalid bytes with U+FFFD instead of failing, and Go doesn't validate. Could you reword it to say values are stored as raw bytes, and that Java replaces invalid UTF-8 with U+FFFD when it reads them? - Minor: rethrowing based on the text
"size exceeds 127"in the exception message is fragile. Checking the node count before thetry, or using a dedicated exception, would be more robust.
Not for this PR: the stream overload allocates length bytes before reading them, as serde<std::string> already does. We should bound that across the library separately.
5.3.1 is in its vote now, so this will be merged after it closes.
leerho
left a comment
There was a problem hiding this comment.
Thanks, Hyeonho. All three points are addressed:
- Both
deserialize()overloads now checktotal_bytesagainst the bytes actually read for each item, and throwstd::invalid_argumenton a mismatch, as Java'sArrayOfStringsSummarydoes. - The comments now say that strings are stored as raw bytes, and that Java replaces invalid UTF-8 with U+FFFD when it reads them.
check_num_nodes()throwsstd::invalid_argument, and the handlers catch that type instead of matching the message text.
I merged this into current master locally and ran tuple_test with SERDE_COMPAT and the Java-generated files, under ASan and UBSan: all 72 test cases pass, including the 15 AoS cases. Against the previous commit's code, the new "total_bytes mismatch rejected" and "too many nodes rejected on deserialize" sections fail, as they should.
One thing to remove before this merges: the last commit adds tuple/test/java, a symlink to /Users/lani/workspace/datasketches-cpp/java_generated_files on your machine. The tests don't use it (they read from serialization_test_data/java_generated_files), so it looks like it was committed by accident. Once it's gone, I'll approve.
leerho
left a comment
There was a problem hiding this comment.
Thanks, Hyeonho. The symlink is gone, and the code is the same as the version I reviewed, apart from the merge of master. CI is green, so approving.
issue: apache/datasketches-go#191
revert from reverted PR and I added fixed encoding logic following go.