feat(theta): add jaccard similarity

[hawkingrei]

Summary

• Add configured Jaccard similarity operators for Theta and Tuple sketches.
• Return a shared result with lower bound, estimate, and upper bound.
• Compare Tuple sketches by retained hash keys only, independent of summary values or types.
• Cover zero-retained estimation-mode sketches without claiming exact equality.

Motivation

This closes a parity gap with the Apache DataSketches Jaccard implementations while keeping seed validation and Theta-family set semantics consistent with the existing Rust operators.

Related implementation: apache/datasketches-cpp Jaccard similarity

Implementation Notes

• Introduce ThetaJaccardSimilarity and TupleJaccardSimilarity as configured operators; the seed is selected when constructing the operator.
• Share the key-only union/intersection and confidence-bound implementation across Theta and Tuple sketches.
• Add iter_hashes to the internal Theta-family view so Tuple similarity does not clone summary payloads.
• Require a non-empty retained union before taking the exact-equality fast path. Logically non-empty sketches with no retained entries return the conservative [0.0, 0.5, 1.0] result.
• Keep result fields private and expose read-only accessors.
• Add cross-language reference vectors for the binomial proportion bounds.

Tests

All required repository workflows pass locally:

• cargo x prepare-testdata
• cargo x check
• cargo x test
• cargo x lint

The test suite covers Theta and Tuple mutable/compact inputs, different Tuple summary values and types, exact and estimation modes, custom seeds, seed mismatches, and zero-retained uncertainty.

[ZENOTME]

We also miss exactly_equal but it's ok to postpone it at next PR.

[tisonkun]

We must not have a unit struct ThetaJaccardSimilarity for creating JaccardSimilarity.

Let me consider the API and review the helper functions location.

Typically, here are a few API issues that should be resolved:

1. Should have JaccardSimilarity::between / JaccardSimilarity::between_with_seed, rather than ThetaJaccardSimilarity::jaccard / ThetaJaccardSimilarity::jaccard_with_seed.
2. JaccardSimilarity should expose its internal fields with getter, not pub fields, because the fields are immutatble.

[hawkingrei]

Addressed the API feedback from @tisonkun:

• Removed the ThetaJaccardSimilarity unit struct.
• Moved the public constructors to JaccardSimilarity::between(...) and JaccardSimilarity::between_with_seed(...).
• Made JaccardSimilarity fields private and exposed immutable getters: lower_bound(), estimate(), and upper_bound().

The tests have been updated to use the new API.

[tisonkun]

@ZENOTME I have two questions:

1. Java's ThetaSketch::getRetainedEntries has a valid param, but our CompactThetaSketch::num_retained doesn't. Could you describe a bit what the design difference between the Java impl and the Rust impl?

2. This PR has a fake ThetaUnion impl. Did we have an equivalent ThetaUnion impl so far? I found it strange to have a pub(super) struct ThetaUnion;.

[ZENOTME]

1. Java's ThetaSketch::getRetainedEntries has a valid param, but our CompactThetaSketch::num_retained doesn't. Could you describe a bit what the design difference between the Java impl and the Rust impl?

Java has a specific update implementation, AlphaSketch, where theta can be reduced without immediately rebuilding the internal cache. This means some cached entries may become invalid under the new theta but remain in the cache until a later rebuild. Therefore, getRetainedEntries(true) returns the valid entries used for estimation, while getRetainedEntries(false) returns the raw internal cache count, including possibly invalid entries.
Personally, I do not think this is an ideal public API shape. num_retained / getRetainedEntries should mean the logical retained count used for estimation and set operations. If the raw cache count needs to be exposed, it would be clearer as a separate API.

2. This PR has a fake ThetaUnion impl. Did we have an equivalent ThetaUnion impl so far? I found it strange to have a pub(super) struct ThetaUnion;.

We don't have ThetaUnion so far. I think it maybe better to implement it first.

I send a PR for it. #145 cc @tisonkun @hawkingrei

[ZENOTME]

I think this implementation should also using RawThetaJaccardSimilarity to share between tuple and theta.

[tisonkun]

@hawkingrei @ZENOTME @ariesdevil You may coordinate to see how to move forward this feature.

[ariesdevil]

As @ZENOTME mentioned, this PR should rebase first, then do the same thing as union and intersection.

[tisonkun]

Or simply creating a new one.

I'd leave @hawkingrei this week for doing the (logical) rebase work, or else anyone can pick it up.

[hawkingrei]

Updated the branch onto current main (head: 419be13).

The Jaccard implementation now follows the same shape as the C++ jaccard_similarity_base:

• it composes the existing shared UnionState and IntersectionState rather than keeping a separate set-operation implementation;
• the common raw layer is generic over ThetaFamilySketchView, so Tuple can reuse it when its Jaccard API is added;
• tests moved into the current theta_test layout and now assert an accurate estimate plus ordered confidence bounds.

Validated locally with cargo test -p datasketches --features theta --test theta_test and cargo clippy -p datasketches --features theta --test theta_test -- -D warnings. The all-feature serialization suite requires the generated Java/Go/C++ fixture set (cargo x prepare-testdata), which is not present locally.

Could you please take another look at the updated API and shared-operation structure?

[hawkingrei]

@ZENOTME Implemented your shared raw-layer direction in 419be13.

RawThetaJaccardSimilarity is now in thetacommon and generic over ThetaFamilySketchView; the Theta API is only a thin wrapper. The raw implementation composes the existing shared UnionState and IntersectionState, matching the C++ jaccard_similarity_base structure rather than maintaining independent set-operation logic.

This makes the common layer ready for Tuple reuse when its public Jaccard API is added, without expanding this PR beyond Theta. Could you please review the updated structure?

[tisonkun]

We should implement JaccardSimilarity for tuple sketch to see what should be extract to thetacommon.

This patch is a good starting point while I may have issues with:

• RawThetaJaccardSimilarity should be renamed to JaccardSimilarity follows other methods.
• RawThetaJaccardSimilarity's fields should be private.
• between_with_seed looks weird. I suppose we should follow ANotBOperator's design.

[hawkingrei]

Implemented the requested Jaccard design changes in aeec2fd:

• added TupleJaccardSimilarity, including different summary-type coverage;
• renamed the shared raw result to JaccardSimilarity and kept its fields private;
• replaced between_with_seed with configured Theta/Tuple operators;
• made the shared implementation key-only so Tuple summaries are not cloned or compared;
• fixed the zero-retained exact-equality edge case and added Theta/Tuple regressions;
• removed the redundant test feature gate;
• fixed the existing htmp typo and added cross-language bound reference vectors.

Local validation passed:

• cargo x prepare-testdata
• cargo x check
• cargo x test
• cargo x lint

The two remaining inline review threads have been answered and resolved. Please take another look when convenient.

[hawkingrei]

Thank you @tisonkun, @ariesdevil, and @ZENOTME for the careful review and design guidance.

I incorporated the feedback in aeec2fd:

• added the Tuple Jaccard implementation to validate the shared extraction;
• renamed the shared result to JaccardSimilarity and kept its fields private;
• followed the configured-operator shape used by A-not-B for seed handling;
• made the common path key-only, so Tuple summary values and types do not affect similarity;
• fixed the logically non-empty, zero-retained edge case and added Theta/Tuple regressions;
• reused the shared union/intersection states and kept exactly_equal outside this PR as previously suggested.

The repository-required local workflows (cargo x check, cargo x test, and cargo x lint) pass, and the GitHub CI jobs are now running. All existing inline threads have been addressed and resolved.

Could you please take another look when convenient? Thanks again for helping improve the API and abstraction boundary.

[tisonkun]

This is now mergable.

But we need to support other methods like:

• exactly_equal
• similarity_test
• dissimilarity_test

... later.