Add robust similarity metrics and normalization to compare.text - #1144
Conversation
|
@ebhills @thomasstvr PR was trested in QA to test it you can use image: dev-1.20.0rc59 . test file |
|
|
||
|
|
||
| def test_recipe_by_production_version(): | ||
| def test_recipe_by_production_version(mocker): |
There was a problem hiding this comment.
@mborodii-prog why are these tests being mocked? It looks like they should be able to be real tests.
It does look like the model id's do not exist, but they shouldn't be mocked.
There was a problem hiding this comment.
tests are mocked because after latest cleaning models list some of the models are deleted
There was a problem hiding this comment.
@thomasstvr I can create new model and update model_ids if it's better
| empty_b=empty_b, | ||
| all_empty=all_empty, | ||
| case_sensitive=case_sensitive, | ||
| case_sensitive=False, |
There was a problem hiding this comment.
Why are we removing case_sensitive?
Removing it and reversing the default to false are both breaking changes.
There was a problem hiding this comment.
@thomasstvr This is in requirements of issue, but we still support old syntax. case_sensitive remains accepted but is ignored, produces a deprecation warning when supplied, and is removed from new examples.
2 participants
Add robust similarity metrics and normalization to
compare.textCloses #1134
Summary
Adds
method: similaritytocompare.text— a symmetric,0.0–1.0bounded similarity score for comparing product descriptions that may be reordered, differently punctuated, or contain typos — while leavingdifference/intersection/overlapoutput shapes unchanged for backward compatibility.What's in this PR
method: similaritywith three metrics (wrangles/compare.py):token_sort(default) — ignores token order, keeps duplicates, penalizes missing/extra contenttoken_set— ignores token order and duplicates; a shorter description fully contained in a longer one can score1.0damerau_levenshtein— character edit distance where an adjacent transposition counts as one editfuzz.token_sort_ratio,fuzz.token_set_ratio,distance.DamerauLevenshtein.normalized_similarity) — new dependency:rapidfuzz>=3.0,<4.0normalize_similarity_text): NFKC Unicode normalization → Unicode-aware case folding → punctuation/separators converted to spaces (not concatenated) → whitespace collapsed. Deliberately does not equateAB-12/AB12, doesn't touch units/numbers, doesn't do synonym/acronym expansion.null, never stringified into"nan"/"None"— checked before anystr()conversion, unlike the legacy.astype(str)path.case_sensitivedeprecated: still accepted (won't break old recipes), value is ignored, logs a deprecation warning only when explicitly supplied.true/false/omitted now produce identical (always case-insensitive) results.contrast()/overlap()casing: previously, case-insensitive matching also lowercased the output indifference/overlap. Now matching is case-insensitive but original casing is preserved in the output (needed once case-insensitive became the permanent default).decimal_placesbug (int(decimal_places)'s result was discarded, causing a laterTypeError).difference/intersection/overlap(includinginclude_ratio) are unchanged in behavior/output shape — all pre-existing tests pass unmodified.Tests
35 pre-existing tests pass unchanged + 23 new tests in
tests/recipes/wrangles/test_compare.py: symmetry, score bounds, exact-match, reordered/duplicate/subset tokens, conflicting attributes, each Damerau-Levenshtein edit type individually (insertion/deletion/substitution/transposition), anagram guard, punctuation (AB-12vsAB12), Unicode normalization, nulls (both library-levelNoneand recipe-level non-stringified), the two regression bugs, and thecase_sensitivedeprecation warning/equivalence.