fix(text): idempotent Unicode cleaning, fewer false encoding alerts, international phones and punycode emails

[kevincostner17]

Summary

• clean_text_value is now idempotent and in the configured Unicode form (clean_text_value output is not NFC or idempotent when a zero-width char precedes an accent #316). Normalization ran before zero-width/control-character removal, so "Jose" + U+200B + U+0301 came out decomposed and a second pass changed it. The configured normalization now re-runs after the removal and custom steps (before whitespace collapse, strip and truncate); the transform name is recorded once.
• Fewer false lint_text_encoding alerts (lint_text_encoding flags ordinary Japanese, Spanish and Portuguese text as high severity #317). Modifier letters (category Lm, e.g. ー, 々) and the ordinal indicators ª/º no longer count as their own script, and KATAKANA-HIRAGANA/IDEOGRAPHIC characters join the Japanese family, so コーヒー, 人々 and Nº 5 are no longer mixed_script. The Ã + ASCII-space mojibake pattern was dropped because it matched uppercase Portuguese (MANHÃ DE SOL); the NBSP form is still detected.
• International phones and punycode emails in validate_fields (validate_fields rejects valid '+49 (0) 30 ...' phone numbers and punycode-TLD emails #318). Phone validation now allows up to 30 formatting characters and checks for 7-15 digits (E.164), so +49 (0) 30 12345678 passes. The same rule applies in detect_value_type, the per-value check and the vectorised suspect scan. Email TLDs may be xn-- punycode.

Behaviour changes

• Phones with more than 15 digits are now rejected (previously only a 17-character shape limit applied).
• Ã followed by a plain space is no longer reported as mojibake.
• Follow-up, not in this PR: semantic/semantic_types.py keeps its own phone/email patterns for type inference.

Tests

New tests/test_text_quality_fixes.py:

• repro for each issue;
• idempotence and normal-form checks over tricky inputs (zero-width chars, BOM, bidi marks or control chars before combining marks, NBSP, fullwidth letters) for NFC/NFD/NFKC/NFKD;
• guards that real mojibake (école, don’t, São Paulo, the NBSP form) and real mixed script (Cyrillic/Greek in Latin text) are still detected;
• valid and invalid phone numbers (too few digits, 16 digits, letters, too long) and emails (a@b.c, a@b.xn--, x@y), checked through both the scalar and the vectorised path.

Verification

• ruff check .: clean
• mypy src/freshdata: clean (202 files)
• pytest -m "not online and not large" on Python 3.12: 4379 passed, 6 skipped
• pytest -m "not online and not large" on Python 3.9: 4375 passed, 10 skipped
• 50,000 random inputs: clean(clean(x)) == clean(x) with default settings

Closes #316
Closes #317
Closes #318

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 9b79c27d-3366-4939-a3eb-42cdbbc88c0d

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)