fix(quality): count detected duplicates and distinct PII columns in quality-debt dimensions

[kevincostner17]

Summary

Fixes two quality-debt dimensions in evaluate_quality_debt whose numbers didn't match their labels.

• duplicates (evaluate_quality_debt: duplicates dimension is always 0 with default drop_duplicates=False #264): the score used report.duplicates_removed, which is always 0 under the default drop_duplicates=False, so a frame with 50% duplicate rows scored 0.0. It now scores max(removed, remaining) / rows_before, where remaining = cleaned.duplicated().sum(). That matches the other dimensions, which describe the cleaned output, and still counts removals when drop_duplicates=True. Unhashable cells fall back to the removed count. The detail now reads "N duplicate row(s) detected (M removed)".
• pii_risk (Quality-debt pii_risk counts PII matches but labels them 'potential PII column(s)' #286): the PII scan records one finding per matching cell, but that count was used as a column count. It now counts distinct columns from scan.by_column(), skipping findings with no column, for both the score and the detail. A 10-row email column in a 2-column frame now reads "1 potential PII column(s)" with score 0.5 instead of "10 ..." / 1.0. The PII scanner itself is unchanged.

Behaviour change: frames with duplicates may now warn or fail the gate under default options; pii_risk scores drop for tall PII columns. docs/examples/quality_debt.html is not regenerated here. If regenerated, duplicates would read "5 duplicate row(s) detected (0 removed)" and pii_risk would change from 1.00 "177 potential PII column(s)" to 0.17 "1 potential PII column(s)", with gate status still WARN.

Limitations: detection compares whole rows (a duplicate_subset passed via clean options isn't visible to the scorer), and rows that only become identical after cleaning also count.

Tests

New tests/test_quality_debt_dimensions.py:

• evaluate_quality_debt: duplicates dimension is always 0 with default drop_duplicates=False #264 repro: detected-but-not-removed duplicates score 0.5 and fail the gate
• drop_duplicates=True still scores 0.5, detail shows 50 removed
• a frame without duplicates scores 0
• Quality-debt pii_risk counts PII matches but labels them 'potential PII column(s)' #286 repro: one email column counts as 1 PII column, score 0.5
• scan findings without a column key are ignored

Verification

• ruff check .: clean
• mypy src/freshdata: no issues (202 files)
• Python 3.12: 4250 passed, 6 skipped, 12 deselected (-m "not online and not large")
• Python 3.9: 4246 passed, 10 skipped, 12 deselected

Closes #264
Closes #286

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e30a98ac-bd2e-45d6-bc4f-259fe487203d

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)