Duplicate column labels crash detect_pii/anonymize, fd.validate and explain_clean #265

Summary

fd.clean accepts DataFrames with duplicate column labels, but these functions call df[col] and receive a DataFrame instead of a Series, then fail with AttributeError/TypeError. #210 covered the same class for fill_missing (PR #219).

Affected functions

• detect_pii and anonymize crash with AttributeError on duplicate column labels (src/freshdata/enterprise/privacy.py:477-478)
• fd.validate crashes with AttributeError on frames with duplicate column labels (src/freshdata/enterprise/contracts.py:683)
• explain_clean and infer_roles crash on duplicate column labels that fd.clean accepts (src/freshdata/explain.py:26)

Environment

freshdata 2.0.0; found at 55a8044 and re-checked on main c87efbd (2026-09-15). Reproduces on Python 3.9.6 / pandas 1.5.3 / numpy 1.26.4 and Python 3.12.14 / pandas 2.3.3 / numpy 2.5.3.

Suggested fix

• Either reject duplicate labels up front with a clear ValueError, or iterate positionally (df.iloc[:, i]) in each function listed.
• Add one parametrised duplicate-label test across these APIs.

Details

1. detect_pii and anonymize crash with AttributeError on duplicate column labels

detect_pii and detection-driven anonymize loop over frame.columns and use frame[col].dtype. With a duplicated label, frame[col] returns a DataFrame, and both functions crash with an unhelpful AttributeError. fd.clean and fd.profile accept such frames. #210 reports the same class of crash in fd.fill_missing.

Reproduction

import pandas as pd
from freshdata.enterprise import detect_pii, anonymize, PIIDetectionConfig

df = pd.DataFrame([["a@b.com", "c@d.com"]], columns=["email", "email"])
detect_pii(df)
anonymize(df, detection_config=PIIDetectionConfig())

Expected

Both columns are scanned (positional iteration), or a clear ValueError names the duplicated labels.

Actual

ACTUAL: {'detect_pii': "AttributeError: 'DataFrame' object has no attribute 'dtype'", 'anonymize(detection)': "AttributeError: 'DataFrame' object has no attribute 'dtype'"}

Where

• src/freshdata/enterprise/privacy.py:477-478 (detect_pii) and :1149-1150 (_anonymize_detected): series = frame[col] / if series.dtype != object ...

2. fd.validate crashes with AttributeError on frames with duplicate column labels

fd.validate with a ValidationSuite raises AttributeError when the frame has duplicate column labels, even if the rule targets a different, unique column. fd.clean and fd.profile accept such frames.

Reproduction

import pandas as pd, freshdata as fd

df = pd.DataFrame([[1, 2, 3]], columns=["age", "age", "id"])
fd.validate(df, suite=fd.ValidationSuite(name="s", rules=[fd.ColumnRule("id", min_value=0)]))

Expected

Either a validation finding reporting the duplicate column labels, or a clear ValueError.

Actual

AttributeError: 'DataFrame' object has no attribute 'name'

Where

src/freshdata/enterprise/contracts.py:683 (_profile_column): name = str(series.name). For a duplicated label, frame[col] returns a DataFrame, not a Series.

3. explain_clean and infer_roles crash on duplicate column labels that fd.clean accepts

fd.clean and fd.profile accept frames with duplicate column labels. fd.explain_clean raises AttributeError on them and fd.infer_roles raises TypeError, because df[col] returns a DataFrame for a duplicated label.

Reproduction

import pandas as pd, freshdata as fd

df = pd.DataFrame([[1.0, 2.0], [None, 4.0], [3.0, 5.0]], columns=["a", "a"])
fd.clean(df, verbose=False); fd.profile(df)   # ok
fd.explain_clean(df, verbose=False)           # AttributeError
fd.infer_roles(df)                            # TypeError

Expected

Handled the way fd.clean/fd.profile handle it, or a clear ValueError naming the duplicated label.

Actual

ACTUAL: {'explain_clean': "AttributeError: 'DataFrame' object has no attribute 'dtype'", 'infer_roles': "TypeError: cannot convert the series to <class 'int'>"}

Where

• src/freshdata/explain.py:26 (_column_stats): "dtype": str(s.dtype) where s = df[col]
• src/freshdata/engine/context.py:258: n_missing = int(s.isna().sum())