fix(dtypes): a complex value in an object column no longer crashes cleaning

[kevincostner17]

Bug

A single complex number in an object column crashes fd.clean on default settings:

fd.clean(pd.DataFrame({"v": [complex(1, 2), "abc", "3"]}))
# TypeError: ufunc 'remainder' not supported for the input types ...

The same happens for a complex value among numeric strings, or among Python numbers in an object column (Python complex, numpy.complex128, numpy.complex64). It reproduces on py3.12 / pandas 2.3.3 and py3.9 / pandas 1.5.3.

Cause — pandas returns uninitialised memory

Once one cell is complex, pd.to_numeric(s, errors="coerce") returns a complex buffer and never writes the text, bytes or bool cells into it. Those cells come back as whatever the last freed buffer of that size held:

s = pd.Series([complex(1, 2), "abc", "not a number"], dtype=object)
buf = np.full(3, 7+7j, dtype=np.complex128); del buf
pd.to_numeric(s, errors="coerce")   # [(1+2j), (7+7j), (7+7j)]

This reproduces on both pandas 2.3.3 and 1.5.3. So "abc" could read as 0j, 7+7j or NaN, and the column passed or failed numeric_threshold depending on what had run earlier in the process. When it passed, the finalizer's integrality check (nonnull % 1 == 0) raised. This is also the mechanism behind a characterization test that was previously order-dependent for reasons that couldn't be identified.

Fix

Every numeric target in steps/dtypes.py is a real dtype, so a complex value is an unparseable straggler, like a word or a Fraction in the same position. In _to_numeric_or_none, when pandas returns a complex result, complex cells are masked before re-parsing (the garbage cannot be filtered out afterwards). The existing paths then handle them: the threshold count, quarantine into coerced_cells, and _warn_type_contamination. Columns without a complex value never take the new path.

Tests

• tests/test_exotic_values_in_numeric_columns.py, 22 tests. They assert what happens to the complex value (preserved, recorded in coerced_cells / coerced_rows, the warning text, parity with Fraction), not just "no exception". Two of them seed the freed buffer so the garbage is deterministic.
• tests/test_dtypes_boolean_mutants.py: two characterization tests that pinned the crash are converted into tests that it no longer crashes.
• 17 fail without the src/ change, on both interpreters. 34 pass with it. Also 285 passed under each of 5 pytest-randomly seeds alongside the neighbouring dtypes and numeric tests.
• Full suite: py3.12 7561 passed; py3.9 7529 passed; 0 failed on both.

Surveyed, not changed here

• A native complex128 column of 4+ rows crashes in the outlier step (steps/outliers.py, s.quantile). That's a different mechanism, which I'll follow up on separately.
• safe_to_numeric's other callers still receive pandas' garbage for complex input. No wrong public-API result has been demonstrated yet.

Compatibility impact

Only columns holding a complex value change, and each such column previously raised (or, depending on process state, came back unchanged). Columns without a complex value parse exactly as before.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: b4ca7c5d-fab2-4939-8fbe-afb40f851568

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)