fix: a repeated index label no longer breaks cleaning

[kevincostner17]

Bug

fd.clean(df) and fd.suggest_plan(df) raise on ordinary input whenever the frame's index repeats a label — the shape you get from concatenating two exports without reset_index. Default settings, no unusual options:

a = pd.DataFrame({"amount": [f"{i}.00" for i in range(30)]})
b = pd.DataFrame({"amount": [f"{i}.00" for i in range(29)] + ["$1,234.56"]})
fd.clean(pd.concat([a, b]))
# ValueError: cannot set using a list-like indexer with a different length than the value

Two steps selected rows by index label, and .loc expands a repeated label to every row carrying it:

site	code	error
steps/dtypes.py formatted-number rescue	parsed.loc[rescued.index] = rescued.to_numpy()	ValueError — left side longer than the values
engine/missing.py coercion-casualty scan	df[col].loc[rows].isna()	IndexError: boolean index did not match indexed array

How the sites were found

Not by patching the first crash. 17 public-API calls were each run on identical data twice — unique index vs. every label repeated — and any difference counted. Exactly these two sites differed; suggest_plan reaches the first one too. The other .loc[ sites in src/ take boolean masks or already have a non-unique fallback.

Fix

Both sites now work by position — the approach the undo-log revert (report.py), the guard (guard.py), and fieldcheck already take for the same reason.

The index itself is deliberately left alone. The first attempt ran the whole pipeline on a RangeIndex and restored labels afterwards (as validate_domain does). It fixed both crashes — and silently dropped a real row: steps/duplicates.py reads isinstance(df.index, pd.DatetimeIndex) to decide that repeated timestamps are real observations rather than duplicates, and replacing the index disarmed that rule. test_timeseries_duplicates_preserved_even_when_removal_requested caught it. engine/context.py reads the index type as well. So the index carries meaning, and replacing it would change decisions, not just mechanics. Withdrawn in favour of the narrower fix.

Tests

tests/test_duplicate_index.py — every test is a pair: identical data under a unique index and a repeated one, asserting the two agree. That keeps the tests meaningful if the implementation changes again.

• 8 of 11 fail without the src/ change, on both py3.12 / pandas 2.3.3 and py3.9 / pandas 1.5.3. The other 3 are unique-index controls.
• The fixture's unparseable value is "not a date", not a date-like string: "5 March 2021" was tried first and is a casualty only on pandas 2 (which infers one format per column); pandas 1.x parses it, so the casualty-scan test passed vacuously there. Found by running the full suite on py3.9.
• tests/test_dtypes_boolean_mutants.py: the characterization test that pinned this bug as "currently breaks" is converted to "no longer breaks".

Verification

Full suite (-m "not online and not large") on the branch rebased onto bdff821:

• py3.12 / pandas 2.3.3: 7420 passed, 0 failed
• py3.9 / pandas 1.5.3: 7388 passed, 0 failed — the first clean end-to-end py3.9 run in this program; the previous venv lacked the [dev] extra, so 14 modules failed collection

ruff check clean.

Compatibility impact

None for a unique index — that path is unchanged. Frames with repeated labels that previously raised now clean, and produce the same result as the identical frame with a unique index.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: cace5198-ecb2-483e-ab43-3ac86714c997

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)