Skip to content

DuckDB projection collisions can silently drop nonempty columns #527

Closed

Description

@WilliamK112

Current DuckDB native cleaning can silently remove a nonempty column when case-insensitive registration names collide with aliases created by column normalization. This is distinct from the dtype-audit issue #465 and was reproduced while addressing the final-name mapping review on #524.

import pandas as pd
import freshdata as fd

frame = pd.DataFrame({
    "A": [1, 2, 3],
    "a": [None, None, None],
    "A 1": pd.Series([4, 5, 6], dtype="Int64"),
})
cleaned, report = fd.clean(
    frame, engine="duckdb", strategy="conservative", fix_dtypes=False,
    return_report=True, verbose=False,
)
print(cleaned)
print(report.columns_dropped)

Actual: only the first column remains; columns_dropped is ['a_1', 'a_1'], and the nonempty [4, 5, 6] column is gone. The pandas reference drops only the truly empty column and retains the third column's values.

Registration exposes A, a_1, and A 1. The cleaning projection creates two a_1 aliases, but later stages keep using that duplicated pre-projection name. Empty-column counting therefore counts the second column twice and drops both source positions. Related collisions can also select the second column twice when an unrelated empty column is removed.

I am preparing a small preservation fix with pandas-reference parity and fallback-policy coverage, after the in-progress #524 review validation completes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions