Skip to content

keep nullable integers intact in to_pandas - #8890

Open
Chessing234 wants to merge 1 commit into
huggingface:mainfrom
Chessing234:fix/pandas-nullable-ints
Open

Chessing234 wants to merge 1 commit into
huggingface:mainfrom
Chessing234:fix/pandas-nullable-ints

Conversation

@Chessing234

Copy link
Copy Markdown

an int64 column with a null was converted to float64. float64 cannot hold every int64, so 2**62 + 1 came back as a different number. to_pandas and the pandas formatter now keep those nulls as missing values, the same way to_csv / to_json / to_sql already do. a column with no nulls stays int64.

checked locally: [2**62 + 1, None, 3] round-trips through to_pandas(), the batched path, and with_format("pandas").

@lhoestq

Made with Cursor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant