[choksi2212]

Fixes #8888

PandasArrayExtensionDtype.__from_arrow__ materialized every fixed-shape ArrayXD column through ArrayExtensionArray.to_numpy, which pads the null rows with NaN and promotes the whole column to float64. Two consequences at the pandas boundary: the Arrow row validity was gone (a null row and a legitimate all-NaN array became indistinguishable), and integer columns lost their exact payloads, with values above 2**53 rounding and Dataset.from_pandas(..., features=...) raising ArrowInvalid when converting the NaN rows back to int64.

A fixed-shape column with null rows is now materialized as one exact ndarray per valid row and None per null row, the representation the dynamic shape already uses, so Dataset.from_pandas(ds.to_pandas(), features=ds.features) keeps the validity bitmap and the exact values:

features = Features({"x": Array2D((1, 1), "float64")})
source = Dataset.from_dict({"x": [[[np.nan]], None]}, features=features)
result = Dataset.from_pandas(source.to_pandas(), features=features)
assert result.data.column("x").is_null().to_pylist() == [False, True]

The pandas dtype now also carries the fixed shape and the value type, so a column whose rows are all null still rebuilds its extension type (on main that round trip silently turned the nulls into valid all-NaN rows), and the dtype is carried through copy, slicing, take, _concat_same_type and _from_sequence. isna() used to crash on None rows (pd.isna(None).any()); it now counts scalar None/NaN rows as null without inspecting them, while ndarray rows keep the existing NaN inspection. Dynamic shapes and fixed shapes without null rows keep their previous representations, so the null-row regression from #8701 and the NumPy formatting behavior kept by #8363 are untouched.

The new tests fail on main and pass here: validity round trips for (1,1) and (2,3) columns with null rows in the middle, exact int64 payloads above 2**53, a sliced column, and an all-null column. Verified locally on the issue's pins (pandas 3.1.0, pyarrow 26.0.0, numpy 2.5.4, Python 3.13): tests/features/test_array_xd.py (111), tests/features/test_features.py, tests/test_table.py (598 passed together), and ruff check/ruff format are clean.

[choksi2212]

CI note: this is the first PR from my fork, so the CI and Build PR Documentation workflows are sitting in action_required until someone approves the first run. Everything in the description is verified locally on the issue's pinned versions (pandas 3.1.0, pyarrow 26.0.0, numpy 2.5.4): the five new test cases fail on main and pass on this branch, and the full test_array_xd.py, test_features.py and test_table.py suites are green with ruff check/ruff format clean.

[sjdoane]

At 9bdf3b3f8d6af10fd78cb6d537159e19024e5e1b, an explicit-feature round trip can still leave the declared fixed shape inconsistent with the underlying Arrow extension type:

from datasets import Array2D, Dataset, Features

features = Features({"x": Array2D((2, 3), "float64")})
source = Dataset.from_dict(
    {"x": [[[0., 1., 2.], [3., 4., 5.]], None]}, features=features
)
result = Dataset.from_pandas(source.to_pandas(), features=features)
assert result.data.column("x").type.shape == result.features["x"].shape
# PR: (None, 3) != (2, 3)

On base ef61228, both shapes are (2, 3); its existing null-validity bug remains. This PR preserves the values and null bitmap in this example.

_to_arrow_extension_type() uses the saved _dtype.shape only when there is no valid ndarray. With a valid row, it still chooses (None, *first_arr.shape[1:]). The mixed valid/null case also needs the saved fixed shape. A regression should inspect the actual Arrow .shape: extension-type equality alone misses this mismatch.

Independently reproduced with Codex-assisted tests on Python 3.12.14, pandas 3.1.0, PyArrow 26.0.0 and NumPy 2.5.4.