[Python/C++] Empty column selection no longer restores index

@fjetter

The index of a dataframe is no longer reconstructed when using empty column selection. This is a regression to 0.12.1 and probably only happens for pd.RangeIndex

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from kartothek.serialization import ParquetSerializer
from storefact import get_store_from_url
print(pa.__version__)
df = pd.DataFrame(
    {"a": [1, 2]}
)
print(df.index)

table = pa.Table.from_pandas(df)
buf = pa.BufferOutputStream()
pq.write_table(table, buf)
reader = pa.BufferReader(buf.getvalue().to_pybytes())
table_restored = pq.read_pandas(reader, columns=[])
df_restored = table_restored.to_pandas()
print(len(df_restored))

Reporter: Florian Jetter / @fjetter

Related issues:

[Python] RangeIndex serialization change implications (is related to)

_{Note: This issue was originally created as ARROW-5139. Please see the migration documentation for further details.}

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
@fjetter thanks for the report! A little bit simpler reproducible example, without parquet (but the same underlying reason, the rangeindex is indeed not constructed again for empty tables):

In [1]: import pyarrow as pa

In [2]: pa.__version__  
Out[2]: '0.12.0'

In [3]: df = pd.DataFrame( 
   ...:     {"a": [1, 2]} 
   ...: ) 

In [4]: table = pa.Table.from_pandas(df, columns=[], preserve_index=True)     

In [5]: table
Out[5]: 
pyarrow.Table
__index_level_0__: int64
metadata
--------
OrderedDict([(b'pandas',
              b'{"index_columns": ["__index_level_0__"], "column_indexes": ['
              b'{"name": null, "field_name": null, "pandas_type": "unicode",'
              b' "numpy_type": "object", "metadata": {"encoding": "UTF-8"}}]'
              b', "columns": [{"name": null, "field_name": "__index_level_0_'
              b'_", "pandas_type": "int64", "numpy_type": "int64", "metadata'
              b'": null}], "pandas_version": "0.23.4"}')])

In [6]: print(table.to_pandas())
Empty DataFrame
Columns: []
Index: [0, 1]

In [7]: table.to_pandas().index
Out[7]: Int64Index([0, 1], dtype='int64')

But the above, now gives:

In [4]: table 
Out[4]: 
pyarrow.Table

metadata
--------
{b'pandas': b'{"index_columns": [{"kind": "range", "name": null, "start": 0, "'
            b'stop": 2, "step": 1}], "column_indexes": [{"name": null, "field_'
            b'name": null, "pandas_type": "unicode", "numpy_type": "object", "'
            b'metadata": {"encoding": "UTF-8"}}], "columns": [], "creator": {"'
            b'library": "pyarrow", "version": "0.13.1.dev126+ga9ae4a9f.d201905'
            b'03"}, "pandas_version": "0.24.2"}'}

In [5]: print(table.to_pandas())   
Empty DataFrame
Columns: []
Index: []

In [6]: table.to_pandas().index    
Out[6]: RangeIndex(start=0, stop=0, step=1)

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
A "quick and dirty" fix would be to special case tables with zero rows and zero columns in the check in _reconstruct_index (

arrow/python/pyarrow/pandas_compat.py

Lines 695 to 697 in 982f341

    
           if len(index_level) != len(table): 
        
               # Possibly the result of munged metadata 
        
               continue

), when recreating the pandas DataFrame from an empty arrow Table.

But I think the proper fix to this would be to allow the Table to have a non-zero num_rows. However, this would need separate fixes depending on where the Table is created (so eg a separate fix in the Table.from_pandas(columns=[]) and the parquet read code, for both examples above), and complicate the code in several places, so I am not sure we want to go down that route. (cc @wesm ?)

@wesm

Wes McKinney / @wesm:
Added to 0.15.0 in case something can be done

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
Now we add the preserve_index=None / True distinction, the original example with preserve_index=True is working again as expected.

When doing the same example with the default of None (pa.Table.from_pandas(df, columns=[], preserve_index=None).to_pandas()), that still gives the same problem as reported.

For that one, I am not sure what is the best solution (do we want to go down the route of a Table having 0 columns but a non-zero num_rows ?)

@pitrou

Antoine Pitrou / @pitrou:
Do we want to act on this?

No longer reproduces, closing!

Navigation Menu

Repository navigation

[Python/C++] Empty column selection no longer restores index #21621

Related issues:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if len(index_level) != len(table):
	# Possibly the result of munged metadata
	continue

[Python/C++] Empty column selection no longer restores index #21621

Description

Related issues:

Activity

asfimport commented on May 6, 2019

asfimport commented on May 6, 2019

asfimport commented on Aug 22, 2019

asfimport commented on Sep 17, 2019

asfimport commented on Aug 5, 2021

thisisnic commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions