[Python] read_row_group fails with Nested data conversions not implemented for chunked array outputs

Hey, I'm trying to concatenate two files and to avoid reading everything to memory at once, I wanted to use read_row_group for my solution, but it fails.

I think it's due to fields like these:

pyarrow.Field<to: list<item: string>>

But I'm not sure. Is this a duplicate? The issue linked in the code is resolved

arrow/cpp/src/parquet/arrow/reader.cc

Line 915 in fd0b90a

// ARROW-3762(wesm): If inout_array is a chunked array, we reject as this is

Stacktrace is

File "/data/teftel/teftel-data/teftel_data/parquet_stream.py", line 163, in read_batches
table = pf.read_row_group(ix, columns=self._columns)
File "/home/kuba/.local/share/virtualenvs/teftel-o6G5iH_l/lib/python3.6/site-packages/pyarrow/parquet.py", line 186, in read_row_group
use_threads=use_threads)
File "pyarrow/_parquet.pyx", line 695, in pyarrow._parquet.ParquetReader.read_row_group
File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs

Reporter: Jakub Okoński

Related issues:

[C++] Support nested data conversions for chunked array (is duplicated by)
[C++][Parquet] 16MB limit on (nested) column chunk prevents tuning row_group_size (relates to)
[C++] Parquet arrow::Table reads error when overflowing capacity of BinaryArray (relates to)

_{Note: This issue was originally created as ARROW-5030. Please see the migration documentation for further details.}

@wesm

Wes McKinney / @wesm:
I fixed some cases where this occurs in ARROW-4688, but it is still possible to hit this error for very large row groups (> 2GB of string data in a row group). I didn't see a follow up JIRA to this or ARROW-3762 so we can use this one for the issue

https://github.com/apache/arrow/blob/master/cpp/src/parquet/arrow/reader.cc#L915

Judah:
[~wesm_impala_7e40] I'm also running into this issue. Is this likely to be fixed / easy to fix? I'd be happy to give it a go but not really sure where to start.

In case someone wants to load a pandas dataframe I want to share my workaround.

For me installing fastparquet and specifying the eninge='fastparquet' argument in the load_parquet function worked.

In case someone wants to load a pandas dataframe I want to share my workaround.

For me installing fastparquet and specifying the eninge='fastparquet' argument in the load_parquet function worked.

Concurring. If the parquet file contains a dictionary/list/struct, then the following

import pandas as pd
df = pd.read_parquet(parquet_path)

throws an error "ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs"

But only if the parquet file is over 1 GB. If it is under 1 GB, then it loads with no problems.

df = pd.read_parquet(parquet_path, engine='fastparquet') should do the magic.
Please ensure you have the fastparquet package installed already.

[Data] Fix read_parquet() ArrowNotImplementedError for nested column …

chore(backend): check total string column chunk sizes (#12647)

Still reproduces in 26.0.0

Navigation Menu

Repository navigation

[Python] read_row_group fails with Nested data conversions not implemented for chunked array outputs #21526

Related issues:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python] read_row_group fails with Nested data conversions not implemented for chunked array outputs #21526

Description

Related issues:

Activity

asfimport commented on Mar 28, 2019

asfimport commented on Apr 27, 2022

Maxl94 commented on May 22, 2024

yuxi-liu-wired commented on Sep 14, 2024

sudarsun commented on Oct 30, 2025

thisisnic commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions