Repository navigation
[C++][Python] Casting extension types with list-based storage segfaults #37004
Description
Describe the bug, including details regarding any error messages, version, and platform.
I'm taking a table and casting it to its own schema. This should always work, but it errors in a certain case.
(this is deep in some library code - please forgive how strange this scenario is; just trust me that it really does happen!)
# An extension type which wraps list-of-int32:
class MyExtensionType(pa.PyExtensionType):
def __init__(self):
pa.PyExtensionType.__init__(self, pa.list_(pa.int32()))
def __reduce__(self):
return MyExtensionType, (),
# an array of this type:
ext_array = pa.array([[1, 2, 3]], MyExtensionType())
# Wrap that in a struct:
struct_array = pa.StructArray.from_arrays([ext_array], "x")
# Wrap the struct in a table:
table = pa.Table.from_arrays([struct_array], ["field1"])All of the above is fine, and then:
table.cast(table.schema)Blows up:
File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/table.pxi:3616, in pyarrow.lib.Table.cast()
File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/table.pxi:3798, in pyarrow.lib.Table.from_arrays()
File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/table.pxi:2962, in pyarrow.lib.Table.validate()
File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()
ArrowInvalid: Column 0: In chunk 0: Invalid: Struct child array #0 invalid: Invalid: Expected 1 child arrays in array of type list<item: int32>, got 0
I can print table just fine:
pyarrow.Table
field1: struct<x: extension<arrow.py_extension_type<MyExtensionType>>>
child 0, x: extension<arrow.py_extension_type<MyExtensionType>>
----
field1: [
-- is_valid: all not null
-- child 0 type: extension<arrow.py_extension_type<MyExtensionType>>
[[1,2,3]]]
This is pyarrow version 12.0.1.
Component(s)
Python
Activity
spenczar commentedon Aug 4, 2023
This seems to happen as well for FixedShapeTensor tensor types, which indicates it happens for fixed-size list types.
AlenkaF commentedon Aug 17, 2023
I can reproduce the first issue but cannot reproduce it for
FixedShapeTensorarray:I am not sure if the issue is in the struct or the underlying list type. But it seems to me that the casting of the column in the
table.castis behaving erroneousarrow/python/pyarrow/table.pxi
Line 3694 in 4fc2731
and I think this should work but it segfaults:
spenczar commentedon Aug 17, 2023
Thanks for taking a look @AlenkaF. Here is a reproducer for FixedShapeTensor - it only happens if you go through the struct. The issue is mis-titled. It's actually a bug with Tables which contain a Struct field which have an ExtensionType which has a ListTyped-storage (whew, that's a lot!):
NellyWhads commentedon Nov 21, 2024
Is this related to #34985?
AlenkaF commentedon May 8, 2026
I asked Copilot to go through the backtrace and I checked the result. It does seem that, in both cases, when casting an extension type with list-based storage, the cast kernel produces
ArrayDatawith the correct type but drops the child data. When the extension type then tries to wrap this into anExtensionArrayand callsMakeArray, it tries to construct a list array from that incompleteArrayDataand crashes.In case that helps anybody starting to look for a fix. Will also update the title to reflect the underlying bug.
[-][Python] pyarrow.Table.cast does not handle extension types which wrap list types[/-][+][C++][Python] Casting extension types with list-based storage segfaults[/+]connorsimms commentedon Jul 13, 2026
Hello! I've looked into this issue and I've diagnosed the cause in
array/data.cc. I've got a fix with C++/Python tests, and i've also checked it against the Python snippets in this thread. If no one is working on this, I'd be more than happy to open a PR :).connorsimms commentedon Jul 13, 2026
take
apacheGH-37004: [C++][Python] Fix dropped child data when viewing/cas…
GH-37004: [C++][Python] Fix dropped child data when viewing/casting e… (
pitrou commentedon Jul 22, 2026
Issue resolved by pull request 50502
#50502