Repository navigation
[Python] Coredump when joining big large_strings #33151
Description
joining large strings in pyarrow results in this error:
terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_default_append
Aborted (core dumped) example code:
note that this needs quite some ram (run on 128GB)
import pyarrow as pa
ids = [x for x in range(2**24)]
text = ['a'*2**10]*2**24
schema = pa.schema([
('Id', pa.int32()),
('Text', pa.large_string()),
])
tab1 = pa.Table.from_arrays([ids, text],schema=schema)
tab2 = pa.Table.from_arrays([ids, text],schema=schema)
joined = tab1.join(tab2, keys='Id', right_keys='Id', left_suffix='tab1') the same results in a segfault, if i use this schema
schema = pa.schema([
('Id', pa.int32()),
('Text', pa.string()),
])
Environment: run inside a fedora container:
registry.fedoraproject.org/fedora-toolbox:36
host information:
uname -a:
Linux ws1 5.18.16-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 3 15:44:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
/etc/os-release:
NAME="Fedora Linux"
VERSION="36 (Container Image)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora Linux 36 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
Reporter: flowpoint
Note: This issue was originally created as ARROW-17943. Please see the migration documentation for further details.
Activity
asfimport commentedon Oct 8, 2022
Yibo Cai / @cyb70289:
The code below triggers same error log. Try it online: https://onlinegdb.com/UpqUsk4Zv
Looks this might be caused by integer overflow which leads to a huge buffer size greater than
std::vector::max_size().asfimport commentedon Oct 8, 2022
Yibo Cai / @cyb70289:
Error comes from below line,
total_length = -2128609280https://github.com/apache/arrow/blob/master/cpp/src/arrow/compute/kernels/row_encoder.cc#L344
asfimport commentedon Oct 10, 2022
Yibo Cai / @cyb70289:
@michalursa, do you have comments? change
offsets_fromint32toint64?asfimport commentedon Oct 10, 2022
flowpoint:
cool, with this tip i think i got it to work for now for my usecase
this is ofc. not a true solution
https://gist.github.com/flowpoint/08e76e9a90544009b298e5bea9219236
asfimport commentedon Oct 11, 2022
Yibo Cai / @cyb70289:
Glad to hear that
[~flowpoint].Since you already have a patch (thought not a true solution), I think it might be worth to submit a PR for review.
idailylife commentedon Dec 8, 2023
Any update on this issue? The patch above doesn't seem to be merged:(
zanmato1984 commentedon Oct 22, 2026
This report covers two separate paths:
large_string: Basic Hash Join’s build-payload offsets overflow near 2 GiB. This remains unresolved and shares the root cause of [C++][Python] pyarrow.Table.join() fatal crash forlarge_stringkeys near 2gb #54885.string: Swiss Join’s separate 4 GiB row-offset limit was removed by GH-43495: [C++][Compute] Widen the row offset of the row table to 64-bit #43389. An equivalent C++ regression fails before that change and passes afterward.Closed as a duplicate of #54885 to consolidate the remaining Basic Join problem, not because the entire report is fixed.
large_stringkeys near 2gb #54885