Segfaults from the bundled mimalloc v3 when the first thread exits and when co-loaded on macOS #151

Summary

aerospike-native 1.0.0 sets mimalloc::MiMalloc as the Rust global allocator (src/lib.rs). The locked mimalloc 0.1.52 / libmimalloc-sys 0.1.49 bundles mimalloc v3.3.2 (MI_MALLOC_VERSION 30302), which has two known bugs that kill the Python process with SIGSEGV:

1. All platforms: if the first thread that used aerospike_native exits, the next allocation on another thread segfaults in mi_thread_init (mimalloc >= 3.3.0 causes segmentation faults when used from multiple threads microsoft/mimalloc#1287, fixed in mimalloc 3.4.1).
2. macOS only: when another library in the same process bundles its own mimalloc v3, e.g. pyarrow 24.0.0, the two copies share a fixed TLS slot and corrupt each other's heap (Apple FIXED_SLOT causes SIGABRT when multiple statically-linked instances coexist — no cflag can select THREAD_LOCAL microsoft/mimalloc#1301, [Python] pyarrow 24.0.0 regression: co-loading a native extension bundling mimalloc v3 SIGSEGVs in bundled mimalloc at interpreter teardown (macOS arm64, py3.14); 22/23 unaffected apache/arrow#50428).

Neither needs an Aerospike server to reproduce. Building with the v2 feature of the mimalloc crate fixed both in our tests (see "Suggested fix").

Environment

• aerospike-native 1.0.0 wheels from PyPI
• macOS 27.0.1 arm64 (Apple M5 Pro), Python 3.11.15 and 3.14.5
• Linux aarch64 and x86_64 in python:3.11-slim and python:3.12-slim containers

1. Segfault after the first thread exits (all platforms)

import threading

mods = {}


def load():
    import aerospike_native

    mods["an"] = aerospike_native


t = threading.Thread(target=load)
t.start()
t.join()  # the thread that first used aerospike_native has exited
an = mods["an"]


def work():
    for _ in range(300):
        [an.Key("ns", "s", f"k{j}") for j in range(50)]
        an.ClientPolicy()


threads = [threading.Thread(target=work) for _ in range(8)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print("OK")

Exit code 139 (SIGSEGV) in 3 of 3 runs on every platform above. Importing aerospike_native on the main thread first avoids it (5 of 5 runs OK on macOS and Linux x86_64). pyarrow is not involved; a script that does the same with only pyarrow 24.0.0 / 25.0.1 / 26.0.0 runs fine.

Backtrace on macOS (lldb, addresses removed):

* thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x18)
  * frame #0: _native.cpython-311-darwin.so`mi_thread_init + 484
    frame #1: _native.cpython-311-darwin.so`_mi_malloc_generic + 760
    frame #2: _native.cpython-311-darwin.so`mi_theap_malloc_zero_aligned_at_generic + 428
    frame #3: _native.cpython-311-darwin.so`pyo3::types::boolobject::_$LT$impl$u20$pyo3..conversion..FromPyObject$u20$for$u20$bool$GT$::extract::h3b6aefcc5697dcb3 + 1292
    frame #4: _native.cpython-311-darwin.so`_$LT$_native..record..PythonValue$u20$as$u20$pyo3..conversion..FromPyObject$GT$::extract::h0e5ab896cec38955 + 120
    frame #5: _native.cpython-311-darwin.so`_native::record::Key::__pymethod___new____::h6a02330659df4f20 + 228

This matches microsoft/mimalloc#1287: mimalloc 3.3.0 to 3.3.2 treats the first thread that initializes it as the main thread, and allocations on other threads crash after that thread exits. It is fixed in mimalloc 3.4.1. pyarrow 25.0.0 shipped the same bug and fixed it in 25.0.1 by moving to mimalloc 3.4.1 (apache/arrow#50471, same mi_thread_init / _mi_malloc_generic / mi_theap_malloc_zero_aligned_at_generic frames and fault address 0x18).

In real code this happens, for example, when the first use is inside a ThreadPoolExecutor that is shut down and a later ThreadPoolExecutor uses aerospike_native again (SIGSEGV in 3 of 3 runs on macOS arm64 and Linux x86_64). Using it from the main thread after the first pool is gone did not crash in our runs; the crash needs a new non-main thread.

2. Segfault next to pyarrow 24.0.0 (macOS only)

import aerospike_native as an
import pyarrow as pa

for _ in range(20000):
    policy = an.ClientPolicy()
    policy.max_conns_per_node = 100
    keys = [an.Key("ns", "s", f"k{j}") for j in range(50)]
    pa.array(list(range(10000)))
print("OK")

The native stack shows _mi_malloc_generic recursing until the stack overflows. Changing the import order or setting ARROW_DEFAULT_MEMORY_POOL=system does not reliably avoid it.

On Apple platforms mimalloc v3 keeps its thread-local heap pointer in a fixed TLS slot that every loaded image shares, without checking which copy owns it (microsoft/mimalloc#1301, microsoft/mimalloc#1302). mimalloc v2 only used the fixed slot in MI_MALLOC_OVERRIDE builds. pyarrow 25.0.1 builds its mimalloc with thread-local variables on macOS (apache/arrow#50428, apache/arrow#50549), so this case is gone with newer pyarrow. aerospike_native still uses the fixed slot, though, so it can collide with any other extension that bundles mimalloc v3 with default settings. apache/arrow#50428 was reported with another PyO3 extension that uses mimalloc v3 as its global allocator.

Suggested fix

Use mimalloc v2 until a mimalloc crate release bundles mimalloc 3.4.1 or later. The latest release, mimalloc 0.1.52 / libmimalloc-sys 0.1.49, still bundles v3.3.2.

mimalloc = { version = "0.1", features = ["v2"] }

With this one-line change, local builds of v1.0.0 passed every reproduction above, while the same builds without the change kept crashing:

The macOS builds used maturin build --release with Python 3.11. The Linux build used the ghcr.io/pyo3/maturin image and was tested in python:3.11-slim. The pyarrow column also covers the reversed import order, pandas string arrays and 8 threads.

One unrelated thing we ran into: on macOS 27 our local build with the v2 change and the default strip = "debuginfo" failed to load with mis-aligned LINKEDIT string pool from dyld. Building with CARGO_PROFILE_RELEASE_STRIP=none avoided it. The released 1.0.0 wheel and our stripped build without the change loaded fine, so this may be specific to our toolchain (rustc 1.96.0), but it may be worth checking that the macOS wheels load on recent macOS.

Other PyO3 projects worked around the same problem by switching to mimalloc v2 (see the commits linked from microsoft/mimalloc#1287). Dropping the custom global allocator is another option.

References

• mimalloc >= 3.3.0 causes segmentation faults when used from multiple threads microsoft/mimalloc#1287 (crash after the first initializing thread exits, fixed in 3.4.1)
• Apple FIXED_SLOT causes SIGABRT when multiple statically-linked instances coexist — no cflag can select THREAD_LOCAL microsoft/mimalloc#1301 and add MI_APPLE_TLS_THREAD_LOCAL option to use THREAD_LOCAL on macOS microsoft/mimalloc#1302 (fixed TLS slot shared by all mimalloc v3 copies on Apple)
• [C++][Python] SIGSEGV in bundled mimalloc mi_thread_init when libarrow is first loaded on a non-main thread that exits (mimalloc 3.3.x, pyarrow 25.0.0) apache/arrow#50471 (the same thread-exit crash in pyarrow 25.0.0, fixed in 25.0.1)
• [Python] pyarrow 24.0.0 regression: co-loading a native extension bundling mimalloc v3 SIGSEGVs in bundled mimalloc at interpreter teardown (macOS arm64, py3.14); 22/23 unaffected apache/arrow#50428 and GH-50428: [C++] Better mimalloc configuration on macOS apache/arrow#50549 (pyarrow 24.0.0 crash next to another mimalloc v3 extension on macOS, fixed in 25.0.1)