Repository navigation
[BUG]: test_mempool_trim crashes intermittently on Windows (suspected driver race) #3350
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.core
Describe the bug
test_mempool_trim, added in #3236, intermittently crashes the Python 3.15t, rtx2080 (WDDM) job (driver 596.36) with an access violation at device.sync(). It crashed in 3 of the 11 runs I checked, including
one on an unrelated PR:
I think this is a driver race. The attached C program uses only driver calls: threads allocate and free from their own pools on the legacy default stream, call cuCtxSynchronize, and destroy their pools. On Linux (RTX 5070 Ti, driver 615.78.08) it segfaults or hangs within seconds, with a NULL dereference inside cuCtxSynchronize. I can't run it on Windows, so the link to the CI crash is inferred.
It fails with 3 or more threads. It runs clean with 2 threads, with per-thread or private streams, or when pools are never destroyed.
I suggest marking the test thread_unsafe and sending the reproducer to the driver team. I can open the PR once this is assigned.
cc @Andy-Jost
How to Reproduce
gcc -O1 -g -I/opt/cuda/include mempool_sync_race.c -o mempool_sync_race -lcuda -lpthread
./mempool_sync_race # legacy default stream: segfault or hang
./mempool_sync_race thread # per-thread default stream: runs clean
Note: this reproducer was written by Claude Opus 5.5 under my direction.
Expected behavior
cuCtxSynchronize should not crash or hang while other threads use the legacy default stream and destroy their own pools.
Operating System
7.3.0-4-cachyos; CI/CD: windows-x64
nvidia-smi output
Another process is using my GPU at the moment.
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 615.78.08 KMD Version: 615.78.08 CUDA UMD Version: 13.4 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5070 Ti Off | 00000000:01:00.0 On | N/A |
| 30% 52C P1 232W / 300W | 5221MiB / 16303MiB | 99% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1230 G /usr/lib/Xorg 33MiB |
| 0 N/A N/A 6842 G Hyprland 194MiB |
| 0 N/A N/A 6894 G Xwayland 4MiB |
| 0 N/A N/A 6908 G /usr/bin/hyprpaper 36MiB |
| 0 N/A N/A 6910 G ...prpolkitagent/hyprpolkitagent 10MiB |
| 0 N/A N/A 7079 G ...rack-uuid=3190708988185955192 10MiB |
| 0 N/A N/A 8360 G /usr/bin/kitty 63MiB |
| 0 N/A N/A 11269 G /usr/bin/hyprlauncher 19MiB |
| 0 N/A N/A 1174864 G /usr/lib/firefox/firefox 49MiB |
| 0 N/A N/A 1583906 C .venv/bin/python 4504MiB |
+-----------------------------------------------------------------------------------------+
Activity
ci: serialize pool trim test for known driver synchronization issue
ci: serialize pool trim test for known driver synchronization issue
test(cuda.core): serialize pool trimming around the driver race
jpascucci-nv commentedon Nov 4, 2026
We caught sight of this in nightly CI.
It failed on https://github.com/NVIDIA/cuda-python/actions/runs/41743621440/job/126296584137 and https://github.com/NVIDIA/cuda-python/actions/runs/41743621440/job/126296584405, and hung on https://github.com/NVIDIA/cuda-python/actions/runs/41743621440/job/126295962803
These runs were from before the test was marked thread_unsafe, so we'll see if that fixes them in the next run.
srimanachanta commentedon Nov 4, 2026
@jpascucci-nv got it, thanks! Do you know if this bug is being investigated by the driver team or should I put in a bug ticket? I assume it would be fixed faster if it came from an internal employee?
rwgk commentedon Nov 4, 2026
— codex GPT-6 on behalf of Ralf —
For tracking: while bringing #3352’s CI to green, Codex autonomously added a
thread_unsafemarker totest_mempool_trim. This is now merged.The test hit the same access violation at
device.sync()twice on Windows/Python 3.14t/CUDA 13.0.2/A100 MCDM. After serialization, both the DeviceMR and PinnedMR cases executed their assertions and passed (successful job).This is a CI mitigation: the test still runs, but it no longer exercises this lifecycle concurrently. The underlying suspected race remains unresolved, and similar failures could still occur elsewhere. Green CI therefore does not establish that the root issue is fixed.
My next step is to check whether the no-IPC native reproducer here should be added to nvbug 6763003, or whether separate tracking is needed. That ticket’s documented reproducer involves IPC imports, so we should not assume these are duplicates. We should also preserve the distinction between the reproduced native Linux failure and its still-inferred connection to the Windows CI crashes.
To keep the underlying issue visible, we should:
Please keep #3350 open through that follow-up. The merged workaround stabilizes CI while the underlying investigation continues.
@juenglin @Andy-Jost for visibility.