[BUG]: cuda.core tests fail under compute-sanitizer memcheck (use-after-free reports, kernelNsysProfile timeout) #3401

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

Sanitizer test runs have been effectively disabled since #2437: no row in ci/test-matrix.yml currently passes the sanitizer gate in ci/tools/env-vars (Linux, Python 3.12, LOCAL_CTK: '1'). #3397 fixes the sanitizer setup and uses the build CTK's compute-sanitizer (13.5.0) for all sanitizer runs. The probe PR #3400 enabled three amd64 Python 3.12 local-CTK rows (CUDA 12.9.1, 13.0.2, 13.5.0) on top of #3397, in run 42271588808. The cuda.bindings tests are clean under the sanitizer, but the cuda.core tests fail on all three rows:

Job logs: 12.9.1, 13.0.2, 13.5.0. The same failures occurred on 13.5.0 in the earlier run 42261639818, so they do not depend on the CTK version under test. Without the sanitizer, all three tests pass (or skip) on the other rows with the same CTK versions.

Use-after-free reports. Both tests report a 4-byte allocation at the same device address, freed with cudaFreeAsync (from cuda.core's _buffer module, i.e. Buffer.close) and later accessed with cudaMemset/cudaMemcpy (cuMemsetD8/cuMemcpyDtoH from cuda.bindings.driver). Excerpt from test_device_context_requires_initialization_on_new_thread:

========= Use-after-free on allocation of size 4 bytes at 0x190e000000
=========     Address 0x190e000000 is potentially accessed after it is free'd
=========     Saved host backtrace up to driver entry point at cudaMemset time
=========         Host Frame: worker in test_device.py:96
=========     Saved host backtrace up to driver entry point at cudaFreeAsync time
=========         Host Frame:  [0x1cf67] in _buffer.cpython-312-x86_64-linux-gnu.so
=========         Host Frame: worker in test_device.py:94

followed by a second report for the cudaMemcpy at test_device.py:98. In that test, line 94 is with device.allocate(4, stream=stream) as buffer: and the accesses at lines 96 and 98 are inside the with block, so in program order they precede the buffer's free. The test_program.py reports have the same shape (same address and size, cudaFreeAsync from _buffer, then cudaMemcpy). Possible explanations to check: the sanitizer attributes the free of an earlier allocation at the same memory-pool address and misses the re-allocation (a false positive with stream-ordered pool reuse), or there is a genuine ordering problem between the stream-ordered free and the legacy-stream accesses.

kernelNsysProfile timeout. The sample runs in a subprocess, which --target-processes=application-only should leave uninstrumented, yet it hits the 300 s sample timeout under the sanitizer and passes without it. It is also the single largest cost of a sanitizer job (about 5 of the 8 minutes of cuda.core tests).

How to Reproduce

In CI (after #3397 merges): set LOCAL_CTK: '1' on an amd64 Python 3.12 row of the linux.pull-request matrix in ci/test-matrix.yml, as #3400 did, and open a PR. The sanitizer runs in the "Run cuda.core tests" step of that job.

Locally on Linux with an NVIDIA GPU (CI used an L4 with driver 595.99.02):

1. Install a CTK that includes compute-sanitizer 13.5.0, or a mini CTK plus the 13.5.0 cuda_sanitizer_api redist component, and put its bin on PATH.

2. Install cuda-pathfinder, cuda-bindings, and cuda-core from this repository with their test dependencies (for example pip install -e . --group test in each package directory).

3. From cuda_core/, run the affected tests with the command line CI uses:

   SAN="compute-sanitizer --target-processes=application-only --launch-timeout=0 --tool=memcheck --error-exitcode=1 --report-api-errors=no --padding=32"
   $SAN pytest -v tests/test_device.py::test_device_context_requires_initialization_on_new_thread
   $SAN pytest -v tests/test_program.py::test_nvrtc_automatic_pch_preserves_compiled_kernel
   $SAN pytest -v "tests/example_tests/test_samples.py::test_sample[cuda_core/kernelNsysProfile]"

   Whether the use-after-free reports reproduce with the tests in isolation has not been verified; they may depend on memory-pool state left by earlier tests. If they do not, run the whole suite as CI does: $SAN pytest -rsxXfE -v --randomly-dont-reorganize tests/.

Expected behavior

The cuda.core test suite passes under compute-sanitizer --tool=memcheck with ERROR SUMMARY: 0 errors, so that sanitizer rows can be re-enabled in ci/test-matrix.yml. If a report turns out to be a sanitizer false positive, it should be suppressed or the test adjusted, with a note explaining why.

Operating System

Linux (CI container on linux-amd64-gpu-l4-latest-1)

nvidia-smi output

| NVIDIA-SMI 595.99.02              Driver Version: 595.99.02      CUDA Version: 13.2     |
|   0  NVIDIA L4                      On  |   00000000:0A:00.0 Off |                    0 |