[BUG][pathfinder] Wheel layout order can override site-packages root priority #3369

Is this a duplicate?

• I checked for duplicate issues and agree to the Code of Conduct.

Type of Bug

Silent Failure

Component

cuda.pathfinder

Describe the bug

Site-packages roots are ordered correctly for a venv --system-site-packages, but several finders search one wheel layout across all roots before trying the next layout. As a result, a CUDA 13 layout in a lower-priority user or system site-packages can override a CUDA 12 layout in the virtual environment. The same inversion can happen between user and system site-packages.

This is related to #1716 and #1717: those changes order the roots when searching a single layout, but the outer loop over alternative layouts can still reverse that priority. Dynamic libraries, headers, binaries, static libraries and bitcode libraries are affected.

How to Reproduce

From a cuda-python checkout on Linux, using only empty placeholder files (no NVIDIA library or GPU required):

WORK_DIR=$(mktemp -d)
python3 -m venv --system-site-packages "$WORK_DIR/venv"
export PYTHONUSERBASE="$WORK_DIR/user"
mkdir -p "$("$WORK_DIR/venv/bin/python" -m site --user-site)"
"$WORK_DIR/venv/bin/python" -m pip install -e ./cuda_pathfinder
"$WORK_DIR/venv/bin/python" - <<'PY'
import site
import sys
from pathlib import Path
from cuda.pathfinder import locate_bitcode_lib

venv = Path(site.getsitepackages()[0])
user = Path(site.getusersitepackages())
assert site.ENABLE_USER_SITE
assert sys.path.index(str(venv)) < sys.path.index(str(user))
for root, rel in (
    (venv, "nvidia/cuda_nvcc/nvvm/libdevice/libdevice.10.bc"),
    (user, "nvidia/cu13/nvvm/libdevice/libdevice.10.bc"),
):
    path = root / rel
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text("placeholder")
result = Path(locate_bitcode_lib("device").abs_path)
print("selected:", "venv" if result.is_relative_to(venv) else "user")
PY

On the tested upstream revision, this prints selected: user, even though the venv site appears first on sys.path. The placeholders are only used for locating files and are not valid bitcode.

Separately, with two CPU-only test shared libraries in the corresponding CUDA 12 venv and CUDA 13 user layouts, a fresh process calling load_nvidia_dynamic_lib("cudart") actually loaded the lower-priority user-site test library. This does not test real NVIDIA libraries or GPU behavior.

Expected behavior

Search all applicable layouts in the higher-priority site root before moving to the next root. Preserve the declared layout order when alternatives exist within the same root.

System information

Ubuntu 24.04.4 x86_64; Python 3.12.3; upstream main deb3d040b6d31b8514f19d026c298982c9b6260b. No GPU was used. Windows has not been tested on a Windows host.