Fix ROCm CPU performance, update the ROCm toolchain, and bundle Windows ARM64 CUDA libraries

[oobabooga]

Summary

Three fixes to the prebuilt bundles, each measured or traced on real artifacts:

1. The ROCm bundles (Linux and Windows) ship a CPU backend built as generic scalar code, which roughly halves generation speed whenever MoE experts run on the CPU.
2. The ROCm download index we use stopped updating at 10.1.0a20260822. The current index needs two more runtime libraries, and without them every bundle fails to load while CI stays green.
3. The Windows ARM64 CUDA bundle contains no cuBLAS, so its CUDA backend cannot load.

The ROCm flags, ROCm index and libraries, and ARM64 toolkit follow upstream or lemonade-sdk; the dependency checks and shipping the ARM64 CUDA DLLs inside the bundle are new.

1. ROCm CPU backend

-DCMAKE_SYSTEM_NAME together with -DCMAKE_CROSSCOMPILING=ON leaves CMAKE_SYSTEM_PROCESSOR empty. ggml then reports GGML_SYSTEM_ARCH: UNKNOWN and builds ggml-cpu with -DGGML_CPU_GENERIC in every ROCm leg's nightly logs.

llama.cpp enables --fit by default, so this affects MoE models whose expert weights overflow VRAM onto the CPU.

Measured on a Ryzen AI MAX+ 395 (gfx1151, Linux). Source is upstream b11139, the ROCm toolchain is the same, and arms were run interleaved:

Qwen3 on gfx1151	Shipped	Fixed
30B-A3B generation, 24 layers' experts on CPU	28.4 t/s	50.8 t/s
30B-A3B generation, all experts on CPU	22.2 t/s	38.0 t/s
8B CPU-only generation	11.5 t/s	22.8 t/s
8B CPU-only prompt processing	14.6 t/s	230 t/s

With every layer on the GPU the two builds are identical (8B generation 42.4 vs 42.5 t/s). The published Windows gfx1151 zip shows the same loss on its CPU path (30B-A3B CPU-only generation 24.5 t/s vs 41.2 t/s for our windows-x64-cpu zip on the same machine).

The fix drops both flags and enables GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS, matching upstream's Linux ROCm release. Dropping the flags alone restores generation speed but imposes an AVX2 baseline, breaking pre-AVX2 CPUs. Runtime variant selection preserves compatibility and improves CPU prompt speed (153 t/s with AVX2 vs 230 t/s with zen4).

BACKEND_DL makes ggml-hip a loadable module alongside ggml-cpu-*. Studio's existing ROCm health-check globs match this layout.

2. ROCm toolchain refresh

rocm.nightlies.amd.com/tarball-multi-arch has not published anything after 10.1.0a20260822 for any of the 14 target and OS combinations. AMD now publishes to nightly.repo.amd.com/rocm/core/tarball (10.2.0a20260924 at the time of writing, same file naming). lemonade-sdk moved in lemonade-sdk/llamacpp-rocm#141.

In 10.2, libhipblaslt depends on a new libtensilelite-host, which on Linux in turn needs the unversioned llvm/lib/libomp.so. rocblas imports libhipblaslt, so the whole HIP backend depends on them. I built bundles with the copy list from this workflow and ran them in a clean ubuntu:24.04 container on gfx1151:

Bundle	Result
10.1.0a20260822, current copy list	runs
10.2.0a20260924, current copy list	exit 127, libtensilelite-host.so.1 not found
10.2, + libtensilelite-host	exit 127, libomp.so not found
10.2, + libtensilelite-host + libomp.so	runs

On Windows, the shipped gfx1151 zip with the 10.2 DLLs from the current list exits with 0xC0000135 (DLL not found) and runs once libtensilelite-host.dll is added. This matches lemonade-sdk/llamacpp-rocm#146.

Changes:

• Both jobs download from the new index.
• Linux copies libtensilelite-host.so* and llvm/lib/libomp.so; Windows copies libtensilelite-host.dll.
• The copy steps tolerate missing files, so new checks catch omitted dependencies:
  • Linux runs ldd with LD_LIBRARY_PATH unset and rejects missing dependencies or paths outside the bundle, except glibc, libstdc++ and libgcc_s. This prevents runner-installed libraries from hiding omissions.
  • Windows uses pefile to reject imports of DLLs present in TheRock's bin directory but absent from the bundle.

The refresh itself does not change speed: on gfx1151, 10.1.0a20260822, 10.2.0a20260924 and the stable 10.0.0 release are within 1.5% of each other on Qwen3-8B and Qwen3-30B-A3B.

3. Windows ARM64 CUDA runtime

app-...-windows-arm64-cuda13-portable.zip ships ggml-cuda.dll, which imports cublas64_13.dll, but the zip contains no CUDA runtime DLLs. package_bundle.py leaves the runtime out on purpose because the installer pairs x64 bundles with the user's PyTorch. download.pytorch.org has no win_arm64 wheels for cu128, cu129, cu130 or cu132, and Studio only pairs a runtime archive with upstream llama- assets. So on a Windows ARM64 machine without a CUDA toolkit, the CUDA backend cannot load and ggml runs on the CPU.

Changes:

• The leg replaces the 13.4.46 Developer Preview archives with 13.4.1 GA components from the redist CDN, matching ggml-org/llama.cpp#28687.
• cudart64_13.dll, cublas64_13.dll and cublasLt64_13.dll from bin\arm64 are copied into the bundle input.
• package_bundle.py keeps them when BUNDLE_CUDA_RUNTIME=1. Only this leg sets it, so every other bundle is unchanged.
• The DLLs carry NVIDIA's Authenticode signature; the final bundle signature check remains in place. Studio installs them and checks for all three names.

Not changed

• The CUDA 13 toolkit stays on 13.3 for x64 and Linux. 13.4.1 measured within noise on RTX 3090 and RTX 6000 Ada.
• Weekly pinning of ROCm nightlies stays. It works unchanged against the new index.

Verification

• A/B builds and llama-bench on a gfx1151 Linux runner for the CPU backend flags and for the three toolchains (tables above).
• Clean-container load tests of bundles assembled with the old and new copy lists (Linux), and the DLL swap load test on the Windows gfx1151 runner.
• Local dry run of the edited Linux job, executing its run: blocks from this branch in ubuntu:22.04:
  • It picked 10.2.0a20260924 from the new index, configured with the full CPU variant set, and the dependency check passed.
  • The check rejects missing libtensilelite-host, and rejects either it or libomp.so resolving from a system library directory. A plain not found check misses the latter case.
  • The packaged bundle loaded libggml-hip.so and the alderlake CPU variant on an Intel host.
• package_bundle.py run locally on a Windows-layout input with and without BUNDLE_CUDA_RUNTIME=1: the zip gains exactly the three runtime DLLs, and is unchanged without the flag.
• actionlint, check_workflow_scalars.py, check_workflow_triggers.py and check_workflow_outputs.py pass.

Not verified:

• The Windows ROCm build with the new CMake flags has not been compiled.
• The Windows ARM64 CUDA leg has not run, and the bundle has not been tested on Windows ARM64 NVIDIA hardware.
• None of the edited workflows have run in GitHub Actions yet.

[oobabooga]

Reached Codex review convergence at oobabooga#6.

[danielhanchen]

@codex review

[danielhanchen]

@codex security review

[danielhanchen]

@codex review

[LeoBorcherding]

tested the windows rocm leg locally (the part listed as not verified) on windows 11, ryzen 5 2600, rx 6500 xt (gfx1034), rocm 10.2.0a20260924 from the new index, msvc 14.44, and the pr's exact cmake line.

the diagnosis is right, but the windows build fails

configure now reports GGML_SYSTEM_ARCH: x86 and adds all 14 cpu variants. but amd clang 24 (rocm 10.2) stops on the first one:

quants.c:782:22: error: incompatible pointer types passing 'const block_q4_0 *'
  to parameter of type 'const char *' [-Wincompatible-pointer-types]
  782 |         _mm_prefetch(&x[ib] + sizeof(block_q4_0), _MM_HINT_T0);

(same error at 783, 800 and 801)

rocm's clang has never compiled the x86 simd files before, because the cpu backend used to be generic. on windows, clang takes msvc's _mm_prefetch(char const*, int) declaration, while clang's own casting macro only exists under #ifndef _MSC_VER. that's why linux passes. upstream avoids this by building only --target ggml-hip with rocm clang on windows. since this pr moves to 10.2, the first windows rocm run in actions will fail.

fix (tested): add a cast on the 4 lines. with the pr's flags unchanged, all 14 variants then build clean:

_mm_prefetch((const char *)(&x[ib] + sizeof(block_q4_0)), _MM_HINT_T0);

(-Wno-error=incompatible-pointer-types works too: the full build passes and these 4 lines are the only hits.)

with the fix, everything else checks out

• the pefile check passes, and fails correctly if libtensilelite-host.dll is removed
• it loads ggml-hip + ggml-cpu-haswell and sees gfx1034
• heads up: with GGML_BACKEND_DL, a missing rocm dll now means a silent cpu fallback with exit 0. before, it crashed with 0xC0000135, since the shipped ggml.dll imports ggml-hip.dll directly. so the new ci checks are now the only guard. a ci load test that requires a ROCm0 device could help

speed, shipped zip vs pr (t/s)

case	shipped	pr
qwen3-1.7b, -ngl 0, tg128	5.5-6.9	11.7-13.1 (~2x)
gemma-4-26b-a4b, all experts on cpu, tg64 (warm, 3 rounds, <1% spread)	3.24	10.27 (3.2x)
qwen3-1.7b, full gpu	85.5	86.1 (same)

greedy output is coherent on both builds. i didn't test the arm64 cuda leg (no hardware).

[oobabooga]

Thanks @LeoBorcherding, reproduced with the workflow's Windows steps on a Ryzen AI MAX+ 395 (gfx1151) with ROCm 10.2.0a20260924. The sse42 variant fails on the same four _mm_prefetch errors in arch/x86/quants.c.

Fix: 62c131b adds -DCMAKE_C_FLAGS="-Wno-error=incompatible-pointer-types" to the Windows ROCm configure. A flag rather than the cast, because these builds compile the upstream source tarball and a source change would have to go through the pin set. With the flag, the full build passes and those four lines are the only warnings of that kind. The DLL check passes, and the build loads ggml-hip with ggml-cpu-zen4.

Same machine, shipped zip vs this branch:

case	shipped	PR
Qwen3-8B CPU-only, pp512	14.7 t/s	239 t/s
Qwen3-8B CPU-only, tg128	11.4 t/s	20.7 t/s
Qwen3-30B-A3B, all experts on CPU, tg128	17.9 t/s	27.5 t/s
Qwen3-8B full GPU, tg128	35.3 t/s	39.5 t/s (within run-to-run noise)

With libtensilelite-host.dll removed, llama-bench --list-devices exits 0 and lists no devices. The pefile check fails the build in that case (libhipblaslt.dll -> libtensilelite-host.dll). GitHub-hosted Windows runners have no AMD GPU, so a CI load test requiring ROCm0 is not possible and the static check is the guard.

Upstream builds only ggml-hip with ROCm's clang on Windows and takes the CPU backend and tools from its CPU build, so it does not hit this.