Cache the ROCm and plain-clang Windows builds through sccache

.github/build.bat

@@ -20,9 +20,14 @@ REM sccache is on PATH AND the probe succeeds; otherwise the build proceeds
 REM uncached and green. The Visual Studio generator jobs do NOT set USE_CACHE, so
 REM this stays inert for them (and the VS generator ignores
 REM CMAKE_{C,CXX}_COMPILER_LAUNCHER anyway -- only Ninja/Makefiles honor it).
+REM SCCACHE_IDLE_TIMEOUT=0 (unless the job sets it): the server the probe starts lives
+REM until the job ends. With sccache's default of 600 s it stopped during the long
+REM stretches of a build and a new one counted only what came after, so the stats
+REM table below showed a fraction of the requests (the Windows ROCm job: 67 of ~160).
 REM ---------------------------------------------------------------------------
 set "LAUNCH="
 if /I "%USE_CACHE%"=="true" (
+    if not defined SCCACHE_IDLE_TIMEOUT set "SCCACHE_IDLE_TIMEOUT=0"
     where sccache >nul 2>&1
     if errorlevel 1 (
         echo build.bat: USE_CACHE=true but sccache not on PATH; building WITHOUT cache.

.github/build.sh

@@ -67,6 +67,12 @@ if [ "${USE_CACHE:-true}" = "true" ] && [ -n "${SCCACHE_WEBDAV_TOKEN:-}${SCCACHE
   fi
 fi
 
+# The server the probe below starts should live until the job ends. With sccache's default idle
+# timeout (600 s) it stopped during the long stretches of a build and a new one counted only what
+# came after, so the stats table showed a fraction of the requests (the Linux ROCm job: 2 of ~160).
+# A job (or a developer) that sets SCCACHE_IDLE_TIMEOUT keeps its own value.
+export SCCACHE_IDLE_TIMEOUT="${SCCACHE_IDLE_TIMEOUT:-0}"
+
 # Health-check before trusting sccache as the compiler launcher. Because sccache *is* the
 # launcher (cmake runs `sccache <compiler> ...` for every TU), a present-but-crashing sccache
 # fails every compile and reds the whole build — exactly the in-container panic that stalled
@@ -110,8 +116,12 @@ sccache_can_wrap_compiler() {
 # USE_CACHE is true AND sccache + a cache token are present AND the probe confirms sccache can
 # wrap the compiler — so it stays inert before the DEPOT_TOKEN secret is configured, on fork PRs
 # (secrets hidden), and when sccache would crash; all of those just compile normally. sccache is
-# content-addressed, so a cache hit is bit-identical to a fresh -O3 compile (release-safe), and
-# it degrades to direct compilation if the cache is unreachable.
+# content-addressed, so a cache hit is the object the same command line compiles (release-safe),
+# and it degrades to direct compilation if the cache is unreachable. "The same command line" is
+# the precise claim: for HIP TUs clang hashes the command line into the per-TU CUID symbol
+# (`__hip_cuid_<hash>`), and sccache runs the compiler with a slightly different one, so a HIP
+# object built through sccache differs from one built without it in that name -- code identical,
+# measured with a pinned -cuid: byte-identical. Between two sccache builds nothing differs.
 LAUNCH=""
 if [ "${USE_CACHE:-true}" = "true" ] && command -v sccache >/dev/null 2>&1 \
    && [ -n "${SCCACHE_WEBDAV_TOKEN:-}${SCCACHE_GHA_ENABLED:-}" ] \
@@ -141,6 +151,21 @@ if [ "${USE_CACHE:-true}" = "true" ] && command -v sccache >/dev/null 2>&1 \
         echo "build.sh: nvcc runs uncached (CUDA build detected, SCCACHE_WRAP_NVCC is not 'true'); sccache wraps the C/C++ TUs only"
       fi ;;
   esac
+  # ROCm builds: on Linux ggml compiles its ~150 HIP TUs as the CMake language HIP, which the
+  # C/CXX launchers do not reach -- the Linux ROCm job's stats table counted 2 requests (both
+  # C/C++) for a 2:50 h build of 27 GPU targets. sccache 0.18.0 caches clang's HIP compiles
+  # (measured in Docker with TheRock's clang 23, the job's 10.0.0 wheels and a WebDAV backend:
+  # 157/157 hits on the second build, 310 s -> 10 s for one target). One caveat, local only: with
+  # the *disk* backend sccache's preprocessor cache mode reads clang's -E output as text, and with
+  # --offload-compress that output is a compressed bundle -- some TUs then fail with "Failed to
+  # parse included file path" (mozilla/sccache#2788, unmerged). That mode exists only for the disk
+  # backend, so CI over Depot never takes that path; a local disk-cache build sets
+  # SCCACHE_DIRECT=false.
+  case " $* " in
+    *" -DGGML_HIP=ON "* | *" -DGGML_HIP=1 "* | *" -DGGML_HIP=on "*)
+      LAUNCH="$LAUNCH -DCMAKE_HIP_COMPILER_LAUNCHER=sccache"
+      echo "build.sh: sccache will also wrap the HIP compiler (HIP build detected)" ;;
+  esac
   echo "build.sh: sccache ON (endpoint=${SCCACHE_WEBDAV_ENDPOINT:-default}), building with -j${JOBS}"
 else
   echo "build.sh: sccache OFF, building with -j${JOBS}"

.github/workflows/publish.yml

@@ -1780,13 +1780,10 @@ jobs:
   #   * -DBUILD_TESTING: the option refuses it (jllama_test calls the CPU and RPC backends, which
   #     are modules here, not link-time libraries) -- test-cpp-windows-x86_64 runs the C++ suite in
   #     a static build with the same compiler, the split the Linux CPU jobs have.
-  #   * sccache: sccache 0.18.0 refuses any TU carrying an `-Xclang` argument, and CMake's
-  #     Platform/Windows-Clang.cmake puts one on every plain-clang TU for the runtime-library
-  #     selection (`-Xclang --dependent-lib=libcmt`; llama.cpp adds `-Xclang -fno-pch-timestamp`
-  #     on top for its own TUs) -- measured: every TU a miss, the build green and the cache inert
-  #     -- so the install would only cost time. The cl.exe TUs of the other Windows Ninja jobs
-  #     still cache; this one builds cold on every run, and so does the ROCm job (TheRock's clang
-  #     is a plain clang too; CLAUDE.md "sccache on every Windows Ninja job but one").
+  #   * sccache: like every Windows Ninja job. It went without from b11538 until the override
+  #     llama/cmake/msvc-runtime-gnu-clang.cmake: CMake's runtime-library flags for plain clang
+  #     carry `-Xclang -flto-visibility-public-std`, which sccache refuses, so every TU was a miss
+  #     (the ROCm job, TheRock's plain clang, the same; CLAUDE.md "sccache on every Windows Ninja job").
   # The MSVC job above keeps the Visual Studio generator as the test-only second toolchain; 32-bit
   # Windows is not built any more (upstream builds none either, and a 32-bit process addresses
   # ~2 GB, less than most models).
@@ -1796,6 +1793,11 @@ jobs:
     name: Build Windows 2025 x86_64 (modular, CPU variants, clang, default)
     needs: [startgate, build-webui]
     runs-on: windows-2025-vs2026
+    # sccache again since llama/cmake/msvc-runtime-gnu-clang.cmake: CMake no longer puts the
+    # -Xclang runtime flags sccache refuses on these plain-clang TUs (CLAUDE.md, "sccache on every
+    # Windows Ninja job").
+    env:
+      SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
       - name: Download shared WebUI assets
@@ -1810,6 +1812,10 @@ jobs:
       - name: Display CPU Info
         shell: bash
         run: bash .github/print-host-info.sh
+      - name: Install sccache (shared compiler cache)
+        if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != ''
+        continue-on-error: true
+        uses: ./.github/actions/install-sccache-windows
       # `clang --version` first: which LLVM the image put on PATH decides whether patches/0017's
       # compiler half is exercised as a warning (clang 20) or as the error it fixes (clang 22+).
       - name: Build libraries

CHANGELOG.md

@@ -696,6 +696,19 @@ the first two items break every existing consumer at upgrade.
   backend filter had changed loaded a second copy of the library, or extracted a second module set, and
   replaced the build info while the JVM kept using the first library. A further call now loads nothing
   and names a property changed since the load once on stderr; a failed attempt is retried.
+- **CI: both ROCm jobs and the Windows plain-clang CPU job compile through sccache again.** On Windows,
+  CMake's runtime-library flags for plain clang carry `-Xclang -flto-visibility-public-std`, which
+  sccache refuses, so every TU of `build-windows-x86_64` and of the ROCm module (TheRock's clang, ~150
+  HIP TUs for 23 GPU targets, the pipeline's critical path at 2:40-4:15 h) was a miss; a make-rules
+  override (`llama/cmake/msvc-runtime-gnu-clang.cmake`) selects the same runtime with the driver flag
+  `-fms-runtime-lib=static` instead. On Linux the HIP TUs are the CMake language HIP, which the C/C++
+  launchers never reached; `build.sh` adds `CMAKE_HIP_COMPILER_LAUNCHER`. Measured with one GPU target
+  before the change: 325 s -> 25 s (Windows) and 310 s -> 10 s (Linux) on a warm cache, objects
+  identical (the duplicate `/DEFAULTLIB:libcmt.lib` CMake wrote is gone; HIP objects built through
+  sccache differ from uncached ones in the per-TU CUID symbol name only). A cold run -- the first after
+  this, and a llama.cpp bump that touches the HIP headers -- takes as long as before. `build.sh` and
+  `build.bat` also keep the sccache server alive for the whole job, so the job summary's statistics
+  count every request.
 - **CI: the Linux CUDA build no longer routes nvcc through sccache by default** (`SCCACHE_WRAP_NVCC=true`
   in `build.sh` opts back in). With CUDA 13.4 and sccache 0.18.0 the nvcc launcher failed on the first `.cu`
   files in two confirmed runs (`Missing "cubin" file output`, `Compiler killed by signal 126`), and the