From 8a6d8215058b5e45d2c5e72888c71c7a1828f4c4 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Sun, 25 Oct 2026 23:36:59 +0100 Subject: [PATCH 01/13] Move main to 6.1.0-SNAPSHOT versions:set over the six reactor poms (it also moved project.build.outputTimestamp), the standalone agent pom by hand, the README's snapshot snippet, the agent's CLAUDE.md and the ReflowProbe command line. The release snippets stay at 6.0.0. TODO: the bump item is done, the upstream submissions remain. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AJGh28Gu8pLWYbTCqG1LZf --- README.md | 2 +- TODO.md | 5 ++--- llama-atmosphere-agent/CLAUDE.md | 2 +- llama-atmosphere-agent/pom.xml | 2 +- .../net/ladenthin/llama/atmosphere/probe/ReflowProbe.java | 2 +- llama-ffm/pom.xml | 2 +- llama-kotlin/pom.xml | 2 +- llama-langchain4j/pom.xml | 2 +- llama-platform/pom.xml | 2 +- llama/pom.xml | 4 ++-- pom.xml | 2 +- 11 files changed, 13 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 1ddab0e1f..2d21bd234 100644 --- a/README.md +++ b/README.md @@ -188,7 +188,7 @@ To use the latest snapshot, add the repository and dependency to your `pom.xml`: net.ladenthin llama-platform - 6.0.0-SNAPSHOT + 6.1.0-SNAPSHOT pom ``` diff --git a/TODO.md b/TODO.md index c7cf12c15..0bbfe1be1 100644 --- a/TODO.md +++ b/TODO.md @@ -63,9 +63,8 @@ and footer, `SECURITY.md` on `6.x`) is the pull request `release/6.0.0`. contribution by its author; KoboldCpp, where it came from, is AGPL-3.0); the third-party logos the WebUI embeds (trademarks, no license text). The WebUI package list is an upper bound (imports + lockfile); an exact list from the bundle would need a plugin in upstream's build. -- [ ] After publication: bump `main` to the next `-SNAPSHOT` in all seven poms and the README's - snapshot line (it stays `6.0.0-SNAPSHOT` in the release commit), and carry out the upstream - submissions queued below. +- [ ] After publication: carry out the upstream submissions queued below (`main` is at + `6.1.0-SNAPSHOT` since the post-release bump). - [ ] After publication: srcmorph (#238) builds against `llama:6.0.0` from Central; its Build, plugin IT, code style, CodeQL and SonarQube jobs are red until then, and a local srcmorph build stops at its license collector against a llama snapshot older than the natives license diff --git a/llama-atmosphere-agent/CLAUDE.md b/llama-atmosphere-agent/CLAUDE.md index 5b54f2604..a1714ab74 100644 --- a/llama-atmosphere-agent/CLAUDE.md +++ b/llama-atmosphere-agent/CLAUDE.md @@ -10,7 +10,7 @@ that pairs [Atmosphere](https://github.com/Atmosphere/atmosphere)'s built-in Ope agent runtime with this project's `OpenAiCompatServer`. Like `android-llmservice/` it is a **standalone Maven project, NOT a reactor module** — it is an application, it needs Java 21 (Atmosphere's floor) while the core stays Java 8, and it is built without a parent so the folder can be -copied out and run on its own. **Its version is the core's** (`6.0.0-SNAPSHOT` on `main`), and +copied out and run on its own. **Its version is the core's** (`6.1.0-SNAPSHOT` on `main`), and `check-natives.py` fails when the two differ: `versions:set` does not reach this pom, so a release bump that forgot it would otherwise publish an agent naming the wrong core. `llama.version` defaults to `${project.version}`; CI still passes `-Dllama.version=` (the same value). diff --git a/llama-atmosphere-agent/pom.xml b/llama-atmosphere-agent/pom.xml index 8497e7f0b..bb6bff807 100644 --- a/llama-atmosphere-agent/pom.xml +++ b/llama-atmosphere-agent/pom.xml @@ -23,7 +23,7 @@ SPDX-License-Identifier: MIT --> net.ladenthin llama-atmosphere-agent - 6.0.0 + 6.1.0-SNAPSHOT jar ${project.groupId}:${project.artifactId} diff --git a/llama-atmosphere-agent/src/test/java/net/ladenthin/llama/atmosphere/probe/ReflowProbe.java b/llama-atmosphere-agent/src/test/java/net/ladenthin/llama/atmosphere/probe/ReflowProbe.java index 991c6b614..ad01a57dc 100644 --- a/llama-atmosphere-agent/src/test/java/net/ladenthin/llama/atmosphere/probe/ReflowProbe.java +++ b/llama-atmosphere-agent/src/test/java/net/ladenthin/llama/atmosphere/probe/ReflowProbe.java @@ -36,7 +36,7 @@ * terminal library the agent uses: * * 
- * mvn -q test-compile exec:java -Dexec.classpathScope=test  *     -Dexec.mainClass=net.ladenthin.llama.atmosphere.probe.ReflowProbe  *     "-Dllama.version=6.0.0-SNAPSHOT" "-Djline.version=4.4.6-atmosphere"
+ * mvn -q test-compile exec:java -Dexec.classpathScope=test  *     -Dexec.mainClass=net.ladenthin.llama.atmosphere.probe.ReflowProbe  *     "-Dllama.version=6.1.0-SNAPSHOT" "-Djline.version=4.4.6-atmosphere"
  *
* *

What its answers were used for, and the numbers they produced, is in diff --git a/llama-ffm/pom.xml b/llama-ffm/pom.xml index 205613999..8c886dbc4 100644 --- a/llama-ffm/pom.xml +++ b/llama-ffm/pom.xml @@ -12,7 +12,7 @@ SPDX-License-Identifier: MIT net.ladenthin llama-parent - 6.0.0 + 6.1.0-SNAPSHOT ../pom.xml diff --git a/llama-kotlin/pom.xml b/llama-kotlin/pom.xml index 2e650ae91..6b402b2af 100644 --- a/llama-kotlin/pom.xml +++ b/llama-kotlin/pom.xml @@ -12,7 +12,7 @@ SPDX-License-Identifier: MIT net.ladenthin llama-parent - 6.0.0 + 6.1.0-SNAPSHOT ../pom.xml diff --git a/llama-langchain4j/pom.xml b/llama-langchain4j/pom.xml index 84aaa5f75..70134a4cb 100644 --- a/llama-langchain4j/pom.xml +++ b/llama-langchain4j/pom.xml @@ -12,7 +12,7 @@ SPDX-License-Identifier: MIT net.ladenthin llama-parent - 6.0.0 + 6.1.0-SNAPSHOT ../pom.xml diff --git a/llama-platform/pom.xml b/llama-platform/pom.xml index 7776ef357..2c28ab817 100644 --- a/llama-platform/pom.xml +++ b/llama-platform/pom.xml @@ -12,7 +12,7 @@ SPDX-License-Identifier: MIT net.ladenthin llama-parent - 6.0.0 + 6.1.0-SNAPSHOT ../pom.xml diff --git a/llama/pom.xml b/llama/pom.xml index 4e6eda65c..63ae102d3 100644 --- a/llama/pom.xml +++ b/llama/pom.xml @@ -12,7 +12,7 @@ SPDX-License-Identifier: MIT net.ladenthin llama-parent - 6.0.0 + 6.1.0-SNAPSHOT ../pom.xml @@ -103,7 +103,7 @@ SPDX-License-Identifier: MIT 3.10.4 2.102.0 UTF-8 - 2026-10-25T21:44:25Z + 2026-10-25T22:38:31Z net.ladenthin llama-parent - 6.0.0 + 6.1.0-SNAPSHOT pom ${project.groupId}:${project.artifactId} From 9898cad065b217d70aaa215c8d33a8f463876255 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 00:23:37 +0100 Subject: [PATCH 02/13] s390x: build against the manylinux_2_28 sysroot, load every Linux CPU jar on RHEL 8 in CI The 6.0.0 cpu-linux-s390x jar was cross-compiled with the ubuntu runner's GCC 13 against its glibc 2.39 sysroot and referenced GLIBC_2.38, GLIBCXX_3.4.32 and CXXABI_1.3.13: it failed to load on RHEL 8/9 and Ubuntu 22.04 while every job was green. build-linux-s390x now cross-compiles with clang-18 against the pypa manylinux_2_28_s390x image unpacked as a sysroot (.github/manylinux/manylinux_2_28-s390x-sysroot, pinned by digest, nothing of it executed) through .github/manylinux/s390x-toolchain.cmake: --gcc-install-dir takes gcc-toolset-14's crt files, libgcc and libstdc++ (GCC 8's libstdc++.so.6 plus libstdc++_nonshared.a) as a native toolset build would, binutils-s390x-linux-gnu links, -march=z13 keeps the former arch11 baseline. The emulator passes the sysroot as QEMU_LD_PREFIX (an environment variable, because googletest's death tests re-exec through binfmt_misc without qemu's -L), and the sysroot path is cached so a reconfigure finds it. Runner pinned to ubuntu-24.04. Measured locally (b11887): GLIBC_2.27, GLIBCXX_3.4.25, CXXABI_1.3.11, new dependencies libpthread.so.0 and libdl.so.2; 694/694 C++ tests under qemu against the sysroot and natively on a LinuxONE z16 (cfarm191); the jar loads on RHEL 8.10 s390x, the 6.0.0 one fails there with GLIBC_2.38 not found. Checks, so it cannot regress unseen, for every architecture: - verify-native-deps.py holds Linux/s390x/cpu to GLIBC_2.28 and every floor directory to GLIBCXX_3.4.25 / CXXABI_1.3.11 (LIBSTDCXX_CEILING, met by x86-64, aarch64 and CUDA today); elf_info reports the three families and skips GLIBC_PRIVATE / CXXABI_TM_1 instead of crashing. The s390x job runs it on its own tree too. - smoke-glibc-floor: package uploads the floor set (check-natives.py floor-set, derived from GLIBC_CEILING) as llama-glibc-floor, and one matrix row per jar loads it with NativeLoadSmoke in the UBI 8 OpenJDK 21 runtime image (RHEL 8.10, glibc 2.28) on its own architecture, s390x under qemu (smoke-glibc-floor.sh). check-natives.py holds the rows to the set; both publish gates wait for the job. Locally green on all three and red on the 6.0.0 s390x jar. - Unit tests for the libstdc++ versions, the s390x and libstdc++ ceilings, floor_set and check_floor (103 tests, green on Linux). README row back to glibc 2.28, CHANGELOG [Unreleased], CLAUDE.md "Linux s390x" and the manylinux and check tables, docs/local-test-plan.md, TODO: what is done and what only the first CI run can show. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AJGh28Gu8pLWYbTCqG1LZf --- .github/buildcheck/nativedeps.py | 74 +++++++++---- .github/buildcheck/natives.py | 52 ++++++++- .github/buildcheck/tests/test_nativedeps.py | 38 ++++++- .github/buildcheck/tests/test_natives.py | 37 +++++++ .github/check-natives.py | 5 + .../manylinux/manylinux_2_28-s390x-sysroot | 72 +++++++++++++ .github/manylinux/s390x-toolchain.cmake | 74 +++++++++++++ .github/package-smoke-sets.sh | 21 +++- .github/smoke-glibc-floor.sh | 86 +++++++++++++++ .github/workflows/publish.yml | 100 +++++++++++++++--- CHANGELOG.md | 20 ++++ CLAUDE.md | 51 +++++++-- README.md | 2 +- TODO.md | 95 ++++++++--------- docs/local-test-plan.md | 1 + 15 files changed, 625 insertions(+), 103 deletions(-) create mode 100755 .github/manylinux/manylinux_2_28-s390x-sysroot create mode 100644 .github/manylinux/s390x-toolchain.cmake create mode 100755 .github/smoke-glibc-floor.sh diff --git a/.github/buildcheck/nativedeps.py b/.github/buildcheck/nativedeps.py index c72b689d8..90075d1ed 100755 --- a/.github/buildcheck/nativedeps.py +++ b/.github/buildcheck/nativedeps.py @@ -32,8 +32,9 @@ or is a module (whose libggml-base lies next to it only after extraction), must carry the run path `$ORIGIN` and nothing else (a build-tree path would point at the CI runner) -- except on Android, where a library carries no run path at all and only a build-tree path fails (see violations()) -- -and the directories in GLIBC_CEILING, built in a manylinux image, may not reference a glibc symbol -version above the floor they promise. Android libraries must also have every LOAD segment 16 KB +and the directories in GLIBC_CEILING, built in a manylinux image or against its sysroot, may not +reference a glibc symbol version above the floor they promise, nor a libstdc++ one (GLIBCXX_*, +CXXABI_*) above LIBSTDCXX_CEILING. Android libraries must also have every LOAD segment 16 KB aligned. That every listed build arrived is merge-native-artifacts.sh's check. @@ -41,6 +42,7 @@ """ import os +import re import struct import sys @@ -81,7 +83,9 @@ def ucrt_forwarder(name): # What each CPU library needed when this check was introduced (5.1.0 plus the RPC backend, # which adds nothing: its sockets are libc/libSystem/WS2_32, all already present). The manylinux_2_28 -# builds (glibc 2.28, before the libpthread/libdl/librt merge of 2.34) name those three separately. +# builds (glibc 2.28, before the libpthread/libdl/librt merge of 2.34) name those three separately; +# s390x is one of them since its build moved to the manylinux_2_28 sysroot (2026-10-25), and its +# single static library needs libpthread and libdl, not librt (measured). # libgomp.so.1 is deliberately absent from the three Linux cpu entries: llama/CMakeLists.txt forces # GGML_OPENMP=OFF there so the jar loads in slim and distroless images, which do not ship it, and # this list is what keeps it that way -- a build that regains the dependency fails `package` instead @@ -92,7 +96,8 @@ def ucrt_forwarder(name): "libm.so.6", "libgcc_s.so.1", "libc.so.6", "ld-linux-x86-64.so.2"}, "Linux/aarch64/cpu": {"libdl.so.2", "libpthread.so.0", "librt.so.1", "libstdc++.so.6", "libm.so.6", "libgcc_s.so.1", "libc.so.6", "ld-linux-aarch64.so.1"}, - "Linux/s390x/cpu": {"libstdc++.so.6", "libm.so.6", "libgcc_s.so.1", "libc.so.6", "ld64.so.1"}, + "Linux/s390x/cpu": {"libdl.so.2", "libpthread.so.0", "libstdc++.so.6", "libm.so.6", "libgcc_s.so.1", + "libc.so.6", "ld64.so.1"}, "Linux-Android/aarch64/cpu": {"liblog.so", "libm.so", "libdl.so", "libc.so", "libandroid.so"}, "Linux-Android/x86_64/cpu": {"liblog.so", "libm.so", "libdl.so", "libc.so", "libandroid.so"}, # Exactly WINDOWS_OS, plus any api-ms-win-crt-* forwarder through ucrt_forwarder() -- see @@ -122,12 +127,27 @@ def ucrt_forwarder(name): # The glibc floor a directory promises (README, "Runtime requirement"): the highest GLIBC_x.y symbol # version any of its libraries references may not exceed it. These are the builds made in a # manylinux_2_28 image (publish.yml: crosscompile-linux-x86_64, crosscompile-linux-aarch64, -# crosscompile-linux-x86_64-cuda); the other Linux builds inherit the floor of the ubuntu runner. +# crosscompile-linux-x86_64-cuda) or cross-compiled against one unpacked as the sysroot +# (build-linux-s390x, .github/manylinux/s390x-toolchain.cmake); the other Linux builds inherit the +# floor of the ubuntu runner. s390x joined on 2026-10-25: its cross build against the ubuntu +# runner's own sysroot had shipped GLIBC_2.38 without any check noticing (CLAUDE.md, "Linux s390x"). GLIBC_CEILING = { "Linux/x86_64/cpu": (2, 28), "Linux/aarch64/cpu": (2, 28), "Linux/x86_64/cuda13": (2, 28), + "Linux/s390x/cpu": (2, 28), } +# The libstdc++ half of the same floor, for the same directories: the system libstdc++.so.6 of +# manylinux_2_28 is GCC 8's (GLIBCXX_3.4.25, CXXABI_1.3.11), and gcc-toolset-14 links whatever its +# GCC 14 headers need beyond that from libstdc++_nonshared.a. A version above it means the library +# was linked against a newer libstdc++ and fails to load on RHEL 8 even when its glibc symbols fit +# (the s390x jar of 6.0.0 needed GLIBCXX_3.4.32 and CXXABI_1.3.13). Measured on the CI artifacts of +# 2026-10-25: x86-64 and aarch64 reference 3.4.25 / 1.3.11 exactly, the CUDA module 3.4.21 / 1.3.9. +LIBSTDCXX_CEILING = {"GLIBCXX": (3, 4, 25), "CXXABI": (1, 3, 11)} +# The symbol versions elf_info() reports, by family. libstdc++ on s390x (and ppc64) also defines +# GLIBCXX_LDBL_x / CXXABI_LDBL_x for its second long double ABI; they count as their family. +# Anything else (GLIBC_PRIVATE, CXXABI_TM_1, ...) carries no floor and is skipped. +SYMBOL_VERSION = re.compile(r"(GLIBC|GLIBCXX|CXXABI)_(?:LDBL_)?(\d+(?:\.\d+)*)") # Google Play's 16 KB page-size requirement (Android 15+ targets): every LOAD segment of an # Android library must be aligned to a multiple of it. CMake pins -Wl,-z,max-page-size=16384. @@ -172,10 +192,12 @@ def _cstring(data, start): def elf_info(data): - """DT_NEEDED, the run path (DT_RUNPATH, else DT_RPATH, else None) and the highest GLIBC_x.y - symbol version the library references (a tuple, or None when it references none).""" + """DT_NEEDED, the run path (DT_RUNPATH, else DT_RPATH, else None) and the highest GLIBC_x.y, + GLIBCXX_x.y.z and CXXABI_x.y.z symbol version the library references (`glibc`, `glibcxx`, + `cxxabi`: a tuple each, or None when it references none of that family).""" is64, end, sections = _elf_sections(data) - needed, runpath, rpath, glibc = [], None, None, [] + needed, runpath, rpath = [], None, None + versions = {"GLIBC": [], "GLIBCXX": [], "CXXABI": []} for sh_type, sh_offset, sh_size, sh_link, sh_info in sections: strtab = sections[sh_link][1] if sh_link < len(sections) else 0 if sh_type == SHT_DYNAMIC: @@ -197,15 +219,16 @@ def elf_info(data): aux = off + vn_aux for _ in range(vn_cnt): _, _, _, vna_name, vna_next = struct.unpack_from(end + "IHHII", data, aux) - name = _cstring(data, strtab + vna_name) - if name.startswith("GLIBC_"): - glibc.append(tuple(int(p) for p in name[len("GLIBC_"):].split("."))) + match = SYMBOL_VERSION.fullmatch(_cstring(data, strtab + vna_name)) + if match: + versions[match.group(1)].append(tuple(int(p) for p in match.group(2).split("."))) aux += vna_next if vn_next == 0: break off += vn_next + top = {family: max(found) if found else None for family, found in versions.items()} return {"needed": needed, "runpath": runpath if runpath is not None else rpath, - "glibc": max(glibc) if glibc else None} + "glibc": top["GLIBC"], "glibcxx": top["GLIBCXX"], "cxxabi": top["CXXABI"]} def elf_needed(data): @@ -482,15 +505,16 @@ def denied(deps): return [d for d in deps if any(bad in d.lower() for bad in DENIED)] -def glibc_name(version): - return "GLIBC_" + ".".join(str(p) for p in version) +def glibc_name(version, family="GLIBC"): + return family + "_" + ".".join(str(p) for p in version) -def violations(rel, deps, alignments=(), siblings=(), runpath=None, glibc=None): +def violations(rel, deps, alignments=(), siblings=(), runpath=None, glibc=None, glibcxx=None, cxxabi=None): """The violations of one library, `rel` being its path below the natives root (...////); `alignments` are its LOAD segment alignments (checked for Android libraries), `siblings` the names of the other files in its directory, `runpath` - its ELF run path and `glibc` the highest glibc symbol version it references.""" + its ELF run path, `glibc` the highest glibc symbol version it references and `glibcxx` / + `cxxabi` the highest libstdc++ ones.""" failures = [f"{rel} needs {d}, which no consumer can be expected to have" for d in denied(deps)] parts = rel.split("/") key = "/".join(parts[-4:-1]) if len(parts) >= 4 else "" @@ -553,7 +577,14 @@ def violations(rel, deps, alignments=(), siblings=(), runpath=None, glibc=None): ceiling = GLIBC_CEILING.get(key) if ceiling and glibc and glibc > ceiling: failures.append(f"{rel} references {glibc_name(glibc)}, above the {glibc_name(ceiling)} floor its " - f"directory promises -- was it built in the manylinux image?") + f"directory promises -- was it built in (or against) the manylinux image?") + if ceiling: + for family, found in (("GLIBCXX", glibcxx), ("CXXABI", cxxabi)): + top = LIBSTDCXX_CEILING[family] + if found and found > top: + failures.append(f"{rel} references {glibc_name(found, family)}, above the {glibc_name(top, family)} " + f"of manylinux_2_28's libstdc++ -- was it linked against gcc-toolset's libstdc++ " + f"(the system libstdc++.so.6 plus libstdc++_nonshared.a)?") allowed = ALLOWED.get(key) if allowed is None: if key.rsplit("/", 1)[-1] not in CPU_BACKENDS: @@ -579,23 +610,26 @@ def main(argv): checked += 1 rel = os.path.relpath(path, root).replace(os.sep, "/") siblings = {n for n in os.listdir(os.path.dirname(path)) if n != os.path.basename(path)} - alignments, runpath, glibc, note = (), None, None, "" + alignments, runpath, glibc, glibcxx, cxxabi, note = (), None, None, None, None, "" try: if path.endswith(".so"): with open(path, "rb") as f: data = f.read() info = elf_info(data) deps, runpath, glibc = info["needed"], info["runpath"], info["glibc"] + glibcxx, cxxabi = info["glibcxx"], info["cxxabi"] if "/Linux-Android/" in "/" + rel: alignments = elf_load_alignments(data) - note = (f" [runpath={runpath}]" if runpath else "") + (f" [{glibc_name(glibc)}]" if glibc else "") + note = (f" [runpath={runpath}]" if runpath else "") + "".join( + f" [{glibc_name(v, family)}]" for family, v in (("GLIBC", glibc), ("GLIBCXX", glibcxx), + ("CXXABI", cxxabi)) if v) else: deps = dependencies(path) except (ValueError, struct.error) as e: failures.append(f"{rel}: not a readable native library ({e})") continue print(f"{rel}: {' '.join(deps)}{note}") - failures += violations(rel, deps, alignments, siblings, runpath, glibc) + failures += violations(rel, deps, alignments, siblings, runpath, glibc, glibcxx, cxxabi) if os.path.basename(path) in JLLAMA_LIBRARIES: try: names = exports(path) diff --git a/.github/buildcheck/natives.py b/.github/buildcheck/natives.py index 5e563ecac..6e37134a4 100644 --- a/.github/buildcheck/natives.py +++ b/.github/buildcheck/natives.py @@ -43,7 +43,8 @@ KINDS = ("library", "module") # Platforms that get no smoke set: `java -cp` does not apply on Android (the AAR and the emulator -# job do), and no GitHub runner exists for s390x (its C++ suite runs under qemu instead). +# job do), and no GitHub runner exists for s390x (its C++ suite runs under qemu instead, and +# smoke-glibc-floor loads its jar under qemu on RHEL 8 -- a load, not the model-backed server smoke). SMOKE_EXCLUDED_OSES = ("Linux-Android",) # The job that re-uploads the Windows natives artifacts Authenticode-signed (CLAUDE.md, "Code signing"). SIGN_JOB = "sign-windows-natives" @@ -134,6 +135,54 @@ def smoke_set(natives, smoke_target): return [r for r in natives if target(r) == smoke_target] +# The glibc-floor smoke (publish.yml smoke-glibc-floor): every library jar whose directory promises a +# glibc floor (nativedeps.GLIBC_CEILING) is loaded on the oldest distribution of that floor, RHEL 8 +# (glibc 2.28 and GCC 8's libstdc++.so.6), on its own architecture -- s390x under qemu. verify-native- +# deps.py reads the symbol versions; only a load on the old runtime proves the jar starts there, and +# none ran before the s390x jar of 6.0.0 shipped needing glibc 2.38 (CLAUDE.md, "Linux s390x"). `package` +# uploads one set (the classes jar, its dependencies and these jars) as FLOOR_ARTIFACT, and the job has +# one matrix row per floor target, which check_floor holds to the list. +FLOOR_ARTIFACT = "llama-glibc-floor" +FLOOR_JOB = "smoke-glibc-floor" + + +def floor_set(natives): + """The library rows whose directory promises a glibc floor (nativedeps.GLIBC_CEILING), in list order. + Module rows are left out: a GPU module needs its vendor runtime, which an RHEL 8 image does not have.""" + return [r for r in libraries(natives) if r["directory"] in nativedeps.GLIBC_CEILING] + + +def floor_targets(natives): + """The - of every floor_set row, sorted: one smoke-glibc-floor matrix row each.""" + return sorted(target(r) for r in floor_set(natives)) + + +def check_floor(natives, jobs): + """`package` uploads the floor set as FLOOR_ARTIFACT, and FLOOR_JOB downloads it and launches one + matrix row per floor target: a library jar that promises a glibc floor does not ship unloaded on it.""" + targets = floor_targets(natives) + if not targets: + return [] + failures = [] + package = jobs.get("package") + if package is not None and FLOOR_ARTIFACT not in package.uploads(): + failures.append(f"publish.yml: package does not upload {FLOOR_ARTIFACT}, the set {FLOOR_JOB} loads " + f"(package-smoke-sets.sh writes it)") + job = jobs.get(FLOOR_JOB) + if job is None: + return failures + [f"publish.yml has no `{FLOOR_JOB}` job, which loads {', '.join(targets)} on RHEL 8"] + downloads = any(any("actions/download-artifact@" in line for line in step) + and any(line.strip() == f"name: {FLOOR_ARTIFACT}" for line in step) for step in job.steps()) + if not downloads: + failures.append(f"publish.yml {FLOOR_JOB} does not download {FLOOR_ARTIFACT}") + launched = {m.group(1) for m in map(MATRIX_ROW.match, job.lines) if m} + failures += [f"publish.yml {FLOOR_JOB}: no matrix row for {t}, whose jar promises the glibc floor of " + f"nativedeps.GLIBC_CEILING" for t in sorted(set(targets) - launched)] + failures += [f"publish.yml {FLOOR_JOB}: the matrix row {t} is no library jar with a glibc floor " + f"(nativedeps.GLIBC_CEILING)" for t in sorted(launched - set(targets))] + return failures + + def compare(what, expected, actual): return ([f"{what}: missing {name}" for name in sorted(set(expected) - set(actual))] + [f"{what}: {name} is not in .github/natives.csv" for name in sorted(set(actual) - set(expected))]) @@ -514,6 +563,7 @@ def check(root): + check_pom(natives, read(root, "llama/pom.xml")) + check_platform(natives, read(root, "llama-platform/pom.xml")) + check_workflow(natives, workflow.parse(read(root, ".github/workflows/publish.yml"))) + + check_floor(natives, workflow.parse(read(root, ".github/workflows/publish.yml"))) + check_loader(natives, read(root, "llama/src/main/java/net/ladenthin/llama/loader/LlamaLoader.java")) + check_cmake(natives, read(root, "llama/CMakeLists.txt")) + check_dependency_allowlist(natives, nativedeps.ALLOWED) diff --git a/.github/buildcheck/tests/test_nativedeps.py b/.github/buildcheck/tests/test_nativedeps.py index cdaa412a8..05991af3d 100644 --- a/.github/buildcheck/tests/test_nativedeps.py +++ b/.github/buildcheck/tests/test_nativedeps.py @@ -155,7 +155,16 @@ def test_elf_run_path_and_glibc_versions(self): self.assertEqual((plain["runpath"], plain["glibc"]), (None, None)) info = nativedeps.elf_info(elf64(["libggml.so", "libc.so.6"], runpath="$ORIGIN", glibc=["GLIBC_2.28", "GLIBC_2.2.5", "GLIBC_2.17"])) - self.assertEqual(info, {"needed": ["libggml.so", "libc.so.6"], "runpath": "$ORIGIN", "glibc": (2, 28)}) + self.assertEqual(info, {"needed": ["libggml.so", "libc.so.6"], "runpath": "$ORIGIN", "glibc": (2, 28), + "glibcxx": None, "cxxabi": None}) + + def test_elf_libstdcxx_versions(self): + # the highest of each family; s390x's long double twins count as their family, and the + # versions without a number carry no floor (a crash on int("PRIVATE") before) + info = nativedeps.elf_info(elf64(["libstdc++.so.6", "libc.so.6"], glibc=[ + "GLIBC_2.17", "GLIBC_PRIVATE", "GLIBCXX_3.4.9", "GLIBCXX_3.4.21", "GLIBCXX_LDBL_3.4.25", + "CXXABI_1.3.9", "CXXABI_TM_1", "CXXABI_LDBL_1.3"])) + self.assertEqual((info["glibc"], info["glibcxx"], info["cxxabi"]), ((2, 17), (3, 4, 25), (1, 3, 9))) def test_elf_load_alignments(self): self.assertEqual(nativedeps.elf_load_alignments(elf64_segments([16384, 65536])), [16384, 65536]) @@ -329,7 +338,32 @@ def test_the_manylinux_directories_keep_their_glibc_floor(self): self.assertIn("GLIBC_2.28", failures[0]) # a build on the ubuntu runner promises no floor self.assertEqual(nativedeps.violations("x/Linux/x86_64/vulkan/libggml-vulkan.so", - ["libggml-base.so", "libc.so.6"], runpath="$ORIGIN", glibc=(2, 39)), []) + ["libggml-base.so", "libc.so.6"], runpath="$ORIGIN", glibc=(2, 39), + glibcxx=(3, 4, 33), cxxabi=(1, 3, 15)), []) + + def test_s390x_keeps_the_manylinux_floor(self): + # what the 6.0.0 jar, cross-built against the ubuntu runner's sysroot, referenced + rel = "x/Linux/s390x/cpu/libjllama.so" + deps = ["libstdc++.so.6", "libm.so.6", "libgcc_s.so.1", "libc.so.6", "ld64.so.1"] + failures = nativedeps.violations(rel, deps, glibc=(2, 38), glibcxx=(3, 4, 32), cxxabi=(1, 3, 13)) + self.assertEqual(len(failures), 3, failures) + for found, floor in (("GLIBC_2.38", "GLIBC_2.28"), ("GLIBCXX_3.4.32", "GLIBCXX_3.4.25"), + ("CXXABI_1.3.13", "CXXABI_1.3.11")): + self.assertTrue(any(found in f and floor in f for f in failures), (found, failures)) + # the build against the manylinux_2_28 sysroot: glibc 2.28 names libpthread and libdl itself + self.assertEqual(nativedeps.violations(rel, deps + ["libpthread.so.0", "libdl.so.2"], glibc=(2, 27), + glibcxx=(3, 4, 25), cxxabi=(1, 3, 11)), []) + + def test_the_manylinux_directories_keep_their_libstdcxx_floor(self): + for rel in ("x/Linux/x86_64/cpu/libllama.so", "x/Linux/aarch64/cpu/libllama.so"): + self.assertEqual(nativedeps.violations(rel, ["libc.so.6"], glibcxx=(3, 4, 25), cxxabi=(1, 3, 11)), []) + failures = nativedeps.violations(rel, ["libc.so.6"], glibcxx=(3, 4, 26)) + self.assertEqual(len(failures), 1, failures) + self.assertIn("GLIBCXX_3.4.26", failures[0]) + self.assertIn("libstdc++_nonshared.a", failures[0]) + rel = "x/Linux/x86_64/cuda13/libggml-cuda.so" + self.assertEqual(len(nativedeps.violations(rel, ["libggml-base.so", "libc.so.6"], runpath="$ORIGIN", + cxxabi=(1, 3, 12))), 1) def test_gpu_modules_are_held_to_the_denylist_and_must_be_modules(self): rel = "x/Linux/x86_64/cuda13/libggml-cuda.so" diff --git a/.github/buildcheck/tests/test_natives.py b/.github/buildcheck/tests/test_natives.py index 52329958a..2391da96c 100644 --- a/.github/buildcheck/tests/test_natives.py +++ b/.github/buildcheck/tests/test_natives.py @@ -79,6 +79,15 @@ def matrix_jobs(targets, download_name="llama-smoke-${{ matrix.target }}"): ("smoke-macos", "package", download("llama-smoke-macos-aarch64"))])) +def floor_jobs(targets, upload_set=True, download_name="llama-glibc-floor"): + """`package` uploading the glibc-floor set (unless `upload_set` is false) and the floor job as a + matrix over `targets`, downloading `download_name`.""" + rows = "\n".join(f" - {{ target: {t}, runner: ubuntu-24.04 }}" for t in targets) + body = [" strategy:", " matrix:", " include:", rows, " steps:"] + download(download_name) + sets = upload("llama-glibc-floor", "smoke-sets/glibc-floor/") if upload_set else [] + return workflow.parse(workflow_text([("package", None, sets), ("smoke-glibc-floor", "package", body)])) + + class ListTest(unittest.TestCase): def test_rows_skip_comments(self): @@ -243,6 +252,34 @@ def test_a_matrix_launches_its_rows(self): self.assertEqual(natives.check_workflow(ROWS, matrix_jobs(["windows-aarch64"], "llama-smoke-linux-x86-64")), ["publish.yml smoke runs (a job downloading llama-smoke-): missing windows-aarch64"]) + def test_the_floor_set_is_the_library_jars_with_a_glibc_ceiling(self): + # cuda13-linux-x86-64 has a ceiling too, but is a module (it needs a CUDA runtime); the + # library jars of Windows, macOS and Android promise no glibc floor + self.assertEqual([r["classifier"] for r in natives.floor_set(ROWS)], ["cpu-linux-x86-64", "cpu-linux-s390x"]) + self.assertEqual(natives.floor_targets(ROWS), ["linux-s390x", "linux-x86-64"]) + + def test_every_floor_jar_is_loaded_on_rhel8(self): + self.assertEqual(natives.check_floor(ROWS, floor_jobs(["linux-x86-64", "linux-s390x"])), []) + self.assertEqual(natives.check_floor(ROWS, floor_jobs(["linux-x86-64"])), + ["publish.yml smoke-glibc-floor: no matrix row for linux-s390x, whose jar promises the " + "glibc floor of nativedeps.GLIBC_CEILING"]) + self.assertEqual(natives.check_floor(ROWS, floor_jobs(["linux-x86-64", "linux-s390x", "windows-aarch64"])), + ["publish.yml smoke-glibc-floor: the matrix row windows-aarch64 is no library jar with a " + "glibc floor (nativedeps.GLIBC_CEILING)"]) + self.assertEqual(natives.check_floor(ROWS, floor_jobs(["linux-x86-64", "linux-s390x"], upload_set=False)), + ["publish.yml: package does not upload llama-glibc-floor, the set smoke-glibc-floor loads " + "(package-smoke-sets.sh writes it)"]) + self.assertEqual(natives.check_floor(ROWS, floor_jobs(["linux-x86-64", "linux-s390x"], + download_name="llama-smoke-linux-x86-64")), + ["publish.yml smoke-glibc-floor does not download llama-glibc-floor"]) + self.assertEqual(natives.check_floor(ROWS, jobs()), + ["publish.yml: package does not upload llama-glibc-floor, the set smoke-glibc-floor loads " + "(package-smoke-sets.sh writes it)", + "publish.yml has no `smoke-glibc-floor` job, which loads linux-s390x, linux-x86-64 on RHEL 8"]) + # a list without a floor jar needs no such job + macos = [r for r in ROWS if r["directory"].startswith("Mac/")] + self.assertEqual(natives.check_floor(macos, jobs()), []) + def test_a_module_jar_without_a_smoke_target_is_refused(self): rows = natives.rows(CSV + "vulkan-linux-s390x,Linux/s390x/vulkan,libggml-vulkan.so,no,module,no\n") failures = natives.check_workflow(rows, jobs(rows=rows)) diff --git a/.github/check-natives.py b/.github/check-natives.py index 3e4b69a45..3057c8d75 100755 --- a/.github/check-natives.py +++ b/.github/check-natives.py @@ -12,6 +12,8 @@ check-natives.py pom print the natives jar executions for llama/pom.xml check-natives.py smoke-targets print the smoke targets (-), one per line check-natives.py smoke-set print the natives jar classifiers of smoke target , one per line + check-natives.py floor-set print the classifiers of the library jars with a glibc floor (the + set smoke-glibc-floor loads on RHEL 8), one per line """ import os @@ -37,6 +39,9 @@ def main(argv): return 2 print("\n".join(r["classifier"] for r in natives.smoke_set(rows, argv[2]))) return 0 + if argv[1:] == ["floor-set"]: + print("\n".join(r["classifier"] for r in natives.floor_set(rows))) + return 0 if argv[1:]: print(__doc__, file=sys.stderr) return 2 diff --git a/.github/manylinux/manylinux_2_28-s390x-sysroot b/.github/manylinux/manylinux_2_28-s390x-sysroot new file mode 100755 index 000000000..fb96ecb12 --- /dev/null +++ b/.github/manylinux/manylinux_2_28-s390x-sysroot @@ -0,0 +1,72 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +# Unpacks the pypa manylinux_2_28 s390x image as a SYSROOT for the cross build of the +# cpu-linux-s390x natives jar (s390x-toolchain.cmake next to this file): the x86-64 runner compiles +# with clang against the image's headers and libraries, and nothing of the image is executed -- +# the build stays at native speed, and pulling a foreign-architecture image needs no qemu. +# +# Why this image: it is what the other two Linux CPU jars are built in (../dockcross +# manylinux_2_28-x64, manylinux_2_28-aarch64), so s390x gets the same floor by the same means. +# Contents (checked 2026-10-25): AlmaLinux 8.10, glibc 2.28, gcc-toolset-14 (GCC 14.2.1) with the +# system libstdc++ of GCC 8 (GLIBCXX_3.4.25). gcc-toolset's libstdc++.so is a linker script that +# takes the system libstdc++.so.6 plus libstdc++_nonshared.a, which links what GCC 14's headers +# need beyond 3.4.25 statically -- the gcc-toolset treatment, here exactly as in a native build, +# because clang uses the toolset's install directory (--gcc-install-dir) for headers and libraries. +# The previous build used the cross toolchain of the ubuntu runner, whose glibc 2.39 sysroot and +# GCC 13 libstdc++ made the jar need glibc 2.38 and GLIBCXX_3.4.32 (CLAUDE.md, "Linux s390x"). +# +# Pinned by digest, tag 2026.10.03-1 (the tag of the aarch64 image). Bump it deliberately, with a +# Publish run: the image decides the artifact's glibc floor (verify-native-deps.py holds +# Linux/s390x/cpu to GLIBC_2.28) and its C++ runtime. +# +# Only what a compile, a link and the qemu run of the test binary read is unpacked (~300 MB): +# the dynamic loader symlinks, /usr/lib64, /usr/include and gcc-toolset-14's headers and +# compiler directory. The image's absolute symlinks (e.g. /usr/lib/ld64.so.1 -> /usr/lib64/...) +# are rewritten relative so that they resolve inside the sysroot, not on the host. +# +# Usage: .github/manylinux/manylinux_2_28-s390x-sysroot +set -euo pipefail + +IMAGE="quay.io/pypa/manylinux_2_28_s390x@sha256:c64d9390efad901ed09ca959df4bc441468c65478f5750e37197813a6017a49c" +GCC_DIR="opt/rh/gcc-toolset-14/root/usr/lib/gcc/s390x-redhat-linux/14" + +if [ "$#" -ne 1 ]; then + echo "usage: $0 " >&2 + exit 2 +fi +SYSROOT="$1" +mkdir -p "$SYSROOT" +if [ -n "$(ls -A "$SYSROOT")" ]; then + echo "$0: $SYSROOT is not empty" >&2 + exit 2 +fi + +docker pull --quiet --platform linux/s390x "$IMAGE" >/dev/null +cid="$(docker create --platform linux/s390x "$IMAGE" /bin/true)" +trap 'docker rm "$cid" >/dev/null' EXIT +docker export "$cid" | tar -x --no-same-owner -C "$SYSROOT" \ + lib lib64 usr/lib/ld64.so.1 usr/lib64 usr/include \ + opt/rh/gcc-toolset-14/root/usr/include "$GCC_DIR" + +# Absolute symlinks -> relative ones inside the sysroot. A target outside the unpacked set stays +# dangling, as it would be at an absolute path; only those the build reads matter, checked below. +SYSROOT="$(cd "$SYSROOT" && pwd)" +find "$SYSROOT" -type l -lname '/*' -print0 | while IFS= read -r -d '' link; do + target="$(readlink "$link")" + ln -sfn "$(realpath -m --relative-to="$(dirname "$link")" "$SYSROOT$target")" "$link" +done + +# What the toolchain relies on; a different image layout fails here, not in the middle of a build. +for f in "lib/ld64.so.1" "usr/lib64/libc.so.6" "usr/lib64/libstdc++.so.6.0.25" "usr/lib64/libgcc_s.so.1" \ + "$GCC_DIR/crtbeginS.o" "$GCC_DIR/libstdc++.so" "$GCC_DIR/libstdc++_nonshared.a" \ + "opt/rh/gcc-toolset-14/root/usr/include/c++/14/vector"; do + if [ ! -e "$SYSROOT/$f" ]; then + echo "$0: the sysroot lacks $f (image layout changed?)" >&2 + exit 1 + fi +done +echo "s390x sysroot ready: $SYSROOT ($(du -sh "$SYSROOT" | cut -f1), glibc $(basename "$(readlink "$SYSROOT/usr/lib64/libc.so.6")"))" diff --git a/.github/manylinux/s390x-toolchain.cmake b/.github/manylinux/s390x-toolchain.cmake new file mode 100644 index 000000000..5a850da47 --- /dev/null +++ b/.github/manylinux/s390x-toolchain.cmake @@ -0,0 +1,74 @@ +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +# CMake toolchain for the cpu-linux-s390x natives jar (publish.yml build-linux-s390x): clang on the +# x86-64 runner cross-compiles for IBM Z against the manylinux_2_28 s390x image unpacked as a sysroot +# by manylinux_2_28-s390x-sysroot, and GNU ld of binutils-s390x-linux-gnu links. The result has the +# floor of the other Linux CPU jars: glibc 2.28, GLIBCXX_3.4.25, CXXABI_1.3.11 (verify-native-deps.py +# holds GLIBC; the README states all three). +# +# JLLAMA_S390X_SYSROOT the unpacked sysroot (cache variable or environment variable, required) +# CMAKE_C_COMPILER / +# CMAKE_CXX_COMPILER clang-18 / clang++-18 by default (Ubuntu 24.04's clang; the runner is +# pinned to ubuntu-24.04 for it) +# +# Why clang and not a GCC: no packaged x86-64 GCC targets s390x with glibc 2.28, and the image's own +# gcc-toolset-14 is an s390x binary, which would have to run under qemu for the whole build (hours, +# and every tag build is uncached). clang is a cross compiler by construction: --gcc-install-dir +# points it at gcc-toolset-14's directory inside the sysroot, so it takes the same crt files, libgcc +# and libstdc++ (GCC 14 headers, the system libstdc++.so.6 plus libstdc++_nonshared.a through the +# toolset's linker script) that a native gcc-toolset build would. The linker scripts name absolute +# paths (/usr/lib64/libstdc++.so.6, /lib64/libgcc_s.so.1), which ld resolves inside --sysroot. +# +# -march=z13 is the arch11 baseline the previous GCC cross build defaulted to (and RHEL 8's): z13 is +# the oldest machine RHEL 8, SLES 15 and Ubuntu support. It does not switch on ggml's vector code: +# that needs __VEC__ (-mzvector), which nothing passes -- the job builds the scalar s390x path, see +# the GGML_NATIVE note in publish.yml. +# +# CMAKE_CROSSCOMPILING_EMULATOR runs the test binaries under qemu-user with the sysroot as the +# loader prefix, so ctest exercises the jar's libraries against glibc 2.28 and the GCC 8 +# libstdc++.so.6 -- the oldest runtime the jar promises, not the build host's. The prefix goes in +# as QEMU_LD_PREFIX, not as qemu's -L: googletest's death tests re-execute the test binary, and that +# child reaches qemu through binfmt_misc without any argument, so only the environment carries the +# prefix to it (with -L the six RpcComm death tests failed on "Could not open '/lib/ld64.so.1'"). + +set(CMAKE_SYSTEM_NAME Linux) +set(CMAKE_SYSTEM_PROCESSOR s390x) + +# Cached, so that the reconfigure a build triggers on its own (a changed CMakeLists.txt) still finds +# the sysroot when the environment variable is gone. +if(NOT JLLAMA_S390X_SYSROOT AND DEFINED ENV{JLLAMA_S390X_SYSROOT}) + set(JLLAMA_S390X_SYSROOT "$ENV{JLLAMA_S390X_SYSROOT}" CACHE PATH + "The manylinux_2_28 s390x sysroot (.github/manylinux/manylinux_2_28-s390x-sysroot)") +endif() +if(NOT JLLAMA_S390X_SYSROOT OR NOT EXISTS "${JLLAMA_S390X_SYSROOT}/usr/lib64/libc.so.6") + message(FATAL_ERROR "s390x-toolchain.cmake: set JLLAMA_S390X_SYSROOT to a sysroot unpacked by " + ".github/manylinux/manylinux_2_28-s390x-sysroot (got '${JLLAMA_S390X_SYSROOT}')") +endif() +# try_compile projects re-read this file; hand them the sysroot too. +list(APPEND CMAKE_TRY_COMPILE_PLATFORM_VARIABLES JLLAMA_S390X_SYSROOT) + +set(CMAKE_SYSROOT "${JLLAMA_S390X_SYSROOT}") +if(NOT CMAKE_C_COMPILER) + set(CMAKE_C_COMPILER clang-18) +endif() +if(NOT CMAKE_CXX_COMPILER) + set(CMAKE_CXX_COMPILER clang++-18) +endif() +set(CMAKE_C_COMPILER_TARGET s390x-linux-gnu) +set(CMAKE_CXX_COMPILER_TARGET s390x-linux-gnu) + +set(_jllama_s390x_gcc "${JLLAMA_S390X_SYSROOT}/opt/rh/gcc-toolset-14/root/usr/lib/gcc/s390x-redhat-linux/14") +set(CMAKE_C_FLAGS_INIT "--gcc-install-dir=${_jllama_s390x_gcc} -march=z13") +set(CMAKE_CXX_FLAGS_INIT "--gcc-install-dir=${_jllama_s390x_gcc} -march=z13") +foreach(_kind EXE SHARED MODULE) + set(CMAKE_${_kind}_LINKER_FLAGS_INIT "--ld-path=/usr/bin/s390x-linux-gnu-ld") +endforeach() + +set(CMAKE_CROSSCOMPILING_EMULATOR /usr/bin/env "QEMU_LD_PREFIX=${JLLAMA_S390X_SYSROOT}" /usr/bin/qemu-s390x-static) + +set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) +set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) diff --git a/.github/package-smoke-sets.sh b/.github/package-smoke-sets.sh index 4263cdf0b..677fd8610 100755 --- a/.github/package-smoke-sets.sh +++ b/.github/package-smoke-sets.sh @@ -11,6 +11,11 @@ # plus the GPU classifiers), with no fat jar in between. The sets are the only thing the smoke jobs # download: the natives jars of the other platforms (ROCm alone is hundreds of MB) stay behind. # +# One more set, /glibc-floor/: the classes jar, its dependencies and every library jar whose +# directory promises a glibc floor (check-natives.py floor-set: today the three Linux CPU jars), which +# the smoke-glibc-floor job loads on RHEL 8, on each jar's own architecture (smoke-glibc-floor.sh). +# Its name is no -, so it never collides with a smoke target. +# # Fail-loud: a natives jar the list names that is not in aborts (the package job's # natives profile builds one per row; a missing one means a row without a build), and so does a # target without any jar. @@ -18,7 +23,7 @@ # Usage: package-smoke-sets.sh # directory holding llama-.jar and the llama--.jar natives jars # the classes jar's runtime dependencies as jars (mvn dependency:copy-dependencies) -# output directory; one subdirectory per smoke target +# output directory; one subdirectory per smoke target, plus glibc-floor/ set -euo pipefail TARGET_DIR="${1:?usage: package-smoke-sets.sh }" @@ -55,3 +60,17 @@ for target in $targets; do find "$set_dir" -maxdepth 1 -name 'llama-*.jar' -exec basename {} \; | sort | sed 's/^/ /' done echo "$(wc -w <<< "$targets") smoke set(s) in $OUT_DIR" + +floor="$(python3 "$CHECK" floor-set)" +[ -n "$floor" ] || fail "check-natives.py floor-set printed no classifier" +floor_dir="$OUT_DIR/glibc-floor" +mkdir -p "$floor_dir" +cp "$classes" "$floor_dir/" +cp "$DEPS_DIR"/*.jar "$floor_dir/" +for classifier in $floor; do + jar="$TARGET_DIR/llama-$version-$classifier.jar" + [ -f "$jar" ] || fail "natives jar missing for the glibc-floor set: $jar" + cp "$jar" "$floor_dir/" +done +echo "glibc-floor set ($(du -sh "$floor_dir" | cut -f1)):" +find "$floor_dir" -maxdepth 1 -name 'llama-*.jar' -exec basename {} \; | sort | sed 's/^/ /' diff --git a/.github/smoke-glibc-floor.sh b/.github/smoke-glibc-floor.sh new file mode 100755 index 000000000..d9a1a65b3 --- /dev/null +++ b/.github/smoke-glibc-floor.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +# Loads one Linux library jar of the glibc-floor set (package-smoke-sets.sh, check-natives.py +# floor-set) on the oldest distribution its floor promises: RHEL 8.10 -- glibc 2.28 and the system +# libstdc++.so.6 of GCC 8 -- in Red Hat's UBI 8 OpenJDK 21 runtime image, on the jar's own +# architecture. A target that is not the runner's (s390x on an x86-64 runner) runs under qemu-user +# through binfmt_misc, which the job installs (qemu-user-static); the image is multi-arch, so the same +# digest serves every row. +# +# What it proves and nothing else checks: that the dynamic loader of the old runtime accepts the +# library -- every GLIBC_x.y, GLIBCXX_x.y.z and CXXABI_x.y.z it references exists there, and so does +# every DT_NEEDED file. verify-native-deps.py compares the version numbers with the ceiling; this is +# the load itself, from the jar, as a consumer's JVM does it (LlamaLoader extracts the directory and +# System.load()s it), followed by a JNI call (NativeLoadSmoke.java, the model-free load smoke the +# macOS row runs too). The s390x jar of 6.0.0 needed GLIBC_2.38 and failed exactly here +# ("UnsatisfiedLinkError ... version `GLIBC_2.38' not found"), with every job green. +# +# The set holds every floor jar; the loader picks the one of the JVM's os.arch, and the build info it +# prints must name cpu- -- a load of another jar, or none, fails. The image must report glibc +# 2.28, so a drifted image cannot turn the check into a load on a newer runtime. +# +# The smoke class is compiled on the runner (javac --release 8 against the set), so the image needs +# only a JRE. +# +# Pinned by digest: the multi-arch index (amd64, arm64, ppc64le, s390x) of ubi8/openjdk-21-runtime +# 1.23, checked 2026-10-26 -- RHEL 8.10, glibc 2.28, OpenJDK 21.0.12, uid 185. Bump it deliberately; +# it must stay an RHEL 8 image. +# +# Usage: smoke-glibc-floor.sh +# the glibc-floor set: the classes jar, its dependencies, the floor jars +# linux-x86-64 | linux-aarch64 | linux-s390x +set -euo pipefail + +SET_DIR="${1:?usage: smoke-glibc-floor.sh }" +TARGET="${2:?usage: smoke-glibc-floor.sh }" +IMAGE="registry.access.redhat.com/ubi8/openjdk-21-runtime@sha256:4ce885c8f0c78dfaa11d53b12e99b4e61c02c30b65943ac3ac2e77923ce36906" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +fail() { + echo "::error::$*" >&2 + exit 1 +} + +case "$TARGET" in + linux-x86-64) platform="linux/amd64"; machine="x86_64" ;; + linux-aarch64) platform="linux/arm64"; machine="aarch64" ;; + linux-s390x) platform="linux/s390x"; machine="s390x" ;; + *) fail "no glibc-floor platform for target '$TARGET'" ;; +esac +classifier="cpu-$TARGET" +SET_DIR="$(cd "$SET_DIR" && pwd)" +compgen -G "$SET_DIR/llama-*-$classifier.jar" >/dev/null || fail "the set in $SET_DIR has no $classifier jar" +# A foreign target needs a binfmt_misc entry for it: Ubuntu's qemu-user-static registers qemu-s390x, +# Docker Desktop plain s390x. Without one, docker run fails with "exec format error", which the image +# checks below would misreport. +if [ "$(uname -m)" != "$machine" ] && [ -e /proc/sys/fs/binfmt_misc/status ] \ + && ! compgen -G "/proc/sys/fs/binfmt_misc/*$machine*" >/dev/null; then + fail "the runner is $(uname -m) and binfmt_misc has no entry for $machine: install qemu-user-static first" +fi + +classes="$(mktemp -d)" +trap 'rm -rf "$classes"' EXIT +javac -nowarn -Xlint:-options --release 8 -cp "$SET_DIR/*" -d "$classes" "$SCRIPT_DIR/smoke/NativeLoadSmoke.java" +chmod -R a+rX "$classes" + +docker pull --quiet --platform "$platform" "$IMAGE" >/dev/null +log="$(mktemp)" +# The image runs as uid 185; the set and the class are mounted read-only, the extraction goes to /tmp. +set +e +docker run --rm --platform "$platform" -v "$SET_DIR":/set:ro -v "$classes":/cls:ro "$IMAGE" \ + sh -c 'cat /etc/redhat-release; uname -m; ldd --version | head -n 1; java -version 2>&1 | head -n 1; + exec java -cp "/cls:/set/*" NativeLoadSmoke' 2>&1 | tee "$log" +rc=${PIPESTATUS[0]} +set -e + +grep -q "^Red Hat Enterprise Linux release 8\." "$log" || fail "the image is no RHEL 8 (see the first line above)" +grep -q "^$machine\$" "$log" || fail "the container does not run on $machine" +grep -qE "^ldd .* 2\.28\$" "$log" || fail "the image's glibc is not 2.28 -- the load proves nothing about the floor" +[ "$rc" -eq 0 ] || fail "$classifier does not load on RHEL 8 ($machine), exit $rc -- see the output above" +grep -q "native load smoke OK: .*classifier=$classifier[,}]" "$log" \ + || fail "the JVM did not load $classifier (no build info naming it above)" +echo "$classifier loads on RHEL 8 ($machine, glibc 2.28)" diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 0e7d81120..e149740b4 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -888,19 +888,25 @@ jobs: build-linux-s390x: name: Build and Test Linux s390x (big-endian, qemu) needs: [startgate, build-webui] - # Cross-compile for IBM Z (s390x, BIG-ENDIAN) with the GCC cross toolchain, then run the full - # C++ unit suite under qemu-user — a real big-endian correctness gate for our helpers and - # serializers (esp. the little-endian WAV writer, JSON/token/embedding transforms). The BUILD - # is native speed (x86 cross-gcc); only the tiny test binary is emulated. s390x is a CPU - # platform like aarch64 and ships as the cpu-linux-s390x natives jar. Model-backed Java tests - # are NOT run under emulation (a JVM + GGUF inference under qemu-user is slow/flaky); the - # C++ gate covers the actual byte-order risk since the Java<->JNI boundary uses host-native - # array copies. GGML_OPENMP=OFF avoids cross-libgomp issues (ggml uses - # its own std::thread pool). CMAKE_CROSSCOMPILING_EMULATOR makes ctest run the s390x exe via qemu; - # QEMU_LD_PREFIX lets the emulated binary find the s390x sysroot libs. - runs-on: ubuntu-latest + # Cross-compile for IBM Z (s390x, BIG-ENDIAN), then run the full C++ unit suite under qemu-user -- + # a real big-endian correctness gate for our helpers and serializers (esp. the little-endian WAV + # writer, JSON/token/embedding transforms). The BUILD is native speed: clang on the x86-64 runner + # against the manylinux_2_28 s390x image unpacked as the sysroot (.github/manylinux/ + # manylinux_2_28-s390x-sysroot and s390x-toolchain.cmake), linked by binutils-s390x-linux-gnu, so + # the jar has the floor of the other Linux CPU jars -- glibc 2.28, GLIBCXX_3.4.25 -- which + # verify-native-deps.py holds and smoke-glibc-floor proves on RHEL 8. (Until 2026-10-25 the job + # used the runner's GCC 13 cross toolchain and its glibc 2.39 sysroot: the 6.0.0 jar needed glibc + # 2.38 and GCC 13's libstdc++, CLAUDE.md "Linux s390x".) Only the test binary is emulated, and it + # runs against the sysroot's glibc 2.28 and GCC 8 libstdc++.so.6, the oldest runtime the jar + # promises. s390x is a CPU platform like aarch64 and ships as the cpu-linux-s390x natives jar. + # Model-backed Java tests are NOT run under emulation (a JVM + GGUF inference under qemu-user is + # slow/flaky); the C++ gate covers the actual byte-order risk since the Java<->JNI boundary uses + # host-native array copies. GGML_OPENMP=OFF avoids cross-libgomp issues (ggml uses its own + # std::thread pool). The toolchain's CMAKE_CROSSCOMPILING_EMULATOR runs the s390x executables + # through qemu with QEMU_LD_PREFIX pointing at the sysroot. + # Pinned to ubuntu-24.04: the toolchain names its clang-18. + runs-on: ubuntu-24.04 env: - QEMU_LD_PREFIX: /usr/s390x-linux-gnu SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 @@ -915,10 +921,14 @@ jobs: with: distribution: 'temurin' java-version-file: .java-version - - name: Install s390x cross toolchain + qemu-user + - name: Install clang, the s390x binutils and qemu-user run: | sudo apt-get update - sudo apt-get install -y gcc-s390x-linux-gnu g++-s390x-linux-gnu qemu-user-static + sudo apt-get install -y clang-18 binutils-s390x-linux-gnu qemu-user-static + - name: Unpack the manylinux_2_28 s390x sysroot + run: | + .github/manylinux/manylinux_2_28-s390x-sysroot "$RUNNER_TEMP/s390x-sysroot" + echo "JLLAMA_S390X_SYSROOT=$RUNNER_TEMP/s390x-sysroot" >> "$GITHUB_ENV" # NOTE: GGML_NATIVE=OFF is required here (an x86 build host must not bake -march=native into an # s390x artifact), and it has a non-obvious second effect: ggml declares # `option(GGML_VXE "ggml: enable vxe" ${GGML_NATIVE})`, so VXE is off too. No `-mvx -mzvector` is @@ -926,16 +936,20 @@ jobs: # never self-defines `__VXE__`/`__VXE2__`. This job therefore builds a SCALAR s390x binary -- which is # exactly right for what it is (a big-endian correctness gate for our own layer, not a perf target). # Do NOT "fix" a VXE-related compile error by adding -DGGML_VXE=ON: that define sets __VXE__ and - # __VXE2__ together while -march stays at the toolchain default arch11, so every z14+ builtin is - # rejected. See the `0013` note in docs/history/dropped-llama-patches.md for the measured + # __VXE2__ together while -march stays at z13 (arch11, s390x-toolchain.cmake), so every z14+ + # builtin is rejected. See the `0013` note in docs/history/dropped-llama-patches.md for the measured # comparison (the patch itself is gone -- upstream merged it as ggml-org/llama.cpp#28775). - name: Build libraries (cross-compile s390x) shell: bash run: | mvn --no-transfer-progress -f llama/pom.xml compile - .github/build.sh "-DGGML_NATIVE=OFF -DGGML_OPENMP=OFF -DBUILD_TESTING=ON -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=s390x -DCMAKE_C_COMPILER=s390x-linux-gnu-gcc -DCMAKE_CXX_COMPILER=s390x-linux-gnu-g++ -DCMAKE_CROSSCOMPILING_EMULATOR=/usr/bin/qemu-s390x-static -DOS_NAME=Linux -DOS_ARCH=s390x" + .github/build.sh "-DGGML_NATIVE=OFF -DGGML_OPENMP=OFF -DBUILD_TESTING=ON -DCMAKE_TOOLCHAIN_FILE=$GITHUB_WORKSPACE/.github/manylinux/s390x-toolchain.cmake -DOS_NAME=Linux -DOS_ARCH=s390x" - name: Run C++ unit tests under qemu-s390x (big-endian gate) run: ctest --test-dir llama/build --no-tests=error --output-on-failure + # The same check `package` runs over the merged tree, here so a lost floor fails in this job: + # GLIBC_2.28, GLIBCXX_3.4.25, CXXABI_1.3.11 and the exact dependency list. + - name: Check the dependencies and the glibc floor + run: python3 .github/verify-native-deps.py llama/src/main/natives/net/ladenthin/llama/ - name: Upload artifacts uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7 with: @@ -3273,6 +3287,56 @@ jobs: compression-level: 0 retention-days: 7 if-no-files-found: error + # The library jars that promise a glibc floor (check-natives.py floor-set), with the classes + # jar and its dependencies, for smoke-glibc-floor below. + - name: Upload the glibc-floor set + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7 + with: + name: llama-glibc-floor + path: smoke-sets/glibc-floor/ + compression-level: 0 + retention-days: 7 + if-no-files-found: error + + # Every library jar that promises a glibc floor (nativedeps.GLIBC_CEILING: the three Linux CPU + # jars) loaded on the oldest distribution of that floor, RHEL 8.10 (glibc 2.28, GCC 8's + # libstdc++.so.6), on its own architecture: x86-64 and aarch64 natively, s390x under qemu-user + # (smoke-glibc-floor.sh, NativeLoadSmoke.java in Red Hat's UBI 8 OpenJDK 21 runtime image). + # verify-native-deps.py in `package` compares the symbol versions with the ceiling; this is the load + # itself. Before it no job ran a jar on an old runtime, and the s390x jar of 6.0.0 shipped needing + # glibc 2.38 with every job green (CLAUDE.md, "Linux s390x"). check-natives.py holds the rows to the + # floor set. Model-free and quick (the s390x row JIT-starts a JVM under qemu: ~15 s locally). + smoke-glibc-floor: + name: Load the Linux CPU jars on RHEL 8 (${{ matrix.target }}) + needs: [package] + strategy: + fail-fast: false + matrix: + include: + - { target: linux-x86-64, runner: ubuntu-24.04 } + - { target: linux-aarch64, runner: ubuntu-24.04-arm } + - { target: linux-s390x, runner: ubuntu-24.04 } + runs-on: ${{ matrix.runner }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false + - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 + with: + name: llama-glibc-floor + path: floor/ + # javac for the smoke class; the image brings its own JRE. + - uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6 + with: + distribution: 'temurin' + java-version-file: .java-version + - name: Install qemu-user (s390x on the x86-64 runner) + if: matrix.target == 'linux-s390x' + run: | + sudo apt-get update + sudo apt-get install -y qemu-user-static + - name: Load the jar on RHEL 8 + run: .github/smoke-glibc-floor.sh floor ${{ matrix.target }} # Every smoke set, launched on a GPU-less runner of its OS/arch: the loader puts every GPU module # jar of the platform next to the CPU library (the loader line names each one, checked against @@ -3553,6 +3617,7 @@ jobs: - smoke-natives - smoke-natives-macos - smoke-agent-linux + - smoke-glibc-floor - vmlens - repository-checks if: needs.check-snapshot.result == 'success' && inputs.publish_to_central @@ -3678,6 +3743,7 @@ jobs: - smoke-natives - smoke-natives-macos - smoke-agent-linux + - smoke-glibc-floor - vmlens - repository-checks runs-on: ubuntu-latest diff --git a/CHANGELOG.md b/CHANGELOG.md index d5cac8b6a..9c379304a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,26 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by ## [Unreleased] +### Fixed +- **`cpu-linux-s390x` loads on glibc 2.28, like the other Linux CPU jars** (RHEL 8, Ubuntu 20.04, + Debian 10 and later, z13 or newer). The 6.0.0 jar was cross-compiled against the + ubuntu runner's glibc 2.39 sysroot and needed `GLIBC_2.38`, `GLIBCXX_3.4.32` and `CXXABI_1.3.13`, so + it failed on RHEL 8/9 and Ubuntu 22.04 with `GLIBC_2.38 not found`. `build-linux-s390x` now + cross-compiles with clang against the manylinux_2_28 s390x image unpacked as a sysroot + (`.github/manylinux/manylinux_2_28-s390x-sysroot`, `s390x-toolchain.cmake`: glibc 2.28, + gcc-toolset-14's libstdc++ treatment); the library references `GLIBC_2.27`, `GLIBCXX_3.4.25` and + `CXXABI_1.3.11`, and its C++ suite runs under qemu against that runtime. + +### Added +- **`smoke-glibc-floor`: every Linux CPU jar is loaded on RHEL 8.10 in CI**, on its own architecture + (x86-64 and aarch64 natively, s390x under qemu), from the published jars, in Red Hat's UBI 8 OpenJDK + 21 runtime image (glibc 2.28, GCC 8's libstdc++). The set is derived from the glibc floors + (`check-natives.py floor-set`), and `check-natives.py` fails when the job's rows differ from it. +- **`verify-native-deps.py` also holds the libstdc++ floor**: the directories with a glibc floor may + not reference a version above `GLIBCXX_3.4.25` / `CXXABI_1.3.11`, and `Linux/s390x/cpu` is held to + `GLIBC_2.28` like the other manylinux directories. Its output names the highest `GLIBC`, `GLIBCXX` + and `CXXABI` version of every library. + ## [6.0.0] - 2026-10-25 ### Upgrading from 5.1 diff --git a/CLAUDE.md b/CLAUDE.md index 986df3362..536d03592 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -101,7 +101,8 @@ it or is checked against it: | `examples/jbang/Chat.java`, `examples/server/pom.xml` | the JBang `//DEPS` lines name the classes jar and every `platform=yes` jar; the server example's `` and profiles follow the list — both held to the README's install version | | `.github/check-natives.py` | runs in `code-style` (first minutes of every run); fails when any row of this table disagrees with the list — including CMake's backend names, the dependency allowlists and the README rows | | `.github/merge-native-artifacts.sh` | reads the list: every listed `natives-*` artifact present, no other, each holding its library (or module) and nothing outside its directory, each with a `jllama-build.txt` naming a tag, the provenance keys and the row's classifier, next to a `jllama-sbom.json` naming the same classifier, a module directory holding nothing but its module, the `jllama-*.txt` lists and the BOM, no path claimed twice; writes `jllama-extras.txt` (sibling files loaded before the library, e.g. OpenVINO's `OpenCL.dll` on Windows -- never the files a build's own `jllama-files.txt` names, which are only extracted; each of those must exist) | -| `.github/package-smoke-sets.sh` | reads the list through `check-natives.py smoke-targets` / `smoke-set`: fails when a natives jar of a set was not built | +| `.github/package-smoke-sets.sh` | reads the list through `check-natives.py smoke-targets` / `smoke-set`: fails when a natives jar of a set was not built; also writes the glibc-floor set (`floor-set`) | +| the glibc-floor set | **derived**: the library rows whose directory promises a glibc floor (`nativedeps.GLIBC_CEILING`: the three Linux CPU jars) — `check-natives.py floor-set` prints them; `package` uploads them with the classes jar and its dependencies as `llama-glibc-floor`, and `smoke-glibc-floor` loads each on RHEL 8.10 (glibc 2.28, GCC 8's libstdc++) on its own architecture, s390x under qemu (`smoke-glibc-floor.sh`, `NativeLoadSmoke.java` in the UBI 8 OpenJDK 21 runtime image); `check-natives.py` holds the job's matrix rows to the set | | `.github/verify-native-deps.py` | every library of a directory: exact dependency allowlist per library directory (`cpu`/`metal`) plus the files next to it; **the export set of every `jllama` library** (`JNI_OnLoad`, `JNI_OnUnload`, `Java_*`, `jllama_*` and nothing else -- `cmake/jllama-exports.map` on ELF and `cmake/jllama-exports-macos.txt` on macOS keep the statically linked llama/ggml/mtmd/BoringSSL symbols local, Windows exports only `dllexport`; before the lists the Linux library exported 14,220 symbols, 47 of them JNI), and for the Android OpenCL module (bionic + `libOpenCL.so` + `libggml-base.so`); denylist for the other GPU modules, each of which must import `ggml-base` and, as ELF, reach it only through run path `$ORIGIN` (an Android library carries no run path -- bionic's search path is the app's native-library directory -- and only a build-tree one fails there); the manylinux_2_28 builds (both Linux CPU jars, CUDA) held to `GLIBC_2.28`; every Android library 16 KB page-aligned (Google Play). Runs in `package` and on the staged AAR libraries | | `.github/smoke-natives-jars.sh` (`package` job) | loads the real jars: classes + all 23 natives jars at once, on the classpath **and** the module path; it passes `net.ladenthin.llama.backend` naming **every** module of the runner's platform, opt-in ones included, and the loader line must then name each -- so a module jar the loader skipped fails here, and this is also the only place in CI that loads an opt-in module at all | | `.github/smoke-lib.sh` / `smoke-test-natives.ps1` | the model-backed smoke rows expect every module of the set **except** the opt-in ones (`optin` column), because the loader deliberately leaves those out -- it is the same list, read with one more condition | @@ -299,11 +300,15 @@ measurements behind the Windows step are in `docs/handover/local-agent-report-b1 build. - **Built in manylinux_2_28** (AlmaLinux 8, gcc-toolset-14, glibc 2.28): x86-64 in the dockcross image the CUDA job already used, aarch64 natively on `ubuntu-24.04-arm` inside the pypa image - (`.github/manylinux/manylinux_2_28-aarch64`, see "Linux aarch64"). manylinux2014's GCC 10 lacked - `-mavxvnni` and `-mamx-tile` (`alderlake`, `sapphirerapids`; ggml does not check). The glibc floors - meet at **2.28**: x86-64 rises from 2.17, aarch64 falls from ~2.39. `verify-native-deps.py` reads - every library's version-needs table and holds both directories (and CUDA) to `GLIBC_2.28`; the - modules themselves reference `GLIBC_2.27` at most (measured in the image). + (`.github/manylinux/manylinux_2_28-aarch64`, see "Linux aarch64"); the static s390x library is + cross-compiled against the pypa s390x image as a sysroot (see "Linux s390x"). manylinux2014's GCC 10 + lacked `-mavxvnni` and `-mamx-tile` (`alderlake`, `sapphirerapids`; ggml does not check). The glibc + floors meet at **2.28**: x86-64 rises from 2.17, aarch64 falls from ~2.39, s390x from 2.38. + `verify-native-deps.py` reads every library's version-needs table and holds these directories (and + CUDA) to `GLIBC_2.28`, `GLIBCXX_3.4.25` and `CXXABI_1.3.11` (the system libstdc++ of manylinux_2_28 + is GCC 8's; gcc-toolset links the rest from `libstdc++_nonshared.a`); the modules themselves + reference `GLIBC_2.27` at most (measured in the image). `smoke-glibc-floor` loads each of the three + CPU jars on RHEL 8.10. - **One `ctest` in a modular build job, `c_abi`, and no `jllama_test`**: `jllama_test` calls the CPU/RPC backends directly, so the option refuses `BUILD_TESTING`; what a library-jar build does run (`JLLAMA_C_ABI_TEST`, on by default, not when cross-compiling) is `src/test/c/test_c_abi.c` @@ -2549,9 +2554,37 @@ Wiring: ### Linux s390x: big-endian cross-build + qemu test gate `build-linux-s390x` adds **IBM Z (s390x, big-endian)** — the one target whose -byte order differs from every other platform. It **cross-compiles** with the GCC s390x toolchain -(`g++-s390x-linux-gnu`, native x86 speed — no emulated build) and then runs the **full C++ unit suite -under `qemu-user`** (`CMAKE_CROSSCOMPILING_EMULATOR=/usr/bin/qemu-s390x-static`, `QEMU_LD_PREFIX=/usr/s390x-linux-gnu`). +byte order differs from every other platform. It **cross-compiles** at native x86 speed — no emulated +build — and then runs the **full C++ unit suite under `qemu-user`**. + +**The toolchain gives it the manylinux_2_28 floor** (glibc 2.28, `GLIBCXX_3.4.25`, `CXXABI_1.3.11`), +like the other two Linux CPU jars, without an s390x runner: +`.github/manylinux/manylinux_2_28-s390x-sysroot` unpacks the pypa `manylinux_2_28_s390x` image +(pinned by digest; nothing of it is executed) as a sysroot, and `.github/manylinux/s390x-toolchain.cmake` +compiles with Ubuntu 24.04's `clang-18` (`--target=s390x-linux-gnu --sysroot=… +--gcc-install-dir= -march=z13`) and links with `binutils-s390x-linux-gnu` +(`--ld-path`). clang takes the toolset's crt files, libgcc and libstdc++ exactly as a native +gcc-toolset build would — the system `libstdc++.so.6` of GCC 8 plus `libstdc++_nonshared.a` through +the toolset's linker script, whose absolute paths ld resolves inside the sysroot. No packaged x86 GCC +targets s390x with glibc 2.28, and running the toolset's own s390x gcc under qemu would take hours per +(uncached) tag build. `-march=z13` is the arch11 baseline the former GCC cross build defaulted to (and +RHEL 8's). The toolchain's emulator is `env QEMU_LD_PREFIX= qemu-s390x-static`, so `ctest` +runs against glibc 2.28 and GCC 8's libstdc++ -- the oldest runtime the jar promises; the prefix must +be an environment variable, not qemu's `-L`, because googletest's death tests re-execute the binary +through binfmt_misc without arguments (with `-L` the six `RpcComm` death tests failed on +`Could not open '/lib/ld64.so.1'`). The sysroot path is a cache variable, so a reconfigure without +`JLLAMA_S390X_SYSROOT` in the environment still finds it. The job then runs `verify-native-deps.py` on +its own tree, and `smoke-glibc-floor` loads the jar on RHEL 8.10 s390x under qemu. + +Until 2026-10-25 the job used the runner's `g++-s390x-linux-gnu` (GCC 13) with its glibc 2.39 +sysroot: the 6.0.0 jar referenced `GLIBC_2.38` (`__isoc23_strtol` & co., `fmod`), `GLIBCXX_3.4.32` +and `CXXABI_1.3.13`, and failed to load on RHEL 8/9 and Ubuntu 22.04 while every job was green -- +`GLIBC_CEILING` held only the manylinux directories, and the qemu `ctest` ran against the same new +sysroot. Measured after the move (b11887): `GLIBC_2.27`, `GLIBCXX_3.4.25`, `CXXABI_1.3.11`, new +dependencies `libpthread.so.0` and `libdl.so.2` (glibc 2.28 has not merged them yet), 694/694 C++ +tests under qemu against the sysroot and natively on a LinuxONE z16 (cfarm191, Debian 13), and the +jar loads on RHEL 8.10 s390x where the 6.0.0 one fails with `GLIBC_2.38 not found`. + That `ctest` run is a **real big-endian correctness gate** for the byte-order-sensitive surface — the little-endian WAV writer (`tts_wav.hpp`), the JSON/token/embedding transforms, and the JNI helpers — which is where an endian bug in *our* code could hide. Model-backed **Java** tests are deliberately diff --git a/README.md b/README.md index 2d21bd234..aba1932eb 100644 --- a/README.md +++ b/README.md @@ -265,7 +265,7 @@ iGPU, the example being a Core Ultra 7 165U). |---|---|---|---| | `cpu-linux-x86-64` | CPU | Linux x86-64 | A JDK 8+ JVM; glibc ≥ 2.28 (manylinux_2_28: RHEL 8, Ubuntu 20.04, Debian 10 and later). Ships one CPU backend module per instruction-set level (x86-64 baseline, SSE4.2, AVX, AVX2, AVX-512, AVX-VNNI, AMX), of which ggml loads the best for the running CPU at start-up — a CPU without AVX2 works, and an AVX-512/AMX machine uses its kernels. | | `cpu-linux-aarch64` | CPU | Linux aarch64 | glibc ≥ 2.28 (RHEL 8, Ubuntu 20.04, Debian 10, Amazon Linux 2023 and later) — built in the manylinux_2_28 image on an arm64 runner. Ships one CPU backend module per ARM feature level (armv8.0 up to armv9.2: dotprod, fp16, SVE, i8mm, SVE2, SME), chosen at start-up like the x86-64 ones. | -| `cpu-linux-s390x` | CPU | Linux s390x (IBM Z, big-endian) | A JDK 8+ JVM; **glibc ≥ 2.38 and the libstdc++ of GCC 13** (`GLIBCXX_3.4.32`): Ubuntu 24.04, SLES 15 SP6, Debian 13 and later. **Not RHEL 8 or 9, not Ubuntu 22.04** -- the load fails there with `GLIBC_2.38 not found`. Cross-compiled on `ubuntu-latest`, not in a manylinux image, so the 2.28 floor of the other Linux CPU jars does not apply. | +| `cpu-linux-s390x` | CPU | Linux s390x (IBM Z, big-endian) | A JDK 8+ JVM; glibc ≥ 2.28 (RHEL 8, Ubuntu 20.04, Debian 10 and later) on z13 or newer — cross-compiled against the manylinux_2_28 s390x image, so the floor of the other Linux CPU jars applies; CI loads it on RHEL 8.10. (The 6.0.0 jar needed glibc 2.38 and did not load on RHEL 8/9 or Ubuntu 22.04.) | | `cpu-windows-x86-64` | CPU | Windows x86-64 | A JDK 8+ JVM; **Windows 10 or newer**. Ships one CPU backend module per instruction-set level (as the Linux x86-64 jar: x86-64 baseline, SSE4.2, AVX, AVX2, AVX-512, AVX-VNNI, AMX), of which ggml loads the best for the running CPU at start-up. Built with clang, hybrid CRT: the STL and vcruntime are static (no `msvcp140.dll` / `vcruntime140.dll`, no VC++ redistributable), the Universal CRT is the OS one (`ucrtbase.dll`), so Microsoft's UCRT security updates apply. | | `cpu-windows-aarch64` | CPU | Windows on ARM (Snapdragon X / Surface) | A JDK 8+ JVM; **Windows 10 or newer**. Built natively on `windows-11-arm` with `clang-cl`, same hybrid CRT as the x86-64 jar; one CPU backend module (ggml builds no ARM variant set for Windows), so the Vulkan and OpenCL module jars can join it. | | `metal-macos-aarch64` | Metal + CPU | macOS aarch64 (Apple silicon) | A JDK 8+ JVM. | diff --git a/TODO.md b/TODO.md index 0bbfe1be1..82bfb1b09 100644 --- a/TODO.md +++ b/TODO.md @@ -472,58 +472,49 @@ ROCm anyway. So this needs a real Radeon machine, or a deliberate decision to sh of a green link plus `verify-native-deps.py`'s dependency allowlist. Do not quietly treat "the job is green" as the measurement. -### Linux s390x: glibc 2.38 floor (after the release) - -**Measured 2026-10-25** on the CI artifacts of Publish runs 40217818548 and 40227501358. - -What each Linux CPU jar references: - -| jar | GLIBC | GLIBCXX | built with | -|---|---|---|---| -| `cpu-linux-x86-64` | 2.27 | 3.4.25 | GCC 14.2.1, manylinux_2_28 | -| `cpu-linux-aarch64` | 2.27 | 3.4.25 | GCC 14.2.1, manylinux_2_28 | -| `cpu-linux-s390x` | **2.38** | **3.4.32** | GCC 13.3.0, Ubuntu 24.04 cross toolchain | - -The 2.38 symbols are `__isoc23_strtol` / `strtoul` / `sscanf` and their relatives, and `fmod`. The -2.34 symbols are `pthread_*` and `dl*`. All of them are only the newest symbol versions of the -cross sysroot (glibc 2.39); no code needs them. - -The load was tested as a consumer does it: `llama` + `cpu-linux-s390x` from `llama-jars` on the -classpath, then `LlamaLoader.initialize()`, in Docker with s390x emulated by qemu. - -- **Fails** on RHEL 8.10 (glibc 2.28) with `UnsatisfiedLinkError … GLIBC_2.38 not found`. `ldd` - also lists `GLIBCXX_3.4.26`..`3.4.32` and `CXXABI_1.3.13`, so libstdc++ blocks it as well. -- **Fails** on RHEL 9 (2.34) and Ubuntu 22.04 (2.35). -- **Loads** on Ubuntu 24.04. -- **`ldd` clean** on SLES 15 SP6 (BCI, glibc 2.38), Debian 13 and Ubuntu 25.04. -- **Control:** `cpu-linux-x86-64` and `cpu-linux-aarch64` load on RHEL 8. - -**Not a decision but a side effect.** `GLIBC_CEILING` in `.github/buildcheck/nativedeps.py` holds -only the manylinux directories. Every other Linux build "inherits the floor of the ubuntu runner"; -CLAUDE.md justifies that for GPU modules only. `1559fe4c` never mentions s390x. The CI test cannot -see the gap: its `ctest` runs under qemu against the same 2.39 sysroot. The README and the -CHANGELOG now state the real floor. - -Follow-ups, none of them before the release (no build change now): - -- [ ] **Pin `build-linux-s390x` to `runs-on: ubuntu-24.04`** instead of `ubuntu-latest`. The output - stays identical (24.04 is `ubuntu-latest` today), but a runner image change can no longer - raise the floor to ~2.42 without anyone noticing. -- [ ] **`"Linux/s390x/cpu": (2, 38)` in `GLIBC_CEILING`.** This freezes today's floor, so - `verify-native-deps.py` fails loudly on any further rise instead of passing it. Extend - `test_nativedeps.py` with it. Optionally add a GLIBCXX ceiling too; the check reads GLIBC - only today. -- [ ] **The real fix: floor 2.28 like the other Linux jars.** Then set `(2, 28)`, and put back the - README row and CHANGELOG item 5. - - **Candidates** (measure them, do not guess): - - (a) build in `quay.io/pypa/manylinux_2_28_s390x` (gcc-toolset-14) as aarch64 does. No - GitHub s390x runner exists: either emulate (slow) or check whether IBM's runner offer for - open-source projects applies; - - (b) cross-compile against a glibc 2.28 s390x sysroot (AlmaLinux 8) with a newer cross GCC. - - **libstdc++:** either way it needs the gcc-toolset treatment (newer parts linked statically, - `GLIBCXX` ≤ 3.4.25). - - **Acceptance:** on real LinuxONE hardware, cfarm191/192 (`workspacePrivate/cfarm`), with - RHEL 8 in Docker as the quick check. +### Linux s390x: the glibc floor is 2.28 (fixed on `fix/s390x-glibc-floor`, 2026-10-26) + +**The defect, measured 2026-10-25** on the CI artifacts of Publish runs 40217818548 and 40227501358: +the 6.0.0 `cpu-linux-s390x` jar referenced `GLIBC_2.38` (`__isoc23_strtol` / `strtoul` / `sscanf` +and relatives, `fmod`), `GLIBCXX_3.4.32` and `CXXABI_1.3.13` -- only the newest symbol versions of +the ubuntu runner's cross sysroot (glibc 2.39, GCC 13.3.0), no code needed them. Loaded as a consumer +does it (`llama` + `cpu-linux-s390x`, `LlamaLoader.initialize()`, s390x under qemu in Docker) it +failed on RHEL 8.10 (`UnsatisfiedLinkError … GLIBC_2.38 not found`), RHEL 9 and Ubuntu 22.04, and +loaded on Ubuntu 24.04; x86-64 and aarch64 (2.27 / 3.4.25, manylinux_2_28) loaded on RHEL 8. Not a +decision but a side effect: `GLIBC_CEILING` held only the manylinux directories, and the qemu `ctest` +ran against the same new sysroot. + +**The fix** (CLAUDE.md, "Linux s390x"): clang-18 cross-compiles against the pypa +`manylinux_2_28_s390x` image unpacked as a sysroot, with gcc-toolset-14's libstdc++ treatment, on a +runner pinned to `ubuntu-24.04`. Measured locally (b11887): `GLIBC_2.27`, `GLIBCXX_3.4.25`, +`CXXABI_1.3.11`, new dependencies `libpthread.so.0` and `libdl.so.2`; 694/694 C++ tests under qemu +against the sysroot's glibc 2.28 and natively on cfarm191 (LinuxONE z16, Debian 13, 2.7 s); the jar +loads on RHEL 8.10 s390x (UBI 8 OpenJDK 21 runtime under qemu) where the 6.0.0 jar still fails with +`GLIBC_2.38 not found` -- the same script, so the check really catches it. + +- [x] `build-linux-s390x` pinned to `ubuntu-24.04` (the toolchain names `clang-18`). +- [x] `"Linux/s390x/cpu": (2, 28)` in `GLIBC_CEILING`, plus a libstdc++ ceiling for every floor + directory (`LIBSTDCXX_CEILING`: `GLIBCXX_3.4.25`, `CXXABI_1.3.11`, which x86-64, aarch64 and + the CUDA module already met); `test_nativedeps.py` covers both, and the job runs + `verify-native-deps.py` on its own tree. +- [x] The floor 2.28 itself: candidate (b) of this list, with clang instead of a cross GCC (none is + packaged for glibc 2.28). Candidate (a), the image's own gcc under qemu, would take hours per + uncached tag build. README row back to 2.28; the 6.0.0 CHANGELOG item 5 stays as it was (it is + what 6.0.0 shipped), [Unreleased] records the fix. +- [x] `smoke-glibc-floor`: the three Linux CPU jars are loaded on RHEL 8.10 in every run, s390x under + qemu; `check-natives.py` holds the rows to `floor-set` (derived from `GLIBC_CEILING`). +- [ ] **Read the first Publish run of this branch** -- none has run yet: `build-linux-s390x` (apt + `clang-18` / `binutils-s390x-linux-gnu`, the `docker pull` of the s390x image on the runner, + sccache wrapping clang: hit rate on the second run), its `verify-native-deps.py` step, + `package` (the `llama-glibc-floor` upload) and the three `smoke-glibc-floor` rows -- the s390x + row needs qemu-user-static's binfmt entry to reach Docker on the runner (it did locally, with + Docker Desktop's own entry), the aarch64 row runs natively on `ubuntu-24.04-arm`. +- [ ] Unmeasured: the speed of the clang build against the former GCC 13 build on real s390x (both + scalar, `-march=z13`). No model-backed check runs on s390x at all (a GGUF would have to be + big-endian), so only `llama-bench` on cfarm191 with a converted model could say. +- [ ] **Owner:** cfarm192 refused the connection with "Host key verification failed" on 2026-10-26 + (cfarm191 worked): its host key changed. Compare the new fingerprint with the farm's published + one before accepting it; nothing was changed in `known_hosts`. - **Ruled out:** `ubuntu-22.04` as the runner (glibc 2.35 still misses RHEL 8/9, and its cross GCC 11 cannot build llama.cpp, which needs GCC >= 12). `-static-libstdc++` alone is ruled out too: it fixes GLIBCXX but not GLIBC. diff --git a/docs/local-test-plan.md b/docs/local-test-plan.md index dd7811e93..48bc71d04 100644 --- a/docs/local-test-plan.md +++ b/docs/local-test-plan.md @@ -22,6 +22,7 @@ expectation phrased like a result. | the C++ suite on x86-64, aarch64, s390x (big-endian, under qemu) | `C++ Tests …`, `build-linux-s390x` | | every native build of the 23 natives jars (8 library, 15 GPU module), plus their dependency allowlists | the build jobs, `package` | | the published jars launched as a consumer's classpath (library jar + every GPU module jar of the platform, the modules skipped for lack of a runtime) | the `smoke-natives` matrix, `smoke-natives-macos`, `smoke-agent-linux` | +| the three Linux CPU jars loaded on their glibc floor, RHEL 8.10 (glibc 2.28), each on its own architecture (s390x under qemu) | the `smoke-glibc-floor` matrix | | the agent's `--web` at HTTP level (401, token cookie, console page) and `--acp` at protocol level | `smoke-agent-linux` | So a green CI run already means: it builds everywhere, the unit suites pass, the jars load, and the From a4b8715134c79843731944aa025cf7c38e0460d0 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 08:14:46 +0100 Subject: [PATCH 03/13] Add a post-release check: audit a published release from the outside RELEASE.md describes how a release is made. Nothing described how one is audited once it exists, so the checks were re-derived by hand every time and the interesting ones -- the artifacts resolved from Central and run on the floor JDK -- were the ones most easily skipped. docs/post-release-check.md is that checklist, written so every item is answerable without trusting the run that produced the artifacts: a green pipeline is evidence about the build, and every failure class CLAUDE.md records as having reached a user reached them through a green pipeline. 16 checks, each stating the failure class it catches, how to run it and the pass criterion. Nothing is pinned to a version: the expected sets are derived from .github/natives.csv and the tag, so the file stays correct for the next release. Two checks deliberately recompute rather than read -- the patch-set hash from the tagged patches, and the publication's artifact count from the module layout -- because reading a provenance key out of a jar proves only that the jar is self-consistent. The last section records the standing gaps a reader should be told rather than discover: the unsigned Windows DLLs and their compensating controls, the post-release minor bump (the patch number stays free for an emergency release off the released line), and scope drift between the reactor poms and the Gradle-built AAR poms. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- docs/RELEASE.md | 4 + docs/post-release-check.md | 348 +++++++++++++++++++++++++++++++++++++ 2 files changed, 352 insertions(+) create mode 100644 docs/post-release-check.md diff --git a/docs/RELEASE.md b/docs/RELEASE.md index 895002b4a..be3bc0aae 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -5,6 +5,10 @@ The maintainer-facing release procedure is **centralized in the workspace repo** java-llama.cpp is a **Maven reactor**, so two repo-specific points extend the canonical procedure. +Once a release is published, audit it from the outside with +[`post-release-check.md`](post-release-check.md): that file checks the artifacts on the GitHub +release, on Maven Central and in a real JVM, deliberately without trusting the run that built them. + ## Reactor version bump (all six poms) The root `pom.xml` is the parent (`net.ladenthin:llama-parent`); the `llama/`, diff --git a/docs/post-release-check.md b/docs/post-release-check.md new file mode 100644 index 000000000..079f9b039 --- /dev/null +++ b/docs/post-release-check.md @@ -0,0 +1,348 @@ + + +# Post-release check + +What to verify **after** a release has been published, from the outside: against the GitHub +release, Maven Central and a real JVM, not against the build tree it came from. + +This is not the release procedure — that is [`RELEASE.md`](RELEASE.md) and the canonical +[workspace release procedure](../../workspace/workflows/release-process.md). Those describe how a +release is *made*; this file describes how a release is *audited once it exists*. The two are kept +apart on purpose: a green pipeline is evidence about the build, and every failure class recorded in +`CLAUDE.md` that reached a user reached them *through* a green pipeline — a jar that loaded on the +runner and nowhere else, a dylib assembled from two artifacts, a model-gated suite that reported +nothing and passed. So every check below is phrased to be answerable **without trusting the run that +produced the artifacts**, and several deliberately recompute from the tagged sources what the +artifacts merely claim. + +**Run it on the published artifacts, every release.** A check that can only say "CI was green" does +not belong in this file. + +## Conventions + +Each check states **what it proves** (the failure class it would catch), **how** to run it, and the +**pass criterion**. Nothing is hardcoded to a version: the expected sets are derived from the +repository at the tag — above all [`.github/natives.csv`](../.github/natives.csv), the one list every +packaging step reads (`CLAUDE.md`, "Natives jars"). Throughout: + +```bash +VERSION= # e.g. 6.0.0 +TAG="v$VERSION" +RUN= # the workflow_dispatch run on $TAG +GH=bernardladenthin/java-llama.cpp +CENTRAL=https://repo1.maven.org/maven2/net/ladenthin +``` + +Work in a scratch directory, and keep the downloaded artifacts in a directory of their own — they +are untrusted input until checked, so no script is run from inside it. + +```bash +gh release download "$TAG" --dir assets +``` + +The whole publication is a few hundred MB (the two CUDA module jars dominate). Download it once: +every check below reads those bytes, and Central is proven equal to them in check 6 rather than +fetched a second time. + +## 1. The ref, the run and the tag agree + +**What it proves.** That the artifacts came from the commit the tag names, through the publishing +workflow, and not from a re-pointed tag or a second attempt whose inputs differed. A tag is a moving +reference; during release preparation it is routinely deleted and re-created, so a cancelled run on +an earlier commit is normal and the only safe statement is the one tying *tag → commit → run → +artifact* together. + +```bash +git fetch origin --tags --force +git rev-parse "$TAG^{commit}" +gh run list --workflow publish.yml --limit 10 \ + --json databaseId,headBranch,headSha,event,conclusion,runAttempt +``` + +**Pass.** Exactly one successful `workflow_dispatch` run on `$TAG`; its `headSha` equals the tag +commit; the tag commit is an ancestor of `main`. Note the attempt number — a re-run attempt is not +wrong, but every later check must then name that same attempt. + +## 2. Every job either succeeded or was skipped for a stated reason + +**What it proves.** That nothing red hid behind a release gate, and that the only skipped jobs are +the ones the other channel owns. A skip is not benign: GitHub propagates a skip down the whole +`needs:` chain past any intervening `if:`, which once turned a run "successful" while `package`, +every smoke set and both publish jobs did nothing (`CLAUDE.md`, "Code signing"). + +```bash +gh run view "$RUN" --json jobs \ + --jq '.jobs | group_by(.conclusion) | map({c:.[0].conclusion, n:length})' +gh run view "$RUN" --json jobs --jq '.jobs[] | select(.conclusion!="success") | .conclusion+" "+.name' +``` + +**Pass.** No failure and no cancellation. The skipped set is exactly the *other* channel's jobs — on +a release run, the snapshot channel check, its gate, its Central upload, its GitHub pre-release and +its attestation. Any other skip is a finding, not a detail. + +## 3. The build was cache-free + +**What it proves.** That the shared compiler cache is outside the release's trust chain. A `v*` tag +build must be from scratch whatever the dispatch input said (`CLAUDE.md`, "CI build cache"), so no +shipped object can come out of a third-party cache. + +```bash +gh api "repos/$GH/actions/jobs//logs" | grep -iE 'USE_CACHE|sccache (ON|OFF)' +``` + +**Pass.** `USE_CACHE: false` in the job environment **and** the build script's own line reporting the +launcher off. Both matter: the first is the intent, the second is what the build did — the build +scripts self-enable a cache they find on `PATH` unless explicitly told not to, which has silently +re-enabled one before. + +## 4. The publication is complete, and its size is derived rather than remembered + +**What it proves.** That no artifact is missing and none is unexpected. The number to compare against +is computed from the repository, never copied from the last release: the natives rows of +`natives.csv`, the reactor modules, the standalone agent and the two Android publications. + +```bash +gh release view "$TAG" --json assets --jq '.assets[] | "\(.size)\t\(.name)"' +``` + +Every file carries a detached `.asc`, so the asset count is even and exactly twice the file count. +Derive the expected Central set: for the core, the classes jar + sources + javadoc + pom + CycloneDX +plus one jar per `natives.csv` row; for each other reactor module its own four or five; for the agent +and the two AARs theirs. The GitHub set is that plus the **signed llama.cpp source archive**, which is +a GitHub-only file of the publication (`CLAUDE.md`, "Source archive"). + +**Pass.** The derived Central count equals the count the publication gate reported ("Publication +ready: … N Central artifacts"), and the GitHub file count is that plus one. Two independent routes to +the same number; agreeing on a wrong number requires both to be wrong the same way. + +## 5. Presence on Maven Central + +**What it proves.** That what was uploaded is actually resolvable. The Portal accepts a bundle and +validates asynchronously, so the gate's success is not evidence that the artifacts are published. + +```bash +for c in $(grep -v '^#' .github/natives.csv | tail -n +2 | cut -d, -f1); do + curl -s -o /dev/null -w "%{http_code} $c\n" "$CENTRAL/llama/$VERSION/llama-$VERSION-$c.jar" +done +``` + +**Pass.** HTTP 200 for every natives jar and its `.asc`, for every module's jar/sources/javadoc/pom, +for both AARs and for every pom-only module. A classifier that 404s is invisible to CI, because CI +never resolves from Central. + +## 6. Every signature verifies, from one key + +**What it proves.** That the publication is signed by the release key throughout, not partly. The +publication gate verifies signatures as well, but with the key the runner imported; this re-verifies +with the key a consumer would trust, and additionally pins that **one** signing subkey signed +everything. + +```bash +cd assets +for a in *.asc; do gpg --status-fd 1 --verify "$a" "${a%.asc}" | grep -oE 'VALIDSIG [A-F0-9]+'; done \ + | awk '{print $2}' | sort | uniq -c +gpg --list-keys --with-subkey-fingerprints +``` + +**Pass.** `GOODSIG` for every file; a single `VALIDSIG` fingerprint across all of them; and that +fingerprint is the `[S]` subkey of the release key. A mixed set of signing keys is a finding even when +every signature is individually good. + +## 7. Build provenance covers every file + +**What it proves.** That each file was produced by this workflow at this commit — which the `.asc` +cannot say, since it attests the key and not the build (`CLAUDE.md`, "Modular natives"). Checking a +sample is not enough: the question is whether anything is *un*attested. + +```bash +gh attestation verify --owner bernardladenthin --format json "assets/llama-$VERSION.jar" > att.json +``` + +Then, from the attestations' subject lists, confirm every local file appears **and** its SHA-256 +matches the recorded digest, and report any local file that appears in neither list. + +**Pass.** Each attestation names `refs/tags/$TAG`, the tag commit, `publish.yml` and the run (and +attempt) from check 1; every published file is covered with a matching digest; nothing is unattested. +Expect more than one attestation and more subjects than files — the `package` job attests the +individual native libraries too — so a subject that is not a release file is normal, while a release +file that is not a subject is not. + +## 8. The natives jars match the list + +**What it proves.** That each jar is the one `natives.csv` declares and carries its provenance +record. This is the merge step's check re-run on the *published* jar: the same properties, read from +the shipped bytes rather than from the build tree. + +Per row of `natives.csv`, assert the jar holds exactly one directory +`net/ladenthin/llama//` (plus `META-INF/`), contains the declared library, carries +`jllama-build.txt` with **all** provenance keys — `llama.cpp`, `llama.cpp.commit`, `backend`, `kind`, +`classifier`, `jllama.version`, `jllama.commit`, `patches`, `patches.sha256`, `compiler`, `toolkit`, +`signed` — whose `classifier`, `kind` and `jllama.version` equal the row and the release, has +`jllama-sbom.json` beside it, and declares +`Automatic-Module-Name: net.ladenthin.llama.natives.`. Unfold the manifest's +72-byte line wraps before reading that name: a wrapped value looks truncated and is not. A `module` +row holds exactly four files — the module, the two list files and the BOM. + +**Pass.** Zero disagreements over all rows. Also compare the per-directory file counts with the ones +`CLAUDE.md` documents per platform ("Modular natives"): a plausible-looking count that moved is how a +dropped CPU variant would present. + +## 9. The provenance is internally consistent, and the patch set is recomputed + +**What it proves.** That the binaries were built from the tagged sources — and this is the one check +that takes the artifact's word for nothing. `patches.sha256` is a hash over ` \n` per +patch; recomputing it from the patches committed at the tag ties the shipped binaries to the reviewed +patch set. Reading the key out of the jar proves only that the jar is self-consistent. + +```bash +git ls-tree --name-only "$TAG" llama/patches/ | sort # extract each with git show +for f in *.patch; do printf '%s %s\n' "$f" "$(sha256sum "$f" | cut -d' ' -f1)"; done | sha256sum +``` + +**Pass.** One `llama.cpp` tag and one `llama.cpp.commit` across all natives jars; `jllama.version` +the release; `jllama.commit` the tag commit **with no `-dirty` suffix**; and the recomputed hash equal +to the `patches.sha256` every jar declares. Record each GPU jar's `toolkit` as well — it is the vendor +SDK that module was built against, and nothing else reports it. + +## 10. The bytecode floors + +**What it proves.** That the artifact a consumer loads can be loaded by the JVM the README promises. +The core's floor is Java 8 (major 52), held by two deliberate pom decisions that are easy to undo by +accident (`CLAUDE.md`, "Java 8 bytecode floor"). The other modules have higher floors on purpose, so +a single global threshold would be wrong. + +Scan every class of every jar, skipping `module-info.class` and `META-INF/versions/**` (a classpath +JVM loads neither): + +| Artifact | Expected class-file major | +|---|---| +| the classes jar and every natives jar | 52, plus a single `module-info` at 53 | +| `llama-kotlin` | 52 | +| `llama-langchain4j` | 61 (Java 17) | +| `llama-ffm` | 66 (Java 22) | +| the standalone agent | 65 (Java 21) | +| each AAR's `classes.jar` | 52, and the classes jar's count minus `module-info` | + +**Pass.** No class above its artifact's floor — and an empty scan is a failure rather than a pass: a +directory a failed build left empty is the reason that rule exists. + +## 11. A consumer can resolve it + +**What it proves.** That the published poms resolve from Central for a real build, property +substitutions included — those come only from the parent pom. CI never resolves the release from +Central, so nothing upstream of this check has tried it. + +Write a throwaway consumer pom depending on `llama-platform` (`pom`) plus a GPU +classifier or two, and resolve it against an **empty local repository**: + +```bash +mvn -B -Dmaven.repo.local=./m2 dependency:tree +mvn -B -Dmaven.repo.local=./m2 dependency:copy-dependencies -DoutputDirectory=./cp -DincludeScope=runtime +``` + +**Pass.** `llama-platform` brings the classes jar and the library jar of every `platform=yes` row; the +classes jar brings jspecify, Jackson and `slf4j-api` at compile and `slf4j-simple` at runtime; and the +graph contains **neither logback nor checker-qual** — both would break a Java 8 consumer, and both are +kept out by pom scope alone, which no compile-time check can see. + +## 12. It runs — on the oldest supported JDK + +**What it proves.** The thing nothing else proves. CI runs the jars on its runners; this runs the +*published* jars, resolved from Central, on the floor JDK. And a load is not the check: in the Windows +measurements recorded in `CLAUDE.md` every crash came *after* a successful load that reported the +right device count, so the criterion is real inference, not initialization. + +On the classpath from check 11, compile a small driver with the **floor JDK's** `javac` against the +published jars and run it on that JDK. Assert, in order: the provenance map (`getJllamaBuildInfo()`) +matches check 9; `getLlamaCppBuildInfo()` starts with the `LlamaCppVersion` constant — the pure-Java +pin and the linked binary must agree; a streamed generation yields tokens; a tokenize/detokenize round +trip; a blocking completion. Then read the loader's own line from stderr. + +**Pass.** Every assertion holds on the floor JDK, with no `UnsupportedClassVersionError` and no SLF4J +provider complaint (that is check 11's pom scopes, observed at runtime). The loader line names the +library backend **and every GPU module jar on the classpath**, and ggml's registry line names the +backends and devices it ended up with — on a GPU host the accelerator must appear, which is the only +evidence that a shipped GPU module opened against a real driver instead of failing silently. Record +the throughput; it is the baseline the next release is read against. + +## 13. The source archive stands on its own + +**What it proves.** That the release stays rebuildable without upstream. The archive is the pinned +upstream commit plus `llama/patches/`, and because `FetchContent` skips population for an overridden +source directory, the applier never runs for a consumer rebuilding from it — the stamp is what proves +the patches are in (`CLAUDE.md`, "Source archive"). + +```bash +tar -xzf "assets/llama.cpp--jllama-$VERSION-src.tar.gz" +cat

/.jllama-patches-applied +``` + +**Pass.** The stamp's `head` equals the `llama.cpp.commit` the natives jars declare; its per-patch +hashes equal the tag's patches line for line (the same lines check 9 recomputed); and each patch's +distinctive added symbol is actually present in the unpacked tree — the stamp records intent, a grep +for what each patch adds records the result. The pack is reproducible: a single mtime (the commit's +time), numeric owner `0/0`, no owner names. Entry order is GNU tar's per-directory `--sort=name`, so +it is deterministic without being flat lexicographic — do not read that as unsorted. + +## 14. The Android artifacts + +**What it proves.** The two invariants that make the AARs consumable, each of which has failed before: +Google Play's 16 KB page size, and a `DT_NEEDED` set bionic can satisfy. + +**Pass.** Both AARs declare the documented `minSdkVersion`; the CPU AAR's `classes.jar` is the classes +jar minus `module-info` and it carries every ABI's library set under `jni//`; the additive AAR +holds exactly its one module, an **empty** `classes.jar` (duplicating the API breaks D8) and a pom +depending on the CPU AAR at `aar` and the same version; both carry their consumer +ProGuard rules; and **every** `LOAD` segment of **every** released Android ELF — in the AARs and in +the Android natives jars alike — is aligned to at least 16384. + +## 15. The server surface and the embedded WebUI + +**What it proves.** That the headline consumer-facing surface works from the published jars: the full +upstream server inside `libjllama`, with the WebUI assets compiled in. A local `cmake` build uses the +empty-asset stub, so "the WebUI is served" is a property of the *released* jar specifically and of no +build a developer makes. + +Start `ServerLauncher` from the check-11 classpath with a model, then probe it. + +**Pass.** `/health` answers 200; `GET /` answers 200 with `Content-Type: text/html` and +`Content-Encoding: gzip`, and the body decompresses to the app shell rather than to an error page; +`/props` answers; and a `/v1/chat/completions` request returns a choice whose `system_fingerprint` is +the linked build from check 9. An empty-asset stub answers `/` with 404 — that is the single +observation separating a released jar from a local one. + +## 16. Standing gaps to re-read every release + +Not failures — properties of the current release that a reader should be told rather than left to +discover. Each is observed by a check above and is expected to read a certain way until the work +behind it lands. + +- **`signed=no` in every natives jar.** The Windows DLLs are not Authenticode-signed; the signing path + exists and stays inert until the certificate and its four secrets do (`CLAUDE.md`, "Code signing"). + The compensating controls are the detached `.asc` and the Sigstore attestation of check 7. Re-read + `signed=` each release: once a release signs, the value flips and + `verify-native-deps.py --require-signed` becomes the gate. +- **The next development version: bump the _minor_, not the patch.** After a release, `main` moves to + the next **minor** `-SNAPSHOT` (`6.0.0` → `6.1.0-SNAPSHOT`), deliberately leaving the patch number + free so an emergency release off the released line can always take it (`6.0.1`). The bump covers all + six reactor poms **and** the standalone agent pom — `versions:set` reaches the reactor but not the + agent, so that one is edited by hand (`CLAUDE.md`, "Repository layout"). + + Until the bump lands, `main` still names the released version, and the next snapshot publication is + refused by its own channel check — correctly, since that channel only publishes a `-SNAPSHOT`, but + the refusal reads like a workflow defect. Right after a release the bump is commonly still in flight + on a branch, which is normal; what this check asks is that it is not *forgotten*. So read it off + `main` and, when it is not there yet, name the branch that carries it: + + ```bash + git show origin/main:pom.xml | grep -m1 -A1 llama-parent + git branch -r --contains + ``` +- **Scope drift between the reactor poms and the Gradle-built AAR poms.** The AAR poms take their + dependency *versions* from the core pom, but their *scopes* are the Gradle build's own. Compare them + every release: a dependency the core deliberately keeps out of the consumer graph at `provided` must + not reappear at `compile` in an AAR pom. From 8fa02dea800d5c8638830db06f1e24ae994f0354 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 08:20:04 +0100 Subject: [PATCH 04/13] TODO: record the post-release findings of 6.0.0 Two entries from auditing the published 6.0.0 artifacts. The finding: both AAR poms ship checker-qual at compile scope, which the core deliberately keeps at provided -- its classes are major 55 with RUNTIME retention, the UnsupportedClassVersionError that #411 put on main. Not a blocker and not that failure on Android, since D8 consumes major 55, so it ships in 6.0.0 unfixed. It went unnoticed because the AAR poms take only their dependency versions from the Maven poms while the scopes are the Gradle build's own, and because CLAUDE.md calls checker-qual one of "the core's compile-scope deps", which it is not. Both halves need the same commit. And measured evidence for the open device-dedup question under "Modular natives": on the published jars the discrete GPU is registered twice, as CUDA0 and Vulkan0, so any device_id dedup happens in selection rather than in registration. The selection half stays open, with the reason it could not be answered in the same run -- the load lines are below the default verbosity threshold -- so the next attempt starts with -lv 4 instead of rediscovering that. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- TODO.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/TODO.md b/TODO.md index 82bfb1b09..3b116aa38 100644 --- a/TODO.md +++ b/TODO.md @@ -194,6 +194,18 @@ Kept as a list because each entry names the machine it needs -- the harness is Also check whether CUDA, Vulkan and OpenCL deduplicate the same NVIDIA GPU by `device_id` and which backend wins. Linux CUDA on a Tesla T4 has already been measured; see [the GPU module measurements](docs/benchmarks/gpu-module-jars.md). + **Partly measured on the published 6.0.0 jars** (post-release check, `docs/post-release-check.md` + check 12; RTX 3070 Laptop + AMD iGPU, the three Windows x86-64 jars resolved from Central): the + loader line names both modules (`with GPU module(s) cuda13, vulkan`), ggml registers + `[CUDA, RPC, Vulkan, CPU]`, and the registry then lists **4 devices -- `CUDA0`, `Vulkan0`, + `Vulkan1`, `CPU`** -- with Vulkan's own enumeration naming device 0 as the same RTX 3070 that CUDA + reports. So the discrete GPU is registered **twice**, once per backend, and whatever `device_id` + dedup llama.cpp does happens in device *selection* rather than in registration. That selection + half is still open, and so is the tg128 comparison above: at the default verbosity threshold the + `load_tensors` / per-buffer lines are filtered out of the log (CLAUDE.md, patch `0014` -- llama + INFO maps to TRACE), so a run that answers it has to pass `-lv 4`. The earlier throughput pair on + this host (4 devices 92.0 t/s against cuda13-only 94.3 t/s, within noise) suggests the iGPU is not + drafted in, which is consistent with selection-time dedup but does not establish it. 2. **CUDA for Windows arm64 (`cuda13-windows-aarch64`) -- upstream builds it, we do not.** Their `windows-cuda` job is a matrix over `x64` and `arm64`, the latter through `cmake/arm64-windows-msvc-cuda` plus the arm64 `cudart`/`cublas` redistributables. This project has @@ -586,3 +598,21 @@ measurements and is not evidence that GPU testing is currently blocked. - **REUSE attribution review:** check the remaining Java/documentation attribution to the original fork author with `git blame`, file by file. Attribution is not removed without an explicit owner decision. The generated dockcross wrappers already carry dockcross's own copyright attribution. +- **The two AAR poms ship `checker-qual` at `compile` scope, which the core deliberately does not** + (found by the post-release check of 6.0.0, `docs/post-release-check.md` check 14/16). Both + `llama-android-6.0.0.pom` and `llama-android-opencl-6.0.0.pom` carry + `org.checkerframework:checker-qual:4.3.1` with an explicit `compile`, so it is in + every Android consumer's dependency graph. In `llama/pom.xml` the same dependency is `provided`, and + the property's own comment says why: "neither ships (checker-qual is provided scope)" — its 417 + classes are all class-file major 55 with `@Retention(RUNTIME)`, which is exactly the + `UnsupportedClassVersionError` that #411 put on `main` (CLAUDE.md, "Java 8 bytecode floor"). + **Not a release blocker and not the Java 8 failure on Android**: D8/AGP consume major 55 fine, so + this is an avoidable dependency rather than a crash, which is why it ships in 6.0.0 unfixed. Two + parts to it, and the second is the reason it was never noticed: the scopes in those poms are + `llama-android/build.gradle.kts`'s own (only the *versions* are parsed out of the Maven poms), and + CLAUDE.md's "Android AAR + Kotlin façade" section describes the AAR pom as mirroring "the core's + compile-scope deps (jackson/slf4j-api/jspecify/checker-qual)" — but checker-qual is not a + compile-scope dep of the core. Fix the Gradle build (drop it, or mirror it as `provided`) and + correct that sentence in the same commit; `jspecify` is the one of the four that is genuinely + compile scope and stays. A cheap guard would be to assert the published AAR poms' scopes against + the core pom's, the way the versions are already derived from it. From c91b086b47aa823836793ca4bbfd6065decede7a Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 08:32:37 +0100 Subject: [PATCH 05/13] AAR poms: mirror the core's compile-scope dependencies from llama/pom.xml, check them in CI The 6.0.0 llama-android and llama-android-opencl poms listed checker-qual 4.3.1 at compile scope, which the core keeps provided (major 55 classes, CLAUDE.md "Java 8 bytecode floor"). The AAR build took the versions from the core pom but kept the list and the scopes by hand, so the scope drifted. - llama-android/build.gradle.kts: coreCompileDependencies reads the core's compile-scope, non-optional dependencies (versions and scopes) from llama/pom.xml. The generated poms list jackson-databind, slf4j-api and jspecify, plus the CPU AAR in the OpenCL one. - consumer-proguard.txt: -dontwarn org.checkerframework.**. The Nullness Checker writes its qualifiers into 133 of the core's 146 class files (RUNTIME retention), so without checker-qual a minified app would fail R8 on "Missing class". - .github/verify-aar-poms.py (buildcheck/aarpoms.py, 8 tests): checks the poms publishToMavenLocal wrote, in package-android-aar. Run on the two 6.0.0 poms from Central it reports checker-qual and nothing else. - CLAUDE.md: the AAR sentence corrected and the bytecode references documented. CHANGELOG [Unreleased], the post-release check and TODO updated. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AJGh28Gu8pLWYbTCqG1LZf --- .github/buildcheck/__init__.py | 2 +- .github/buildcheck/aarpoms.py | 127 ++++++++++++++++++++++ .github/buildcheck/tests/test_aarpoms.py | 132 +++++++++++++++++++++++ .github/verify-aar-poms.py | 17 +++ .github/workflows/publish.yml | 11 ++ CHANGELOG.md | 10 ++ CLAUDE.md | 19 +++- TODO.md | 19 ++++ docs/post-release-check.md | 13 ++- llama-android/build.gradle.kts | 51 +++++++-- llama-android/consumer-proguard.txt | 6 ++ 11 files changed, 389 insertions(+), 18 deletions(-) create mode 100644 .github/buildcheck/aarpoms.py create mode 100644 .github/buildcheck/tests/test_aarpoms.py create mode 100644 .github/verify-aar-poms.py diff --git a/.github/buildcheck/__init__.py b/.github/buildcheck/__init__.py index 7f4e880ef..04a223be5 100644 --- a/.github/buildcheck/__init__.py +++ b/.github/buildcheck/__init__.py @@ -2,7 +2,7 @@ # # SPDX-License-Identifier: MIT OR Apache-2.0 """This repository's own build checks: the natives list, the patches, the native dependencies, the -HIP offload bundles, the WebUI license texts. Standard library only. +HIP offload bundles, the WebUI license texts, the AAR poms. Standard library only. The checks it shares with its sibling repositories live in bernardladenthin/actions (actionlib), and some of the modules here build on it (natives.py reads the job graph with its workflow parser). diff --git a/.github/buildcheck/aarpoms.py b/.github/buildcheck/aarpoms.py new file mode 100644 index 000000000..a6577fa8a --- /dev/null +++ b/.github/buildcheck/aarpoms.py @@ -0,0 +1,127 @@ +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +"""Fail when a published Android AAR pom does not mirror the core's compile-scope dependencies. + +The AAR's classes.jar is the core's classes jar, so an Android consumer needs exactly what a Maven +consumer of net.ladenthin:llama gets: the core's compile-scope, non-optional dependencies, at the +same versions -- no more, no less. The AAR poms are written by llama-android/build.gradle.kts, a +Gradle build of its own, so nothing ties them to llama/pom.xml but this check and that build's +parsing. The 6.0.0 poms shipped checker-qual at compile scope although the core keeps it provided +(major 55 classes, CLAUDE.md "Java 8 bytecode floor"): the versions were derived from the core pom, +the scopes came from a hand-kept list. + +Usage (the CLI is .github/verify-aar-poms.py): verify-aar-poms.py ... + +Each AAR pom may carry, besides the mirrored set, the CPU AAR itself (net.ladenthin:llama-android, +type aar), which the OpenCL AAR depends on. Exits non-zero when a pom lacks a mirrored dependency, +carries another one, or carries one at a different version or a scope other than compile. Only the +standard library, so it runs on any runner. +""" + +import os +import re +import sys +import xml.etree.ElementTree as ET + +# The one dependency an AAR pom may add to the mirrored set: the OpenCL AAR plugs into the CPU AAR. +CPU_AAR = ("net.ladenthin", "llama-android") +PROPERTY = re.compile(r"\$\{([^}]+)\}") + + +def _children(element, tag): + """Direct children by local name, so a pom with or without the Maven namespace reads the same.""" + return [c for c in element if c.tag.rsplit("}", 1)[-1] == tag] + + +def _text(element, tag): + found = _children(element, tag) + return found[0].text.strip() if found and found[0].text else None + + +def _properties(root): + props = _children(root, "properties") + return {c.tag.rsplit("}", 1)[-1]: (c.text or "").strip() for c in props[0]} if props else {} + + +def _dependencies(root): + deps = _children(root, "dependencies") + return _children(deps[0], "dependency") if deps else [] + + +def core_compile_dependencies(core_pom, root_pom): + """{(groupId, artifactId): version} of the core's compile-scope, non-optional dependencies. + + A ${property} version is resolved the way llama-android/build.gradle.kts does it: from the core + pom's properties, then from the parent's. + """ + core = ET.parse(core_pom).getroot() + props = _properties(ET.parse(root_pom).getroot()) + props.update(_properties(core)) + result = {} + for dep in _dependencies(core): + if (_text(dep, "scope") or "compile") != "compile" or _text(dep, "optional") == "true": + continue + key = (_text(dep, "groupId"), _text(dep, "artifactId")) + version = _text(dep, "version") + if version is None: + raise ValueError("%s:%s has no in %s" % (key + (core_pom,))) + match = PROPERTY.fullmatch(version) + if match: + if match.group(1) not in props: + raise ValueError("property <%s> of %s:%s is not defined" % ((match.group(1),) + key)) + version = props[match.group(1)] + result[key] = version + return result + + +def published_dependencies(aar_pom): + """[(groupId, artifactId, version, scope, type)] in pom order; scope and type as written.""" + root = ET.parse(aar_pom).getroot() + return [(_text(d, "groupId"), _text(d, "artifactId"), _text(d, "version"), _text(d, "scope"), + _text(d, "type")) for d in _dependencies(root)] + + +def violations(expected, published): + """What is wrong with one AAR pom's dependencies against the core's compile-scope set.""" + problems = [] + seen = set() + for group, artifact, version, scope, kind in published: + key = (group, artifact) + name = "%s:%s" % key + if key in seen: + problems.append("%s is listed twice" % name) + seen.add(key) + if key == CPU_AAR: + if kind != "aar": + problems.append("%s must be aar, not %s" % (name, kind or "jar")) + continue + if key not in expected: + problems.append("%s (scope %s) is not a compile-scope dependency of the core" % (name, scope or "compile")) + continue + if version != expected[key]: + problems.append("%s is %s, the core has %s" % (name, version, expected[key])) + if (scope or "compile") != "compile": + problems.append("%s has scope %s, the core has compile" % (name, scope)) + for key in sorted(set(expected) - seen): + problems.append("%s:%s %s is missing (a compile-scope dependency of the core)" % (key + (expected[key],))) + return problems + + +def main(argv): + if len(argv) < 3: + print("usage: verify-aar-poms.py ...", file=sys.stderr) + return 2 + repo = argv[1] + expected = core_compile_dependencies(os.path.join(repo, "llama", "pom.xml"), os.path.join(repo, "pom.xml")) + print("core compile-scope dependencies: %s" % ", ".join( + "%s:%s:%s" % (k + (v,)) for k, v in sorted(expected.items()))) + failed = 0 + for pom in argv[2:]: + problems = violations(expected, published_dependencies(pom)) + for p in problems: + print("::error file=%s::%s" % (pom, p)) + print("%s: %s" % (pom, "%d problem(s)" % len(problems) if problems else "mirrors the core")) + failed += bool(problems) + return 1 if failed else 0 diff --git a/.github/buildcheck/tests/test_aarpoms.py b/.github/buildcheck/tests/test_aarpoms.py new file mode 100644 index 000000000..10ed2b979 --- /dev/null +++ b/.github/buildcheck/tests/test_aarpoms.py @@ -0,0 +1,132 @@ +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +import os +import tempfile +import unittest + +from buildcheck import aarpoms +from buildcheck.tests.helpers import REPO + +NS = 'xmlns="http://maven.apache.org/POM/4.0.0"' + + +def dep(group, artifact, version=None, scope=None, optional=None, kind=None): + parts = ["%s" % group, "%s" % artifact] + for tag, value in (("version", version), ("type", kind), ("scope", scope), ("optional", optional)): + if value is not None: + parts.append("<%s>%s" % (tag, value, tag)) + return "%s" % "".join(parts) + + +def pom(deps, properties="", ns=NS): + return "%s%s" % ( + ns, properties, "".join(deps)) + + +# The 6.0.0 llama-android-opencl pom's dependencies, as published. +PUBLISHED_600 = [ + ("com.fasterxml.jackson.core", "jackson-databind", "2.22.4", "compile", None), + ("org.slf4j", "slf4j-api", "2.0.20", "compile", None), + ("org.jspecify", "jspecify", "1.0.1", "compile", None), + ("org.checkerframework", "checker-qual", "4.3.1", "compile", None), + ("net.ladenthin", "llama-android", "6.0.0", "compile", "aar"), +] +EXPECTED_600 = { + ("com.fasterxml.jackson.core", "jackson-databind"): "2.22.4", + ("org.slf4j", "slf4j-api"): "2.0.20", + ("org.jspecify", "jspecify"): "1.0.1", +} + + +class AarPomsTest(unittest.TestCase): + + def write(self, folder, name, text): + path = os.path.join(folder, name) + with open(path, "w", encoding="utf-8") as f: + f.write(text) + return path + + def test_core_set_takes_compile_scope_only_and_resolves_properties_core_first(self): + with tempfile.TemporaryDirectory() as d: + root = self.write(d, "root.xml", pom([], "1root")) + core = self.write(d, "core.xml", pom([ + dep("g", "implicit", "${a.version}"), + dep("g", "explicit", "${b.version}", scope="compile"), + dep("g", "literal", "3.0"), + dep("g", "provided", "1", scope="provided"), + dep("g", "runtime", "1", scope="runtime"), + dep("g", "test", "1", scope="test"), + dep("g", "optional", "1", optional="true"), + ], "core")) + self.assertEqual(aarpoms.core_compile_dependencies(core, root), { + ("g", "implicit"): "1", ("g", "explicit"): "core", ("g", "literal"): "3.0"}) + + def test_core_set_fails_loud_on_an_undefined_property_or_a_missing_version(self): + with tempfile.TemporaryDirectory() as d: + root = self.write(d, "root.xml", pom([])) + core = self.write(d, "core.xml", pom([dep("g", "a", "${nowhere}")])) + with self.assertRaisesRegex(ValueError, "nowhere"): + aarpoms.core_compile_dependencies(core, root) + core = self.write(d, "core.xml", pom([dep("g", "a")])) + with self.assertRaisesRegex(ValueError, "no "): + aarpoms.core_compile_dependencies(core, root) + + def test_reads_a_pom_without_the_maven_namespace(self): + with tempfile.TemporaryDirectory() as d: + path = self.write(d, "a.pom", pom([dep("g", "a", "1", scope="compile")], ns="")) + self.assertEqual(aarpoms.published_dependencies(path), [("g", "a", "1", "compile", None)]) + + def test_the_600_pom_fails_on_checker_qual_and_nothing_else(self): + self.assertEqual(aarpoms.violations(EXPECTED_600, PUBLISHED_600), [ + "org.checkerframework:checker-qual (scope compile) is not a compile-scope dependency of the core"]) + + def test_the_fixed_pom_passes_and_the_cpu_aar_dependency_is_allowed(self): + fixed = [d for d in PUBLISHED_600 if d[1] != "checker-qual"] + self.assertEqual(aarpoms.violations(EXPECTED_600, fixed), []) + self.assertEqual(aarpoms.violations(EXPECTED_600, fixed[:3]), []) # the CPU AAR's own pom + + def test_missing_wrong_version_wrong_scope_duplicate_and_a_jar_typed_cpu_aar(self): + published = [ + ("com.fasterxml.jackson.core", "jackson-databind", "2.0", "compile", None), + ("org.slf4j", "slf4j-api", "2.0.20", "runtime", None), + ("org.slf4j", "slf4j-api", "2.0.20", None, None), + ("net.ladenthin", "llama-android", "6.0.0", "compile", None), + ] + self.assertEqual(aarpoms.violations(EXPECTED_600, published), [ + "com.fasterxml.jackson.core:jackson-databind is 2.0, the core has 2.22.4", + "org.slf4j:slf4j-api has scope runtime, the core has compile", + "org.slf4j:slf4j-api is listed twice", + "net.ladenthin:llama-android must be aar, not jar", + "org.jspecify:jspecify 1.0.1 is missing (a compile-scope dependency of the core)", + ]) + + def test_this_repository_core_set(self): + expected = aarpoms.core_compile_dependencies( + os.path.join(REPO, "llama", "pom.xml"), os.path.join(REPO, "pom.xml")) + for key in (("com.fasterxml.jackson.core", "jackson-databind"), ("org.slf4j", "slf4j-api"), + ("org.jspecify", "jspecify")): + self.assertIn(key, expected) + # provided (major 55 classes) and the JVM-only runtime binding stay out of an Android consumer's graph + self.assertNotIn(("org.checkerframework", "checker-qual"), expected) + self.assertNotIn(("org.slf4j", "slf4j-simple"), expected) + for version in expected.values(): + self.assertNotIn("${", version) + + def test_exit_codes(self): + with tempfile.TemporaryDirectory() as d: + os.makedirs(os.path.join(d, "llama")) + self.write(d, "pom.xml", pom([], "1")) + self.write(os.path.join(d, "llama"), "pom.xml", pom([ + dep("g", "a", "${j.version}"), dep("g", "q", "4", scope="provided")])) + good = self.write(d, "good.pom", pom([dep("g", "a", "1", scope="compile")])) + bad = self.write(d, "bad.pom", pom([dep("g", "a", "1", scope="compile"), + dep("g", "q", "4", scope="compile")])) + self.assertEqual(aarpoms.main(["x", d, good]), 0) + self.assertEqual(aarpoms.main(["x", d, good, bad]), 1) + self.assertEqual(aarpoms.main(["x", d]), 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/verify-aar-poms.py b/.github/verify-aar-poms.py new file mode 100644 index 000000000..6c03d01eb --- /dev/null +++ b/.github/verify-aar-poms.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 +"""Fail when a published Android AAR pom does not mirror the core's compile-scope dependencies. + +See buildcheck/aarpoms.py. Usage: verify-aar-poms.py ... +""" + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from buildcheck import aarpoms # noqa: E402 + +if __name__ == "__main__": + sys.exit(aarpoms.main(sys.argv)) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index e149740b4..c82f15b16 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -1185,6 +1185,17 @@ jobs: ls -la llama-android/natives/*/* - name: Assemble AARs + publish to mavenLocal run: gradle -p llama-android aarCpu aarOpencl publishToMavenLocal + # The AAR's classes are the core's, so its pom must carry exactly the core's compile-scope + # dependencies (versions AND scopes, read from llama/pom.xml): the 6.0.0 poms shipped + # checker-qual at compile scope, which the core keeps provided. Checks the published files. + - name: AAR poms mirror the core's compile-scope dependencies + shell: bash + run: | + set -euo pipefail + VERSION=$(mvn -q -DforceStdout help:evaluate -Dexpression=project.version | tail -n1) + python3 .github/verify-aar-poms.py . \ + "$HOME/.m2/repository/net/ladenthin/llama-android/${VERSION}/llama-android-${VERSION}.pom" \ + "$HOME/.m2/repository/net/ladenthin/llama-android-opencl/${VERSION}/llama-android-opencl-${VERSION}.pom" - name: Validate AAR structure shell: bash run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index 9c379304a..002745137 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,8 +18,18 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by (`.github/manylinux/manylinux_2_28-s390x-sysroot`, `s390x-toolchain.cmake`: glibc 2.28, gcc-toolset-14's libstdc++ treatment); the library references `GLIBC_2.27`, `GLIBCXX_3.4.25` and `CXXABI_1.3.11`, and its C++ suite runs under qemu against that runtime. +- **The Android AAR poms no longer carry `checker-qual`.** The 6.0.0 `llama-android` and + `llama-android-opencl` poms listed `org.checkerframework:checker-qual:4.3.1` at `compile` scope, + which the core keeps `provided`, so it was in every Android consumer's dependency graph. Not a crash + on Android (D8 consumes its major 55 classes), an avoidable dependency. The AAR poms now take their + dependency list from `llama/pom.xml`, scopes included, instead of a hand-kept one, and the consumer + R8 rules gain `-dontwarn org.checkerframework.**` for the checker's qualifiers in the class files. + An app that used checker-qual's annotations itself through the AAR has to declare it. ### Added +- **`verify-aar-poms.py`: the published AAR poms are checked against the core pom** in + `package-android-aar` -- exactly the core's compile-scope dependencies, at its versions and at + `compile`. - **`smoke-glibc-floor`: every Linux CPU jar is loaded on RHEL 8.10 in CI**, on its own architecture (x86-64 and aarch64 natively, s390x under qemu), from the published jars, in Red Hat's UBI 8 OpenJDK 21 runtime image (glibc 2.28, GCC 8's libstdc++). The set is derived from the glibc floors diff --git a/CLAUDE.md b/CLAUDE.md index 536d03592..6d27bfe04 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -370,6 +370,7 @@ rejects anything else loudly. | `patches.py` | `check-patches.py` (`code-style`) | every hunk of `llama/patches/*.patch` declares exactly the lines it carries -- the half `git apply --check` does not see (see "Local llama.cpp source patches"); and `docs/patches.md` is what `check-patches.py --render` generates from the patch headers (a stale page fails) | | `nativedeps.py` | `verify-native-deps.py` (`package`, `package-android-aar`) | dependency allowlists, 16 KB Android alignment | | `hipoffload.py` | `verify-hip-offload-compressed.py` (the two ROCm jobs) | no uncompressed GPU code bundle | +| `aarpoms.py` | `verify-aar-poms.py` (`package-android-aar`, on the poms in mavenLocal) | both AAR poms carry exactly the core's compile-scope dependencies, at its versions and at `compile` (plus the CPU AAR, `type aar`, in the OpenCL one) | | `webuilicenses.py` | `webui-licenses.py` (`build-webui`, and `webui-licenses.yml` on every pull request and push) | the WebUI's npm license texts cover the pinned lockfile (digest over LF line ends) and its imports; it fails every run -- until the release run of v6.0.0 it only warned outside a release tag, and a stale list surfaced there first | | Action (bernardladenthin/actions) | Job here | What it checks or does | @@ -2950,7 +2951,12 @@ easy to undo by accident: own `jar-with-dependencies`, which takes scope `runtime`), so no checker-qual class of any version ships. `true` would not have been enough on its own — that descriptor filters on - scope only. Safe because no source imports `org.checkerframework`. + scope only. Safe because no source imports `org.checkerframework` — but the **class files do + reference it**: the Nullness Checker writes its qualifiers (`@NonNull`, `@UnknownKeyFor`, + `@Initialized`, `@Pure`, `@SideEffectFree`, …) into the classes it checks, 133 of the core's 146 + (measured on the 6.0.0-SNAPSHOT jar). Harmless where checker-qual is absent: a JVM or ART that + reflects over an annotated element skips an annotation type it cannot resolve. R8 does not skip it — + it reports the missing class — hence `-dontwarn org.checkerframework.**` in the AAR's consumer rules. **The gate: the `java/verify-bytecode` action of [bernardladenthin/actions](https://github.com/bernardladenthin/actions)** (its `verify-bytecode-version.sh`). It opens every `.class` in every jar it is given and fails on any whose class-file major version @@ -3406,8 +3412,15 @@ the recommended path (README "Importing in Android", Option 1): full set. The additive OpenCL AAR has an empty `classes.jar`: its CPU AAR dependency already supplies the Java API, and duplicating it breaks D8/R8. Built by the **standalone plain-Gradle build** in `llama-android/` (see "Repository layout" for why it is not a Maven module); the POM mirrors the core's - compile-scope deps (jackson/slf4j-api/jspecify/checker-qual, versions parsed from - `llama/pom.xml` — deliberately NOT the SLF4J binding, which is the JVM-only runtime dependency). + compile-scope deps — the list itself, versions **and scopes**, is read from `llama/pom.xml` + (`coreCompileDependencies`: no `` or `compile`, not optional; today jackson-databind, + slf4j-api, jspecify). Not mirrored: the `provided` ones (checker-qual, lombok, animal-sniffer) and the + SLF4J binding, which is the JVM-only runtime dependency. The 6.0.0 AAR poms were written from a + hand-kept list that also carried checker-qual at `compile`, which the core keeps `provided` (major + 55, see "Java 8 bytecode floor"); the versions were derived and stayed right, the scopes drifted. + `.github/verify-aar-poms.py` checks the poms `publishToMavenLocal` wrote against the core pom in + `package-android-aar`. The consumer rules carry `-dontwarn org.checkerframework.**`: the classes + reference the checker's qualifiers (below), which no Android app has. - **`net.ladenthin:llama-kotlin`** — Maven reactor module; pure-Kotlin (2.4, jvmTarget 1.8) coroutines façade: `generateFlow`/`generateChatFlow` (cold `Flow`, source closed on completion/error/cancellation) and `completeSuspend`/`chatSuspend`/`chatCompleteTextSuspend`/ diff --git a/TODO.md b/TODO.md index 3b116aa38..35a4cd8c2 100644 --- a/TODO.md +++ b/TODO.md @@ -616,3 +616,22 @@ measurements and is not evidence that GPU testing is currently blocked. correct that sentence in the same commit; `jspecify` is the one of the four that is genuinely compile scope and stays. A cheap guard would be to assert the published AAR poms' scopes against the core pom's, the way the versions are already derived from it. + **Fixed on `fix/s390x-glibc-floor` (2026-10-26)**, CLAUDE.md "Android AAR + Kotlin façade": + - [x] Dropped, not mirrored as `provided` (a Gradle consumer ignores `provided` in a pom, so it + would document nothing). The AAR build no longer keeps a list at all: `coreCompileDependencies` + reads the core's compile-scope, non-optional dependencies from `llama/pom.xml`, versions and + scopes -- jackson-databind, slf4j-api **and** jspecify are compile scope there (all three + stay; the sentence above undercounts). The Gradle-generated poms (Docker, `gradle:9.8.2`) list + exactly those three, plus the CPU AAR in the OpenCL one. + - [x] The guard: `.github/verify-aar-poms.py` (`buildcheck/aarpoms.py`, `test_aarpoms.py`) checks + the poms `publishToMavenLocal` wrote in `package-android-aar`. Run on the two 6.0.0 poms from + Central it reports checker-qual and nothing else. + - [x] The CLAUDE.md sentence corrected; the "provided" bullet there now also says what the TODO did + not: **the classes do reference checker-qual** -- the Nullness Checker writes its qualifiers + into 133 of the core's 146 class files (RUNTIME retention). So dropping the dependency needs + `-dontwarn org.checkerframework.**` in `consumer-proguard.txt`, or a minified app fails R8 on + "Missing class"; added, like lombok and animal-sniffer. + - [ ] **Read the first Publish run:** `package-android-aar`'s new step, the consumer fixture's R8 + `assembleRelease` (cpu and opencl) now that checker-qual is off its classpath, and the + emulator instrumentation test (Jackson reflecting over the annotated classes without + checker-qual -- ART is expected to skip the unresolvable annotation types, as the JVM does). diff --git a/docs/post-release-check.md b/docs/post-release-check.md index 079f9b039..2dc4c2134 100644 --- a/docs/post-release-check.md +++ b/docs/post-release-check.md @@ -342,7 +342,12 @@ behind it lands. git show origin/main:pom.xml | grep -m1 -A1 llama-parent git branch -r --contains ``` -- **Scope drift between the reactor poms and the Gradle-built AAR poms.** The AAR poms take their - dependency *versions* from the core pom, but their *scopes* are the Gradle build's own. Compare them - every release: a dependency the core deliberately keeps out of the consumer graph at `provided` must - not reappear at `compile` in an AAR pom. +- **Scope drift between the reactor poms and the Gradle-built AAR poms.** Up to 6.0.0 the AAR poms + took their dependency *versions* from the core pom, but their *list and scopes* were the Gradle + build's own, and 6.0.0 shipped checker-qual at `compile` that way. Since 6.1.0 the list is read + from the core pom as well and `package-android-aar` checks the result + (`.github/verify-aar-poms.py`), so this is now a confirmation, not a hunt: run the same check on the + poms resolved from Central — + `python3 .github/verify-aar-poms.py . llama-android-.pom llama-android-opencl-.pom` (at the + tag) — and a dependency the core keeps out of the consumer graph at `provided` must not appear in + either. diff --git a/llama-android/build.gradle.kts b/llama-android/build.gradle.kts index 690e95cfa..32333c4c7 100644 --- a/llama-android/build.gradle.kts +++ b/llama-android/build.gradle.kts @@ -115,6 +115,43 @@ fun coreProperty(name: String): String { ) } +// The core's compile-scope dependencies (no or compile, not optional) as +// (groupId, artifactId, version) -- exactly what a Maven consumer of net.ladenthin:llama gets, and +// therefore what the AAR pom mirrors. Scopes are read here as well as versions: the 6.0.0 AAR poms +// were written from a hand-kept list and shipped checker-qual at compile scope although the core +// keeps it provided (its classes are major 55, CLAUDE.md "Java 8 bytecode floor"); the versions were +// derived and stayed right, the scopes were not and drifted. Provided (lombok, checker-qual, +// animal-sniffer), runtime (the JVM-only slf4j-simple binding: Android consumers pick their own, +// e.g. slf4j-android, or run with the no-op one) and test dependencies are not mirrored. The +// published poms are checked against the core pom in CI (.github/verify-aar-poms.py). +fun coreCompileDependencies(): List> { + val coreDependencies = directChildElement(corePom.documentElement, "dependencies") + ?: error("No found in ../llama/pom.xml") + val result = mutableListOf>() + val children = coreDependencies.childNodes + for (i in 0 until children.length) { + val node = children.item(i) + if (node.nodeType != Node.ELEMENT_NODE || node.nodeName != "dependency") { + continue + } + val dependency = node as Element + val scope = directChildText(dependency, "scope") ?: "compile" + if (scope != "compile" || directChildText(dependency, "optional") == "true") { + continue + } + val groupId = directChildText(dependency, "groupId") ?: error("A core dependency has no ") + val artifactId = directChildText(dependency, "artifactId") ?: error("$groupId: no ") + val version = directChildText(dependency, "version") + ?: error("$groupId:$artifactId has no in ../llama/pom.xml -- the AAR pom needs an explicit one") + val property = Regex("""\$\{([^}]+)}""").matchEntire(version)?.groupValues?.get(1) + result += Triple(groupId, artifactId, if (property != null) coreProperty(property) else version) + } + if (result.isEmpty()) { + error("No compile-scope dependency found in ../llama/pom.xml -- the parsing is broken, not the pom") + } + return result +} + group = "net.ladenthin" version = reactorVersion @@ -289,23 +326,17 @@ fun org.gradle.api.publish.maven.MavenPom.commonMetadata( url.set("https://github.com/bernardladenthin/java-llama.cpp") } // Mirror the core's compile-scope dependencies (the AAR repackages those - // classes, so their imports must resolve on the consumer classpath). - // logback-classic is deliberately NOT mirrored: it is the core's JVM-only - // runtime SLF4J binding and does not run on Android — Android consumers - // pick their own binding (e.g. slf4j-android) or run with the no-op one. + // classes, so their imports must resolve on the consumer classpath). The list + // is read from llama/pom.xml, scopes included -- see coreCompileDependencies. withXml { val dependencies = asNode().appendNode("dependencies") - fun dependency(groupId: String, artifactId: String, versionProperty: String) { + for ((groupId, artifactId, version) in coreCompileDependencies()) { val node = dependencies.appendNode("dependency") node.appendNode("groupId", groupId) node.appendNode("artifactId", artifactId) - node.appendNode("version", coreProperty(versionProperty)) + node.appendNode("version", version) node.appendNode("scope", "compile") } - dependency("com.fasterxml.jackson.core", "jackson-databind", "jackson.version") - dependency("org.slf4j", "slf4j-api", "slf4j.version") - dependency("org.jspecify", "jspecify", "jspecify.version") - dependency("org.checkerframework", "checker-qual", "checker.version") if (dependsOnCpuAar) { // The OpenCL AAR is one backend module; the library it plugs into is the CPU AAR // of the same version (type aar, so AGP merges both jni/ trees into the APK). diff --git a/llama-android/consumer-proguard.txt b/llama-android/consumer-proguard.txt index f047395dc..30ee2e193 100644 --- a/llama-android/consumer-proguard.txt +++ b/llama-android/consumer-proguard.txt @@ -34,6 +34,12 @@ # Android entry points never touch them). # - lombok / animal-sniffer: CLASS-retention build-time annotations referenced # from the bytecode; they exist on no runtime classpath anywhere. +# - org.checkerframework: the Nullness Checker writes its qualifiers (@NonNull, +# @Initialized, @Pure, ...) into the class files it checks. They are RUNTIME +# retention, but checker-qual is provided scope in the core and not in the AAR +# pom (major 55 classes), so no app has it; a runtime that reflects over an +# annotated element skips an annotation type it cannot resolve. -dontwarn com.sun.net.httpserver.** -dontwarn lombok.** -dontwarn org.codehaus.mojo.animal_sniffer.** +-dontwarn org.checkerframework.** From c0bb419447e88395ac829c9b5b53f2562360e149 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 10:10:58 +0100 Subject: [PATCH 06/13] s390x sysroot: make the unpacked tree writable before rewriting its links The first Publish run of the branch (40281901033) failed in the sysroot step: ln: failed to create symbolic link '.../usr/lib64/libnssckbi.so': Permission denied RHEL ships /usr/lib64 read-only (dr-xr-xr-x). The runner unpacks as a non-root user, so the absolute-to-relative symlink rewrite could not write into it. The local test ran as root, which ignores the mode. chmod -R u+w after unpacking fixes it, and the job's cleanup can delete the tree again. Reproduced and verified as uid 1000 (the docker calls replaced by a shim feeding the same image export): without the chmod, the same error; with it, "s390x sysroot ready (289M, glibc libc-2.28.so)", no absolute link left, and rm -rf works. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AJGh28Gu8pLWYbTCqG1LZf --- .github/manylinux/manylinux_2_28-s390x-sysroot | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/manylinux/manylinux_2_28-s390x-sysroot b/.github/manylinux/manylinux_2_28-s390x-sysroot index fb96ecb12..084974f92 100755 --- a/.github/manylinux/manylinux_2_28-s390x-sysroot +++ b/.github/manylinux/manylinux_2_28-s390x-sysroot @@ -51,6 +51,10 @@ trap 'docker rm "$cid" >/dev/null' EXIT docker export "$cid" | tar -x --no-same-owner -C "$SYSROOT" \ lib lib64 usr/lib/ld64.so.1 usr/lib64 usr/include \ opt/rh/gcc-toolset-14/root/usr/include "$GCC_DIR" +# RHEL ships /usr/lib64 read-only (dr-xr-xr-x). Unpacked by a non-root user (the runner), the +# symlink rewrite below could not write into it ("Permission denied" in the first CI run; a root +# shell ignores the mode) and the job's cleanup could not delete it. +chmod -R u+w "$SYSROOT" # Absolute symlinks -> relative ones inside the sysroot. A target outside the unpacked set stays # dangling, as it would be at an absolute path; only those the build reads matter, checked below. From 8c469aa161c0f5526c73d54368833979148b2b07 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 10:15:22 +0100 Subject: [PATCH 07/13] Automate the post-release check: post-release.yml after every release docs/post-release-check.md described how to audit a published release from the outside, and every check was run by hand. Now every check a GitHub runner can answer runs by itself after the Publish run of a v* tag, or by hand with workflow_dispatch and a tag. audit job: - bernardladenthin/actions' new java/audit-publication (checks 1, 2, 4-7, 10, 16): the run and its jobs, the files against the contract, Maven Central (polled while it publishes), the signatures with the release key from a keyserver, every file's provenance, the bytecode floors and the next version. Settings are in the new post-release section of actions-config.json. The pin is the actions branch commit until that action is released. - .github/post-release-check.py (buildcheck/postrelease.py, 11 tests), checks 3, 8, 9, 13 and 14: - the natives jars against natives.csv; - provenance, with patches/patches.sha256 recomputed from the patch blobs at the tag; - the source archive's stamp, its reproducible pack, and an added line of every patch in the tree (held to the last patch touching a file); - the AAR invariants and poms, 16 KB alignment; - no job log of the tag build with the compiler cache on. consumer matrix (checks 11, 12, 15), on Java 8 for Linux x86-64/aarch64, Windows x86-64 and macOS arm64: - post-release-consumer.sh resolves llama-platform (plus a GPU classifier) from Central into an empty local repository. The graph must not contain logback or checker-qual, and slf4j-simple must be at runtime scope. - The existing server smoke then runs on the resolved jars. - On Linux x86-64, post-release-webui.sh checks that GET / of the released library serves the gzip app shell and that system_fingerprint names the build. Tools and settings come from the default branch, the release's facts from a checkout of the tag, so 6.0.0 can be audited too. Run against 6.0.0 locally: - The 8 audit checks pass (next warns: main was at 6.0.0). - natives 23/23, provenance, source and cache (73 logs) pass. - android fails on exactly the known checker-qual finding. - Consumer resolution, server smoke and WebUI pass on the jars from Central (Docker, JDK 21). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AJGh28Gu8pLWYbTCqG1LZf --- .github/actions-config.json | 22 + .github/buildcheck/postrelease.py | 411 +++++++++++++++++++ .github/buildcheck/tests/test_postrelease.py | 288 +++++++++++++ .github/post-release-check.py | 17 + .github/post-release-consumer.sh | 98 +++++ .github/post-release-webui.sh | 68 +++ .github/verify-aar-poms.py | 0 .github/workflows/post-release.yml | 151 +++++++ CHANGELOG.md | 11 + CLAUDE.md | 10 + TODO.md | 15 + docs/RELEASE.md | 2 + docs/post-release-check.md | 21 + 13 files changed, 1114 insertions(+) create mode 100644 .github/buildcheck/postrelease.py create mode 100644 .github/buildcheck/tests/test_postrelease.py create mode 100755 .github/post-release-check.py create mode 100755 .github/post-release-consumer.sh create mode 100755 .github/post-release-webui.sh mode change 100644 => 100755 .github/verify-aar-poms.py create mode 100644 .github/workflows/post-release.yml diff --git a/.github/actions-config.json b/.github/actions-config.json index 2a56c928f..2fdad1a2a 100644 --- a/.github/actions-config.json +++ b/.github/actions-config.json @@ -29,5 +29,27 @@ "ignore-groups": [ "net.ladenthin" ] + }, + "post-release": { + "$comment": "bernardladenthin/actions java/audit-publication (.github/workflows/post-release.yml): the release key's primary fingerprint, and the class-file floor of each artifact (CLAUDE.md, \"Java 8 bytecode floor\"; first match wins).", + "signing-key": "EA6E1AC27DB005538E971E85ED0D9440BF148ED2", + "bytecode": [ + { + "files": "llama-ffm-*.jar", + "major": 66 + }, + { + "files": "llama-langchain4j-*.jar", + "major": 61 + }, + { + "files": "llama-atmosphere-agent-*.jar", + "major": 65 + }, + { + "files": "*", + "major": 52 + } + ] } } diff --git a/.github/buildcheck/postrelease.py b/.github/buildcheck/postrelease.py new file mode 100644 index 000000000..dcbaf1288 --- /dev/null +++ b/.github/buildcheck/postrelease.py @@ -0,0 +1,411 @@ +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +"""The post-release checks of this repository's own artifacts, on the published release files. + +docs/post-release-check.md lists what to verify once a release exists. The checks every repository +of bernardladenthin/actions shares (the run and its jobs, the files against the contract, Maven +Central, the signatures, the provenance attestation, the bytecode floors, the next version) are its +`java/audit-publication` action; this module holds the ones only this repository has. It reads the +files that action downloaded and the checkout of the tag, and takes the artifacts' word for nothing +it can recompute: + + natives (check 8) every natives jar is the one natives.csv declares: one directory, its library, + jllama-build.txt with every provenance key and the row's classifier and kind, + jllama-sbom.json, the Automatic-Module-Name; a module jar holds its module, + the two lists and the BOM only + provenance (check 9) one llama.cpp tag and commit across all jars, the release version, the tag's + commit without -dirty, and patches / patches.sha256 RECOMPUTED from the + patches at the tag -- the one check that ties the binaries to the reviewed + patch set rather than to themselves + source (check 13) the source archive's stamp names the jars' llama.cpp commit and the tag's + patches line for line, an added line of every patch is in the unpacked tree, + and the pack is reproducible (one mtime, owner 0/0, no owner names) + android (check 14) the AAR invariants: minSdk, the CPU AAR's classes.jar is the classes jar minus + module-info and its jni/ libraries are the Android natives jars' bytes, the + OpenCL AAR holds its one module and an empty classes.jar, both carry consumer + rules, both poms mirror the core (aarpoms.py) and the OpenCL one names the + CPU AAR of this version, and every Android ELF is 16 KB aligned + cache (check 3) the tag build ran without the shared compiler cache: no job log shows + USE_CACHE true or a build script reporting sccache ON, and some show it off + +Usage (the CLI is .github/post-release-check.py): post-release-check.py --tag v +--assets [--run ] [--checks natives,provenance,...]. Standard library only. +""" + +import argparse +import hashlib +import io +import os +import re +import subprocess +import sys +import tarfile +import zipfile + +from . import aarpoms, natives, nativedeps + +CHECKS = ("natives", "provenance", "source", "android", "cache") +BUILD_KEYS = ("llama.cpp", "llama.cpp.commit", "backend", "kind", "classifier", "jllama.version", + "jllama.commit", "patches", "patches.sha256", "compiler", "toolkit", "signed") +MODULE_FILES = {"jllama-build.txt", "jllama-files.txt", "jllama-sbom.json"} +ANDROID_ABIS = {"arm64-v8a": "cpu-android-aarch64", "x86_64": "cpu-android-x86-64"} +MIN_SDK = "28" + + +class Result: + def __init__(self, name): + self.name, self.problems, self.notes = name, [], [] + + def need(self, condition, message): + if not condition: + self.problems.append(message) + return condition + + +def read_rows(root): + with open(os.path.join(root, ".github", "natives.csv"), encoding="utf-8") as stream: + return natives.rows(stream.read()) + + +def _files(archive): + return [n for n in archive.namelist() if not n.endswith("/")] + + +def build_info(text): + """The key=value lines of a jllama-build.txt (comments skipped).""" + out = {} + for line in text.replace("\r", "").splitlines(): + if line and not line.startswith("#") and "=" in line: + key, value = line.split("=", 1) + out[key] = value + return out + + +def manifest_attribute(text, name): + """One main-section attribute of a MANIFEST.MF, its 72-byte continuation lines unfolded.""" + unfolded = re.sub(r"\r?\n ", "", text) + for line in unfolded.splitlines(): + if line.startswith(name + ":"): + return line[len(name) + 1:].strip() + return None + + +# --- natives ------------------------------------------------------------------------------------ + +def check_natives(assets, rows, version): + """(result, builds): builds maps each classifier to its jllama-build.txt keys.""" + result = Result("natives") + builds = {} + for row in rows: + classifier, directory, library, kind = row["classifier"], row["directory"], row["library"], row["kind"] + path = os.path.join(assets, f"llama-{version}-{classifier}.jar") + if not result.need(os.path.isfile(path), f"{classifier}: llama-{version}-{classifier}.jar is not in the release"): + continue + with zipfile.ZipFile(path) as jar: + prefix = f"net/ladenthin/llama/{directory}/" + files = _files(jar) + stray = [f for f in files if not f.startswith(prefix) and not f.startswith("META-INF/")] + result.need(not stray, f"{classifier}: files outside {prefix}: {stray[:3]}") + inside = {f[len(prefix):] for f in files if f.startswith(prefix)} + result.need(library in inside, f"{classifier}: {prefix}{library} is missing") + result.need("jllama-sbom.json" in inside, f"{classifier}: no jllama-sbom.json") + if kind == "module": + result.need(inside == MODULE_FILES | {library}, + f"{classifier}: a module jar holds {library} and the lists only, found {sorted(inside)}") + if not result.need("jllama-build.txt" in inside, f"{classifier}: no jllama-build.txt"): + continue + build = build_info(jar.read(prefix + "jllama-build.txt").decode("utf-8")) + builds[classifier] = build + missing = [k for k in BUILD_KEYS if not build.get(k)] + result.need(not missing, f"{classifier}: jllama-build.txt lacks {missing}") + for key, expected in (("classifier", classifier), ("kind", kind), ("jllama.version", version)): + result.need(build.get(key) == expected, f"{classifier}: jllama-build.txt says {key}={build.get(key)}, " + f"expected {expected}") + name = manifest_attribute(jar.read("META-INF/MANIFEST.MF").decode("utf-8"), "Automatic-Module-Name") + result.need(name == natives.module_name(classifier), + f"{classifier}: Automatic-Module-Name {name}, expected {natives.module_name(classifier)}") + result.notes.append(f"{classifier}: {len(inside)} files in {directory}") + result.notes.insert(0, f"{len(builds)} of {len(rows)} natives jars read") + return result, builds + + +# --- provenance --------------------------------------------------------------------------------- + +def git(root, *args): + return subprocess.run(["git", "-C", root, *args], capture_output=True, check=True).stdout + + +def patch_set(root, revision="HEAD"): + """[(name, sha256)] of llama/patches/*.patch|*.diff at revision, in name order -- the blobs, not + the working tree, so a checkout's line-end conversion cannot change a hash.""" + names = git(root, "ls-tree", "--name-only", f"{revision}:llama/patches").decode().split() + names = sorted(n for n in names if n.endswith((".patch", ".diff"))) + return [(n, hashlib.sha256(git(root, "show", f"{revision}:llama/patches/{n}")).hexdigest()) for n in names] + + +def patches_sha256(patches): + """jllama-build-info.cmake's hash: SHA-256 over " \\n" per patch.""" + return hashlib.sha256("".join(f"{n} {h}\n" for n, h in patches).encode()).hexdigest() + + +def patch_ids(patches): + return ",".join(re.sub(r"^([0-9]+)-.*", r"\1", n) for n, _ in patches) + + +def check_provenance(builds, version, commit, patches): + result = Result("provenance") + if not result.need(bool(builds), "no jllama-build.txt was read (see natives)"): + return result + for key in ("llama.cpp", "llama.cpp.commit"): + values = {b.get(key) for b in builds.values()} + result.need(len(values) == 1, f"{key} differs between the natives jars: {sorted(map(str, values))}") + expected_hash, expected_ids = patches_sha256(patches), patch_ids(patches) + for classifier, build in sorted(builds.items()): + result.need(build.get("jllama.version") == version, f"{classifier}: jllama.version {build.get('jllama.version')}") + result.need(build.get("jllama.commit") == commit, + f"{classifier}: jllama.commit {build.get('jllama.commit')}, the tag is {commit}") + result.need(build.get("patches") == expected_ids, f"{classifier}: patches={build.get('patches')}, " + f"the tag has {expected_ids}") + result.need(build.get("patches.sha256") == expected_hash, + f"{classifier}: patches.sha256 {build.get('patches.sha256')}, recomputed from the tag {expected_hash}") + first = next(iter(builds.values())) + result.notes.append(f"llama.cpp {first.get('llama.cpp')} ({first.get('llama.cpp.commit')}), {len(patches)} patches, " + f"patches.sha256 {expected_hash[:16]}… recomputed from the tag") + toolkits = sorted({f"{c}: {b.get('toolkit')}" for c, b in builds.items() if b.get("kind") == "module"}) + if toolkits: + result.notes.append("toolkits: " + "; ".join(toolkits)) + signed = sorted({b.get("signed") for b in builds.values()}) + result.notes.append(f"signed={','.join(map(str, signed))} (docs/post-release-check.md, check 16)") + return result + + +# --- the source archive ------------------------------------------------------------------------- + +def added_lines(patch_text): + """{path: [added lines]} of a unified diff, new files and modified files alike.""" + out, current = {}, None + for line in patch_text.splitlines(): + if line.startswith("+++ "): + target = line[4:].split("\t")[0] + current = None if target == "/dev/null" else re.sub(r"^b/", "", target) + if current: + out.setdefault(current, []) + elif current and line.startswith("+") and not line.startswith("+++"): + out[current].append(line[1:]) + return out + + +def check_source(assets, version, builds, patches, patch_text): + """patch_text(name) -> the patch at the tag, as text.""" + result = Result("source") + archives = [n for n in os.listdir(assets) if re.fullmatch(rf"llama\.cpp-.+-jllama-{re.escape(version)}-src\.tar\.gz", n)] + if not result.need(len(archives) == 1, f"expected one source archive, found {archives or 'none'}"): + return result + with tarfile.open(os.path.join(assets, archives[0])) as tar: + members = tar.getmembers() + top = archives[0][:-len(".tar.gz")] + result.need(all(m.name == top or m.name.startswith(top + "/") for m in members), + f"not everything is below {top}/") + mtimes = {m.mtime for m in members} + result.need(len(mtimes) == 1, f"{len(mtimes)} different mtimes (a reproducible pack has one)") + owners = {(m.uid, m.gid, m.uname, m.gname) for m in members} + result.need(owners == {(0, 0, "", "")}, f"owners {sorted(owners)[:3]}, expected numeric 0/0 without names") + files = {m.name[len(top) + 1:]: m for m in members if m.isfile()} + if not result.need(".jllama-patches-applied" in files, "no .jllama-patches-applied stamp"): + return result + stamp = tar.extractfile(files[".jllama-patches-applied"]).read().decode().splitlines() + head = next((line[5:] for line in stamp if line.startswith("head ")), None) + commits = {b.get("llama.cpp.commit") for b in builds.values()} + result.need(commits == {head}, f"the stamp's head {head}, the natives jars say {sorted(map(str, commits))}") + stamped = [tuple(line.split(" ", 1)) for line in stamp if re.match(r"\S+\.(patch|diff) [0-9a-f]{64}$", line)] + result.need(stamped == patches, "the stamp's patches differ from the tag's: " + f"only stamped {sorted(set(stamped) - set(patches))[:3]}, only at the tag {sorted(set(patches) - set(stamped))[:3]}") + # A later patch may rewrite what an earlier one added (0006/0007 rewrite 0001's lines in + # server.cpp), so a file is held to the LAST patch touching it; a patch whose every file is + # touched again later needs one of its added lines in one of its files. + added = [(name, added_lines(patch_text(name))) for name, _ in patches] + last = {path: name for name, paths in added for path in paths} + texts = {} + + def text(path): + if path not in texts: + member = files.get(path) + texts[path] = tar.extractfile(member).read().decode("utf-8", "replace") if member else None + return texts[path] + + for name, paths in added: + wanted = {p: [line.strip() for line in lines if len(line.strip()) >= 12] for p, lines in paths.items()} + wanted = {p: lines for p, lines in wanted.items() if lines} + if not wanted: + continue + for path in wanted: + result.need(text(path) is not None, f"{name}: {path} is not in the archive") + own = [p for p in wanted if last[p] == name and text(p) is not None] + for path in own: + result.need(any(line in text(path) for line in wanted[path]), f"{name}: none of its added lines is in {path}") + if not own: + result.need(any(text(p) and any(line in text(p) for line in lines) for p, lines in wanted.items()), + f"{name}: none of its added lines survives in {sorted(wanted)}") + result.notes.append(f"{archives[0]}: {len(members)} entries, head {head}, {len(stamped)} patches stamped") + return result + + +# --- Android ------------------------------------------------------------------------------------ + +def zip_entries(data): + with zipfile.ZipFile(io.BytesIO(data)) as archive: + return {i.filename: i.CRC for i in archive.infolist() if not i.filename.endswith("/")} + + +def check_android(assets, version, root): + result = Result("android") + cpu_path = os.path.join(assets, f"llama-android-{version}.aar") + opencl_path = os.path.join(assets, f"llama-android-opencl-{version}.aar") + core_path = os.path.join(assets, f"llama-{version}.jar") + for path in (cpu_path, opencl_path, core_path): + if not result.need(os.path.isfile(path), f"{os.path.basename(path)} is not in the release"): + return result + elves = [] + with zipfile.ZipFile(cpu_path) as cpu, zipfile.ZipFile(opencl_path) as opencl: + for name, aar in (("llama-android", cpu), ("llama-android-opencl", opencl)): + manifest = aar.read("AndroidManifest.xml").decode("utf-8") + result.need(f'android:minSdkVersion="{MIN_SDK}"' in manifest, f"{name}: minSdkVersion is not {MIN_SDK}") + result.need("proguard.txt" in aar.namelist() and aar.read("proguard.txt").strip(), + f"{name}: no consumer rules (proguard.txt)") + elves += [(f"{name}.aar!{n}", aar.read(n)) for n in _files(aar) if n.endswith(".so")] + with open(core_path, "rb") as stream: + core = {n: c for n, c in zip_entries(stream.read()).items() if not n.endswith("module-info.class")} + classes = zip_entries(cpu.read("classes.jar")) + core_classes = {n: c for n, c in core.items() if n.endswith(".class")} + aar_classes = {n: c for n, c in classes.items() if n.endswith(".class")} + result.need(aar_classes == core_classes, "llama-android: classes.jar is not the classes jar minus module-info " + f"({len(aar_classes)} classes, the classes jar has {len(core_classes)}; " + f"differing {sorted(set(aar_classes.items()) ^ set(core_classes.items()))[:2]})") + rows = read_rows(root) + for abi, classifier in ANDROID_ABIS.items(): + row = next(r for r in rows if r["classifier"] == classifier) + with zipfile.ZipFile(os.path.join(assets, f"llama-{version}-{classifier}.jar")) as jar: + prefix = f"net/ladenthin/llama/{row['directory']}/" + expected = {n[len(prefix):]: jar.getinfo(n).CRC for n in _files(jar) + if n.startswith(prefix) and n.endswith(".so")} + elves += [(f"{classifier}!{n}", jar.read(prefix + n)) for n in expected] + actual = {n[len(f"jni/{abi}/"):]: cpu.getinfo(n).CRC for n in _files(cpu) if n.startswith(f"jni/{abi}/")} + result.need(actual == expected, f"llama-android jni/{abi}/ differs from {classifier}: " + f"{sorted(set(actual.items()) ^ set(expected.items()))[:3]}") + jni = sorted(n for n in _files(opencl) if n.startswith("jni/")) + result.need(jni == ["jni/arm64-v8a/libggml-opencl.so"], f"llama-android-opencl: jni/ holds {jni}") + result.need(not any(n.endswith(".class") for n in zip_entries(opencl.read("classes.jar"))), + "llama-android-opencl: classes.jar is not empty (duplicating the API breaks D8)") + for name, data in elves: + bad = [a for a in nativedeps.elf_load_alignments(data) if a % nativedeps.ANDROID_PAGE_ALIGNMENT] + result.need(not bad, f"{name}: LOAD alignment {bad} (Google Play needs 16 KB)") + expected = aarpoms.core_compile_dependencies(os.path.join(root, "llama", "pom.xml"), os.path.join(root, "pom.xml")) + for artifact in ("llama-android", "llama-android-opencl"): + pom = os.path.join(assets, f"{artifact}-{version}.pom") + if not result.need(os.path.isfile(pom), f"{artifact}-{version}.pom is not in the release"): + continue + published = aarpoms.published_dependencies(pom) + result.problems += [f"{artifact} pom: {p}" for p in aarpoms.violations(expected, published)] + if artifact == "llama-android-opencl": + cpu_dep = [d for d in published if (d[0], d[1]) == aarpoms.CPU_AAR] + result.need([d[2] for d in cpu_dep] == [version], + f"llama-android-opencl pom: the CPU AAR dependency is {cpu_dep}, expected version {version}") + result.notes.append(f"{len(elves)} Android ELF files 16 KB aligned; {len(core_classes)} classes in the CPU AAR") + return result + + +# --- the cache-free tag build ------------------------------------------------------------------- + +CACHE_ON = re.compile(r"USE_CACHE: true|build\.sh: sccache ON|build\.bat: sccache probe OK") +CACHE_OFF = re.compile(r"USE_CACHE: false|build\.sh: sccache OFF") + + +def check_cache(logs): + """logs: {job name: log text} of the publishing run.""" + result = Result("cache") + off = 0 + for name, text in sorted(logs.items()): + hits = sorted({m.group(0) for m in CACHE_ON.finditer(text)}) + result.need(not hits, f"{name}: {', '.join(hits)}") + off += bool(CACHE_OFF.search(text)) + result.need(off > 0, "no job log shows the cache off -- the evidence is missing, not the cache") + result.notes.append(f"{len(logs)} job logs read, {off} show the cache off, none on") + return result + + +def run_logs(repository, run_id): + import json + jobs, page = [], 1 + while True: + out = subprocess.run(["gh", "api", f"repos/{repository}/actions/runs/{run_id}/jobs?filter=latest&per_page=50&page={page}"], + capture_output=True, text=True, check=True).stdout + batch = json.loads(out)["jobs"] + jobs += batch + if len(batch) < 50: + break + page += 1 + logs = {} + for job in jobs: + if job["conclusion"] == "skipped": + continue + out = subprocess.run(["gh", "api", f"repos/{repository}/actions/jobs/{job['id']}/logs"], + capture_output=True, check=True).stdout + logs[job["name"]] = out.decode("utf-8", "replace") + return logs + + +# --- the command -------------------------------------------------------------------------------- + +def find_run(repository, tag): + import json + out = subprocess.run(["gh", "api", f"repos/{repository}/actions/workflows/publish.yml/runs?branch={tag}&per_page=50"], + capture_output=True, text=True, check=True).stdout + good = [r["id"] for r in json.loads(out)["workflow_runs"] if r["conclusion"] == "success"] + if len(good) != 1: + raise ValueError(f"expected one successful Publish run on {tag}, found {good}") + return good[0] + + +def main(argv): + parser = argparse.ArgumentParser(prog="post-release-check.py", description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--root", default=".", help="a checkout of the tag (default: the current directory)") + parser.add_argument("--tag", required=True) + parser.add_argument("--assets", required=True, help="the GitHub release files, downloaded") + parser.add_argument("--run", help="the publishing run (default: the one successful Publish run on the tag)") + parser.add_argument("--checks", default=",".join(CHECKS)) + args = parser.parse_args(argv[1:]) + checks = [c for c in args.checks.split(",") if c] + if set(checks) - set(CHECKS): + parser.error(f"unknown check(s) {sorted(set(checks) - set(CHECKS))}; known: {', '.join(CHECKS)}") + root = os.path.abspath(args.root) + rows = read_rows(root) + version = args.tag[1:] + commit = git(root, "rev-parse", "HEAD").decode().strip() + patches = patch_set(root) + results = [] + natives_result, builds = check_natives(args.assets, rows, version) + if "natives" in checks: + results.append(natives_result) + if "provenance" in checks: + results.append(check_provenance(builds, version, commit, patches)) + if "source" in checks: + results.append(check_source(args.assets, version, builds, patches, + lambda n: git(root, "show", f"HEAD:llama/patches/{n}").decode("utf-8", "replace"))) + if "android" in checks: + results.append(check_android(args.assets, version, root)) + if "cache" in checks: + repository = os.environ.get("GITHUB_REPOSITORY", "bernardladenthin/java-llama.cpp") + results.append(check_cache(run_logs(repository, args.run or find_run(repository, args.tag)))) + failed = 0 + for result in results: + print(f"== {result.name}: {'FAIL' if result.problems else 'ok'}") + for note in result.notes: + print(f" {note}") + for problem in result.problems: + print(f"::error::{result.name}: {problem}") + failed += bool(result.problems) + print(f"post-release checks of {args.tag}: {len(results)} checks, {failed} failed") + return 1 if failed else 0 diff --git a/.github/buildcheck/tests/test_postrelease.py b/.github/buildcheck/tests/test_postrelease.py new file mode 100644 index 000000000..e792ffa8b --- /dev/null +++ b/.github/buildcheck/tests/test_postrelease.py @@ -0,0 +1,288 @@ +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +import hashlib +import io +import os +import struct +import subprocess +import tarfile +import tempfile +import unittest +import zipfile + +from buildcheck import natives, postrelease + +CSV = """# the natives jars +classifier,directory,library,platform,kind,optin +cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library,no +cpu-android-aarch64,Linux-Android/aarch64/cpu,libjllama.so,no,library,no +cpu-android-x86-64,Linux-Android/x86_64/cpu,libjllama.so,no,library,no +vulkan-linux-x86-64,Linux/x86_64/vulkan,libggml-vulkan.so,no,module,no +""" +PATCHES = [("0001-a.patch", "1" * 64), ("0012-b.patch", "2" * 64)] +COMMIT = "c" * 40 +UPSTREAM = "u" * 40 + + +def elf(align): + """A minimal little-endian ELF64 with one PT_LOAD segment of the given alignment.""" + header = b"\x7fELF\x02\x01" + b"\x00" * 26 + struct.pack(" \n" per patch, in name order + manifest = "0001-a.patch " + "1" * 64 + "\n0012-b.patch " + "2" * 64 + "\n" + self.assertEqual(postrelease.patches_sha256(PATCHES), hashlib.sha256(manifest.encode()).hexdigest()) + self.assertEqual(postrelease.patch_ids(PATCHES), "0001,0012") + + def test_the_patch_set_is_read_from_the_blobs_at_the_revision(self): + with tempfile.TemporaryDirectory() as repo: + def git(*args): + subprocess.run(["git", "-C", repo, *args], check=True, capture_output=True) + git("init", "-q") + os.makedirs(os.path.join(repo, "llama", "patches")) + for name, body in (("0002-b.patch", b"b\n"), ("0001-a.patch", b"a\n"), ("README.md", b"x")): + with open(os.path.join(repo, "llama", "patches", name), "wb") as f: + f.write(body) + git("add", ".") + git("-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "-m", "x") + with open(os.path.join(repo, "llama", "patches", "0001-a.patch"), "wb") as f: + f.write(b"changed in the working tree\n") + self.assertEqual(postrelease.patch_set(repo), [("0001-a.patch", hashlib.sha256(b"a\n").hexdigest()), + ("0002-b.patch", hashlib.sha256(b"b\n").hexdigest())]) + + +PATCH_A = """--- a/src/x.cpp ++++ b/src/x.cpp +@@ -1,1 +1,2 @@ + int x; ++int jllama_patch_a_marker = 1; +""" +PATCH_B = """--- a/src/x.cpp ++++ b/src/x.cpp +@@ -1,2 +1,2 @@ +-int jllama_patch_a_marker = 1; ++int jllama_patch_b_rewrote_it = 2; +--- /dev/null ++++ b/src/new.h +@@ -0,0 +1 @@ ++#define JLLAMA_NEW_HEADER 1 +""" + + +class SourceTest(Fixture): + def archive(self, stamp, files, owner=(0, 0, "", ""), mtimes=(100,)): + top = "llama.cpp-b100-jllama-1.2.0-src" + buffer = io.BytesIO() + with tarfile.open(fileobj=buffer, mode="w:gz") as tar: + entries = dict(files, **{".jllama-patches-applied": stamp}) + for i, (name, text) in enumerate(entries.items()): + info = tarfile.TarInfo(f"{top}/{name}") + data = text.encode() + info.size, info.mtime = len(data), mtimes[i % len(mtimes)] + info.uid, info.gid, info.uname, info.gname = owner + tar.addfile(info, io.BytesIO(data)) + self.write(top + ".tar.gz", buffer.getvalue()) + + def check(self, patches=PATCHES): + builds = {"cpu-linux-x86-64": {"llama.cpp.commit": UPSTREAM}} + texts = {"0001-a.patch": PATCH_A, "0012-b.patch": PATCH_B} + return postrelease.check_source(self.assets, "1.2.0", builds, patches, texts.get) + + def stamp(self, head=UPSTREAM, patches=PATCHES): + return f"head {head}\n" + "".join(f"{n} {h}\n" for n, h in patches) + "tree abc\n" + + def test_a_reproducible_archive_with_every_patch_in_it_passes(self): + # 0012 rewrote 0001's line: x.cpp is held to 0012, 0001 to "one added line survives" -- which fails + # here, as it must when a later patch removed everything an earlier one added + self.archive(self.stamp(), {"src/x.cpp": "int x;\nint jllama_patch_b_rewrote_it = 2;\n", + "src/new.h": "#define JLLAMA_NEW_HEADER 1\n"}) + self.assertEqual(self.check().problems, + ["0001-a.patch: none of its added lines survives in ['src/x.cpp']"]) + self.archive(self.stamp(), {"src/x.cpp": "int x;\nint jllama_patch_a_marker = 1;\nint jllama_patch_b_rewrote_it = 2;\n", + "src/new.h": "#define JLLAMA_NEW_HEADER 1\n"}) + self.assertEqual(self.check().problems, []) + + def test_stamp_head_stamp_patches_missing_patch_content_and_a_non_reproducible_pack(self): + self.archive(self.stamp(head="z" * 40, patches=PATCHES[:1]), {"src/x.cpp": "int x;\n"}, + owner=(1000, 1000, "runner", "docker"), mtimes=(100, 200)) + problems = self.check().problems + self.assertIn("2 different mtimes (a reproducible pack has one)", problems) + self.assertTrue(any(p.startswith("owners [(1000, 1000, 'runner', 'docker')]") for p in problems)) + self.assertIn(f"the stamp's head {'z' * 40}, the natives jars say ['{UPSTREAM}']", problems) + self.assertTrue(any(p.startswith("the stamp's patches differ from the tag's") for p in problems)) + self.assertIn("0012-b.patch: src/new.h is not in the archive", problems) + self.assertIn("0012-b.patch: none of its added lines is in src/x.cpp", problems) + + +class AndroidTest(Fixture): + def setUp(self): + super().setUp() + with open(os.path.join(self.root, "pom.xml"), "w") as f: + f.write("1") + os.makedirs(os.path.join(self.root, "llama")) + with open(os.path.join(self.root, "llama", "pom.xml"), "w") as f: + f.write("gj" + "${j.version}") + + def release(self, cpu_jni_align=16384, opencl_classes=None, extra_dep="", opencl_cpu_version="1.2.0"): + self.release_natives() + core = {"net/ladenthin/llama/A.class": b"A", "module-info.class": b"M"} + self.write("llama-1.2.0.jar", zipped(core)) + manifest = '' + jni = {} + for abi, classifier in postrelease.ANDROID_ABIS.items(): + row = next(r for r in self.rows if r["classifier"] == classifier) + with zipfile.ZipFile(os.path.join(self.assets, f"llama-1.2.0-{classifier}.jar")) as jar: + prefix = f"net/ladenthin/llama/{row['directory']}/" + for n in jar.namelist(): + if n.startswith(prefix) and n.endswith(".so"): + jni[f"jni/{abi}/{n[len(prefix):]}"] = jar.read(n) + jni["jni/x86_64/libggml-base.so"] = elf(cpu_jni_align) + self.write("llama-android-1.2.0.aar", zipped({"AndroidManifest.xml": manifest, "proguard.txt": "-keep x", + "classes.jar": zipped({"net/ladenthin/llama/A.class": b"A"}), **jni})) + self.write("llama-android-opencl-1.2.0.aar", zipped({ + "AndroidManifest.xml": manifest, "proguard.txt": "-keep x", + "classes.jar": zipped(opencl_classes or {"META-INF/MANIFEST.MF": "x"}), + "jni/arm64-v8a/libggml-opencl.so": elf(16384)})) + dep = "gj1compile" + self.write("llama-android-1.2.0.pom", f"{dep}{extra_dep}") + self.write("llama-android-opencl-1.2.0.pom", f"{dep}{extra_dep}net.ladenthin" + f"llama-android{opencl_cpu_version}aar" + "compile") + + def test_a_correct_pair_passes(self): + self.release() + result = postrelease.check_android(self.assets, "1.2.0", self.root) + self.assertEqual(result.problems, []) + + def test_alignment_jni_bytes_opencl_classes_and_the_poms(self): + checker = ("org.checkerframeworkchecker-qual" + "4compile") + self.release(cpu_jni_align=4096, opencl_classes={"a/A.class": b"A"}, extra_dep=checker, opencl_cpu_version="1.1.0") + problems = postrelease.check_android(self.assets, "1.2.0", self.root).problems + self.assertIn("llama-android.aar!jni/x86_64/libggml-base.so: LOAD alignment [4096] (Google Play needs 16 KB)", problems) + self.assertTrue(any(p.startswith("llama-android jni/x86_64/ differs from cpu-android-x86-64") for p in problems)) + self.assertIn("llama-android-opencl: classes.jar is not empty (duplicating the API breaks D8)", problems) + self.assertIn("llama-android pom: org.checkerframework:checker-qual (scope compile) is not a compile-scope " + "dependency of the core", problems) + self.assertTrue(any("the CPU AAR dependency is" in p and "expected version 1.2.0" in p for p in problems)) + + +class CacheTest(unittest.TestCase): + def test_off_everywhere_passes_and_a_job_with_the_cache_on_fails(self): + logs = {"Build Linux": " USE_CACHE: false\nbuild.sh: sccache OFF, building with -j4\n", "Tests": "ok\n"} + self.assertEqual(postrelease.check_cache(logs).problems, []) + logs["Build Windows"] = " USE_CACHE: true\nbuild.bat: sccache probe OK (wrapped cl.exe).\n" + self.assertEqual(postrelease.check_cache(logs).problems, + ["Build Windows: USE_CACHE: true, build.bat: sccache probe OK"]) + self.assertEqual(postrelease.check_cache({"Tests": "ok"}).problems, + ["no job log shows the cache off -- the evidence is missing, not the cache"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/post-release-check.py b/.github/post-release-check.py new file mode 100755 index 000000000..72a18c4e6 --- /dev/null +++ b/.github/post-release-check.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 +"""This repository's own post-release checks of a published release. + +See buildcheck/postrelease.py. Usage: post-release-check.py --tag v --assets [--checks ...] +""" + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from buildcheck import postrelease # noqa: E402 + +if __name__ == "__main__": + sys.exit(postrelease.main(sys.argv)) diff --git a/.github/post-release-consumer.sh b/.github/post-release-consumer.sh new file mode 100755 index 000000000..4b1184292 --- /dev/null +++ b/.github/post-release-consumer.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +# Post-release check 11 (docs/post-release-check.md): a consumer resolves the published release from +# Maven Central. Nothing in the publish workflow ever resolves the release from Central -- the +# smoke sets are assembled from the build tree -- so the parent pom's property substitutions, the +# scopes a consumer actually gets and the presence of every platform jar are first tried here. +# +# A throwaway consumer pom depends on net.ladenthin:llama-platform (type pom) plus the given extra +# classifiers of net.ladenthin:llama, and is resolved against an EMPTY local repository into +# : the classes jar, its runtime dependencies and the natives jars, flat -- a smoke set, so +# .github/smoke-test-natives.sh and friends run on it unchanged. Asserted on the resolved graph: +# - the classes jar and the library jar of every `platform=yes` row of natives.csv (at the tag); +# - jackson-databind, slf4j-api and jspecify at compile scope, slf4j-simple at runtime; +# - neither logback nor checker-qual anywhere: both break a Java 8 consumer, and only the pom +# scopes keep them out, which no compile-time check sees (6.0.0's AAR poms got this wrong). +# Afterwards the natives jars of other platforms are removed from , so it is the smoke set +# of (-, e.g. linux-x86-64) that the smoke scripts expect: one library jar. +# +# Usage: post-release-consumer.sh [extra classifier...] +set -euo pipefail + +USAGE="usage: post-release-consumer.sh [classifier...]" +VERSION="${1:?$USAGE}" +NATIVES_CSV="${2:?$USAGE}" +SET_DIR="${3:?$USAGE}" +TARGET="${4:?$USAGE}" +shift 4 + +fail() { + echo "::error::$*" >&2 + exit 1 +} + +[ -f "$NATIVES_CSV" ] || fail "no natives list at $NATIVES_CSV" +[ ! -e "$SET_DIR" ] || [ -z "$(ls -A "$SET_DIR")" ] || fail "$SET_DIR is not empty" +WORK="$(mktemp -d)" +trap 'rm -rf "$WORK"' EXIT + +extra="" +for classifier in "$@"; do + extra="$extra + net.ladenthinllama$VERSION$classifier" +done +cat > "$WORK/pom.xml" << EOF + + 4.0.0 + local.post-release + consumer + 1 + + net.ladenthinllama-platform$VERSIONpom$extra + + +EOF + +mvn_consumer() { + mvn -B -ntp -f "$WORK/pom.xml" "-Dmaven.repo.local=$WORK/m2" "$@" +} +mvn_consumer -q dependency:list -DincludeScope=runtime -DoutputFile="$WORK/deps.txt" -DexcludeTransitive=false \ + || fail "net.ladenthin:llama-platform:$VERSION does not resolve from Maven Central" +mvn_consumer -q dependency:copy-dependencies -DincludeScope=runtime "-DoutputDirectory=$(cd "$(dirname "$SET_DIR")" && pwd)/$(basename "$SET_DIR")" +# dependency:list writes " group:artifact:type[:classifier]:version:scope" (plus a module suffix on newer plugins) +deps="$(sed -n 's/^ *\([^ ]*:[^ ]*\).*/\1/p' "$WORK/deps.txt" | grep ':' || true)" +echo "resolved from Central (runtime scope):" +sed 's/^/ /' <<< "$deps" + +has() { grep -qE "^$1:" <<< "$deps"; } +scope_of() { grep -E "^$1:" <<< "$deps" | head -1 | awk -F: '{print $NF}'; } + +for dep in com.fasterxml.jackson.core:jackson-databind org.slf4j:slf4j-api org.jspecify:jspecify; do + has "$dep" || fail "$dep is not in the consumer's graph" + [ "$(scope_of "$dep")" = "compile" ] || fail "$dep has scope $(scope_of "$dep"), expected compile" +done +has org.slf4j:slf4j-simple || fail "org.slf4j:slf4j-simple (the shipped binding) is not in the consumer's graph" +[ "$(scope_of org.slf4j:slf4j-simple)" = "runtime" ] || fail "slf4j-simple has scope $(scope_of org.slf4j:slf4j-simple), expected runtime" +if grep -qE '^(ch\.qos\.logback|org\.checkerframework):' <<< "$deps"; then + fail "logback or checker-qual reached the consumer: $(grep -E '^(ch\.qos\.logback|org\.checkerframework):' <<< "$deps" | tr '\n' ' ')" +fi +[ -f "$SET_DIR/llama-$VERSION.jar" ] || fail "the classes jar llama-$VERSION.jar was not resolved" +missing="" +while IFS=, read -r classifier _ _ platform _; do + [ "$platform" = "yes" ] || continue + [ -f "$SET_DIR/llama-$VERSION-$classifier.jar" ] || missing="$missing $classifier" +done < <(grep -v -e '^#' -e '^classifier,' -e '^$' "$NATIVES_CSV" | tr -d '\r') +[ -z "$missing" ] || fail "llama-platform did not bring the library jar of:$missing" +for classifier in "$@"; do + [ -f "$SET_DIR/llama-$VERSION-$classifier.jar" ] || fail "the classifier $classifier was not resolved" +done +echo "consumer resolution OK: $(find "$SET_DIR" -maxdepth 1 -name '*.jar' | wc -l | tr -d ' ') jars, no logback, no checker-qual" +while IFS=, read -r classifier _; do + case "$classifier" in *-"$TARGET") ;; *) rm -f "$SET_DIR/llama-$VERSION-$classifier.jar" ;; esac +done < <(grep -v -e '^#' -e '^classifier,' -e '^$' "$NATIVES_CSV" | tr -d '\r') +find "$SET_DIR" -maxdepth 1 -name "llama-$VERSION-*-$TARGET.jar" | grep -q . || fail "no natives jar of $TARGET in the set" +echo "smoke set of $TARGET: $(cd "$SET_DIR" && ls llama-*.jar | tr '\n' ' ')" diff --git a/.github/post-release-webui.sh b/.github/post-release-webui.sh new file mode 100755 index 000000000..d4d47df37 --- /dev/null +++ b/.github/post-release-webui.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +# Post-release check 15 (docs/post-release-check.md): the embedded WebUI of the RELEASED library. A +# local cmake build compiles the empty-asset stub, so "the WebUI is served" is a property of the +# published jar and of no build a developer makes: the stub answers GET / with 404, the release +# with the gzip-compressed app shell. Also /props, and a chat completion whose system_fingerprint +# names the build the natives jars declare (jllama-build.txt `llama.cpp`). +# +# Usage: post-release-webui.sh [port] (a smoke set, e.g. from post-release-consumer.sh) +set -euo pipefail + +SET_DIR="${1:?usage: post-release-webui.sh [port]}" +MODEL="${2:?usage: post-release-webui.sh [port]}" +PORT="${3:-18090}" +BASE="http://127.0.0.1:$PORT" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=smoke-lib.sh +. "$SCRIPT_DIR/smoke-lib.sh" + +fail() { + echo "::error::$*" >&2 + exit 1 +} + +CP="$(smoke_classpath "$SET_DIR")" || exit 1 +java -cp "$CP" "$SMOKE_MAIN" -m "$MODEL" --host 127.0.0.1 --port "$PORT" --chat-template chatml \ + > webui-out.log 2> webui-err.log & +PID=$! +trap 'kill "$PID" 2> /dev/null || true' EXIT +for _ in $(seq 1 100); do + kill -0 "$PID" 2> /dev/null || { cat webui-out.log webui-err.log; fail "the server exited before /health answered"; } + [ "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/health" || true)" = "200" ] && break + sleep 3 +done +[ "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/health")" = "200" ] || fail "/health never answered 200" + +curl -s -D webui-headers.txt -o webui-index.gz -H 'Accept-Encoding: gzip' "$BASE/" || fail "GET / failed" +status="$(head -1 webui-headers.txt | tr -d '\r')" +grep -q ' 200' <<< "$status" || fail "GET / answered '$status' -- the empty-asset stub answers 404, a release must serve the WebUI" +grep -qi '^content-type: text/html' webui-headers.txt || fail "GET / is not text/html: $(grep -i '^content-type' webui-headers.txt)" +grep -qi '^content-encoding: gzip' webui-headers.txt || fail "GET / is not gzip-encoded" +python3 - webui-index.gz << 'EOF' || fail "the WebUI body is not the app shell" +import gzip, sys +html = gzip.decompress(open(sys.argv[1], "rb").read()).decode("utf-8", "replace").lower() +assert " +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +name: Post-release check +on: + workflow_run: + workflows: [Publish] + types: [completed] + workflow_dispatch: + inputs: + tag: + description: "The release tag to audit (v), e.g. to re-audit an older release" + required: true + +# docs/post-release-check.md, automated: the published release, checked from the outside after the +# Publish run that made it -- against the GitHub release, Maven Central and a real JVM, not against +# the build tree. A green pipeline is evidence about the build; every failure class CLAUDE.md +# records as having reached a user reached them through one. +# +# Triggered by the end of a successful Publish run dispatched on a v* tag (a release created with +# the workflow token starts no `release:` workflow), or by hand with the tag. The tools and the +# configuration are this checkout (the default branch); the release's own facts (its contract, +# natives.csv, patches, poms) come from a second checkout of the tag in release/, so an older +# release is audited with today's checks. +# +# audit bernardladenthin/actions' java/audit-publication (checks 1, 2, 4-7, 10, 16: the run, +# its jobs, the files, Maven Central -- polled while it still publishes --, the +# signatures, the provenance, the bytecode floors, the next version), then this +# repository's own: natives jars, provenance recomputed, source archive, AARs, the +# cache-free build (checks 3, 8, 9, 13, 14; .github/post-release-check.py) +# consumer per platform, a consumer resolves llama-platform from Central into an empty local +# repository (check 11, .github/post-release-consumer.sh) and runs it on Java 8, the +# floor (check 12): the server smoke of the natives jars -- inference, TLS, an https:// +# model download, the natives directory -- and, on Linux x86-64, the embedded WebUI of +# the released library (check 15, .github/post-release-webui.sh) +# What stays by hand: GPU inference (no GPU runner) -- docs/post-release-check.md, check 12. +permissions: + contents: read + +concurrency: + group: post-release-${{ inputs.tag || github.event.workflow_run.head_branch }} + cancel-in-progress: false + +env: + TAG: ${{ inputs.tag || github.event.workflow_run.head_branch }} + +jobs: + audit: + name: Audit the published release + # Only a successful run dispatched on a v* tag published a release: pull request runs never + # publish, and their head branch is not ours to check out. + if: >- + github.event_name == 'workflow_dispatch' || + (github.event.workflow_run.conclusion == 'success' && + github.event.workflow_run.event == 'workflow_dispatch' && + startsWith(github.event.workflow_run.head_branch, 'v')) + runs-on: ubuntu-latest + timeout-minutes: 240 + permissions: + contents: read + actions: read + attestations: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false + - name: Check out the tag + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + ref: ${{ env.TAG }} + path: release + persist-credentials: false + - name: Audit the release (run, jobs, files, Maven Central, signatures, provenance, bytecode) + uses: bernardladenthin/actions/java/audit-publication@630e3a93f4222f87da5de16cc48761068596f458 # audit-publication branch, pending a release + with: + tag: ${{ env.TAG }} + root: release + assets: release-assets + wait-minutes: '180' + - name: This repository's own artifacts (natives jars, provenance, source archive, AARs, cache-free build) + if: ${{ !cancelled() }} + env: + GH_TOKEN: ${{ github.token }} + RUN_ID: ${{ github.event.workflow_run.id }} + run: python3 .github/post-release-check.py --root release --tag "$TAG" --assets release-assets ${RUN_ID:+--run "$RUN_ID"} + + consumer: + name: A consumer on Java 8 (${{ matrix.target }}) + needs: audit + # After the audit, which waited for Maven Central; a finding there does not hide this one. + if: ${{ !cancelled() && needs.audit.result != 'skipped' }} + strategy: + fail-fast: false + matrix: + include: + - { target: linux-x86-64, runner: ubuntu-latest, jdk: temurin, extra: vulkan-linux-x86-64 } + - { target: linux-aarch64, runner: ubuntu-24.04-arm, jdk: temurin, extra: vulkan-linux-aarch64 } + - { target: windows-x86-64, runner: windows-latest, jdk: temurin, extra: vulkan-windows-x86-64 } + # Temurin publishes no Java 8 for macOS on arm64; Zulu does. + - { target: macos-aarch64, runner: macos-latest, jdk: zulu, extra: '' } + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false + - name: Check out the tag's natives list + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + ref: ${{ env.TAG }} + path: release + sparse-checkout: .github/natives.csv + sparse-checkout-cone-mode: false + persist-credentials: false + - uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6 + with: + distribution: ${{ matrix.jdk }} + java-version: '8' + - name: Resolve the release from Maven Central into an empty local repository (check 11) + shell: bash + run: | + java -version + .github/post-release-consumer.sh "${TAG#v}" release/.github/natives.csv consumer "${{ matrix.target }}" ${{ matrix.extra }} + - name: Download the 1 MB test model + shell: bash + run: | + mkdir -p models + curl -sSfL --retry 3 -o models/stories260K.gguf "$(grep '^stories260K\.gguf,' .github/models.csv | cut -d, -f2-)" + - name: The server smoke on the resolved jars, Java 8 (check 12) + if: runner.os == 'Linux' + run: .github/smoke-test-natives.sh consumer models/stories260K.gguf + - name: The server smoke on the resolved jars, Java 8 (check 12) + if: runner.os == 'Windows' + shell: pwsh + run: .github/smoke-test-natives.ps1 -SetDir consumer -Model models/stories260K.gguf + - name: The macOS smoke on the resolved jars, Java 8 (check 12) + if: runner.os == 'macOS' + run: .github/smoke-native-macos.sh consumer + - name: The embedded WebUI of the released library (check 15) + if: matrix.target == 'linux-x86-64' + run: .github/post-release-webui.sh consumer models/stories260K.gguf + - name: Upload server logs + if: failure() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7 + with: + name: post-release-${{ matrix.target }}-logs + path: | + *.log + webui-headers.txt + if-no-files-found: warn diff --git a/CHANGELOG.md b/CHANGELOG.md index 002745137..859a3a249 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,17 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by An app that used checker-qual's annotations itself through the AAR has to declare it. ### Added +- **The post-release check runs by itself** (`.github/workflows/post-release.yml`, after the Publish + run of a `v*` tag, or by hand with a tag). It audits the published release from the outside: + - through bernardladenthin/actions' new `java/audit-publication`: the run and its jobs, the files + against the contract, Maven Central, the signatures, every file's provenance, the bytecode + floors and the next version; + - through `.github/post-release-check.py`: the natives jars, their provenance with the patch hash + recomputed from the tag, the source archive, the AARs and the cache-free tag build; + - through a consumer on Java 8 per platform: `llama-platform` resolved from Central into an empty + local repository, the server smoke and the embedded WebUI. + + Run against 6.0.0 it reports exactly the one known finding, checker-qual in the AAR poms. - **`verify-aar-poms.py`: the published AAR poms are checked against the core pom** in `package-android-aar` -- exactly the core's compile-scope dependencies, at its versions and at `compile`. diff --git a/CLAUDE.md b/CLAUDE.md index 6d27bfe04..acc107860 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -371,6 +371,7 @@ rejects anything else loudly. | `nativedeps.py` | `verify-native-deps.py` (`package`, `package-android-aar`) | dependency allowlists, 16 KB Android alignment | | `hipoffload.py` | `verify-hip-offload-compressed.py` (the two ROCm jobs) | no uncompressed GPU code bundle | | `aarpoms.py` | `verify-aar-poms.py` (`package-android-aar`, on the poms in mavenLocal) | both AAR poms carry exactly the core's compile-scope dependencies, at its versions and at `compile` (plus the CPU AAR, `type aar`, in the OpenCL one) | +| `postrelease.py` | `post-release-check.py` (`post-release.yml`, after the release) | the published natives jars against `natives.csv`, their provenance with `patches.sha256` recomputed from the tag, the source archive's stamp and reproducible pack, the AAR invariants and poms, the cache-free tag build (docs/post-release-check.md, checks 3, 8, 9, 13, 14) | | `webuilicenses.py` | `webui-licenses.py` (`build-webui`, and `webui-licenses.yml` on every pull request and push) | the WebUI's npm license texts cover the pinned lockfile (digest over LF line ends) and its imports; it fails every run -- until the release run of v6.0.0 it only warned outside a release tag, and a stale list surfaced there first | | Action (bernardladenthin/actions) | Job here | What it checks or does | @@ -381,6 +382,15 @@ rejects anything else loudly. | `java/check-maven-versions` | `repository-checks` | **warns** where a Maven dependency or plugin is used in another version than in a sibling repo | | `java/check-publication` | `repository-checks` | the publication jobs are wired as documented; a dry run of `.github/release-assets.json` collects what it promises | | `java/check-release-ref`, `java/finalize-publication`, `java/publish-central`, `java/publish-github`, `java/attest-publication` | `check-snapshot` / `check-tag`, the publish jobs, the six consumers | the signed publication (see "Signed publication") | +| `java/audit-publication` | `audit` of `post-release.yml` | the published release from the outside: the run and its jobs, the files against the contract, Maven Central (polled), the signatures with the release key from a keyserver, every file's provenance, the bytecode floors, the next version (`post-release` in `.github/actions-config.json`) | + +**The post-release check (`post-release.yml`).** Runs after a successful Publish run dispatched on a +`v*` tag, and by hand with a tag: `audit` (the action above, then `post-release-check.py` on the +downloaded files) and `consumer` (per platform: `post-release-consumer.sh` resolves `llama-platform` +from Central into an empty local repository, then the server smoke on Java 8, and on Linux x86-64 +`post-release-webui.sh`). Tools from the default branch, the release's facts from a checkout of the +tag in `release/`. It is not in the release gate -- it runs after the publication, on what was +published (docs/post-release-check.md). **The release gate.** A job nothing waits for can go red while a release still ships — the natives builds `package` once forgot to wait for, and the aarch64 fat jars that were signed and attached diff --git a/TODO.md b/TODO.md index 35a4cd8c2..7a28159a3 100644 --- a/TODO.md +++ b/TODO.md @@ -65,6 +65,21 @@ and footer, `SECURITY.md` on `6.x`) is the pull request `release/6.0.0`. (imports + lockfile); an exact list from the bundle would need a plugin in upstream's build. - [ ] After publication: carry out the upstream submissions queued below (`main` is at `6.1.0-SNAPSHOT` since the post-release bump). +- [ ] **The automated post-release check** (`.github/workflows/post-release.yml`, + docs/post-release-check.md "Automated"). Run locally against 6.0.0 before the first CI run: + the 8 checks of `java/audit-publication` pass (60 files + 60 signatures from one subkey, 118 + Central files with the release's bytes, 120 files attested by run 40239731871; `next` warns, + `main` was still at 6.0.0), `post-release-check.py` passes natives (23/23), provenance + (patches.sha256 recomputed), source and cache (73 logs, 68 show the cache off, none on) and + fails android on exactly the known checker-qual finding; the consumer resolution, the server + smoke and the WebUI check pass on the jars from Central (in Docker, JDK 21, not 8). Open: + - [ ] Move the `java/audit-publication` pin from the actions branch commit to the actions + release that carries it, before this branch is merged. + - [ ] Read the first real run: a `workflow_dispatch` with `v6.0.0` (expected: one android + finding) and the first `workflow_run` after 6.1.0. Unverified on a runner: the Java 8 + rows (Temurin 8 on Linux and Windows, Zulu 8 on macOS arm64), `smoke-native-macos.sh` + and `smoke-test-natives.ps1` on Java 8, `gh attestation verify` with the workflow token + (`attestations: read`), and the keyserver fetch from a runner. - [ ] After publication: srcmorph (#238) builds against `llama:6.0.0` from Central; its Build, plugin IT, code style, CodeQL and SonarQube jobs are red until then, and a local srcmorph build stops at its license collector against a llama snapshot older than the natives license diff --git a/docs/RELEASE.md b/docs/RELEASE.md index be3bc0aae..222ea8600 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -8,6 +8,8 @@ java-llama.cpp is a **Maven reactor**, so two repo-specific points extend the ca Once a release is published, audit it from the outside with [`post-release-check.md`](post-release-check.md): that file checks the artifacts on the GitHub release, on Maven Central and in a real JVM, deliberately without trusting the run that built them. +`.github/workflows/post-release.yml` runs every check a runner can answer by itself after the +Publish run of the tag; read its result before announcing the release. ## Reactor version bump (all six poms) diff --git a/docs/post-release-check.md b/docs/post-release-check.md index 2dc4c2134..b36d319a6 100644 --- a/docs/post-release-check.md +++ b/docs/post-release-check.md @@ -22,6 +22,27 @@ artifacts merely claim. **Run it on the published artifacts, every release.** A check that can only say "CI was green" does not belong in this file. +## Automated: `.github/workflows/post-release.yml` + +Every check below that a GitHub runner can answer runs by itself after the release: the workflow +starts when a Publish run dispatched on a `v*` tag succeeds (a release created with the workflow +token starts no `release:` workflow), and by hand with `workflow_dispatch` and the tag. The tools and +the settings come from the default branch, the release's own facts (contract, `natives.csv`, +patches, poms) from a second checkout of the tag, so an older release is audited with today's checks +too. Run against 6.0.0 it reports exactly the one finding recorded below, the AAR poms' checker-qual. + +| Check | Where it runs | +|---|---| +| 1 ref/run/tag, 2 jobs, 4 completeness, 5 Central, 6 signatures, 7 provenance, 10 bytecode, 16 next version | `audit` job: `java/audit-publication` of [bernardladenthin/actions](https://github.com/bernardladenthin/actions) (shared by every repository there; settings in the `post-release` section of `.github/actions-config.json`) | +| 3 cache-free build, 8 natives jars, 9 provenance recomputed, 13 source archive, 14 Android | `audit` job: `.github/post-release-check.py` (`buildcheck/postrelease.py`), on the files the action downloaded | +| 11 consumer resolution | `consumer` matrix: `.github/post-release-consumer.sh` -- `llama-platform` plus a GPU classifier into an empty local repository | +| 12 runs on the floor JDK | `consumer` matrix on Java 8 (Linux x86-64 and aarch64, Windows x86-64, macOS arm64): the server smoke of the natives jars on the resolved set -- inference, TLS, an `https://` model download, the natives directory | +| 15 server and WebUI | `consumer`, Linux x86-64: `.github/post-release-webui.sh` | + +**Still by hand:** the GPU half of check 12 (no GPU runner: a real driver loading a shipped module, +the device list, the throughput baseline) and reading check 16's standing gaps. The automated checks +are what to read first; the sections below say what each one proves and how to run it locally. + ## Conventions Each check states **what it proves** (the failure class it would catch), **how** to run it, and the From 385d104c598fdd267a1a34ab5516f67e443d509b Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 11:15:45 +0100 Subject: [PATCH 08/13] Pin bernardladenthin/actions at v1.1.0 v1.1.0 (44390bd4) adds java/audit-publication. post-release.yml pinned that action at its branch commit until the release; now it and every other pin of the workflows name v1.1.0. Nothing changes for the actions already in use. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AJGh28Gu8pLWYbTCqG1LZf --- .github/workflows/codeql.yml | 2 +- .github/workflows/dependency-submission.yml | 2 +- .github/workflows/java-tests.yml | 4 +- .github/workflows/osv-scanner.yml | 2 +- .github/workflows/post-release.yml | 2 +- .github/workflows/publish.yml | 102 ++++++++++---------- .github/workflows/reuse.yml | 2 +- CHANGELOG.md | 4 + TODO.md | 10 +- 9 files changed, 67 insertions(+), 63 deletions(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index f58243f07..baba73a38 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -18,7 +18,7 @@ permissions: jobs: analyze: - uses: bernardladenthin/actions/.github/workflows/java-codeql.yml@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/.github/workflows/java-codeql.yml@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 permissions: actions: read contents: read diff --git a/.github/workflows/dependency-submission.yml b/.github/workflows/dependency-submission.yml index f031c5310..bdc975157 100644 --- a/.github/workflows/dependency-submission.yml +++ b/.github/workflows/dependency-submission.yml @@ -16,6 +16,6 @@ permissions: jobs: submit: - uses: bernardladenthin/actions/.github/workflows/java-maven-dependency-submission.yml@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/.github/workflows/java-maven-dependency-submission.yml@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 permissions: contents: write diff --git a/.github/workflows/java-tests.yml b/.github/workflows/java-tests.yml index cf8af3af4..10b15c55c 100644 --- a/.github/workflows/java-tests.yml +++ b/.github/workflows/java-tests.yml @@ -43,7 +43,7 @@ jobs: with: persist-credentials: false - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: name: ${{ inputs.natives-artifact }} @@ -116,7 +116,7 @@ jobs: # Echo the crash logs into the job log (workspace/policies/ci-test-diagnostics.md 3.1). - name: Print crash logs (on failure) if: failure() - uses: bernardladenthin/actions/java/print-crash-logs@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/print-crash-logs@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: modules: llama - if: failure() diff --git a/.github/workflows/osv-scanner.yml b/.github/workflows/osv-scanner.yml index 474ad01dd..3c47a064e 100644 --- a/.github/workflows/osv-scanner.yml +++ b/.github/workflows/osv-scanner.yml @@ -19,7 +19,7 @@ permissions: jobs: scan: - uses: bernardladenthin/actions/.github/workflows/osv-scanner.yml@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/.github/workflows/osv-scanner.yml@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 permissions: actions: read contents: read diff --git a/.github/workflows/post-release.yml b/.github/workflows/post-release.yml index 9064f0c7b..cac91b2cc 100644 --- a/.github/workflows/post-release.yml +++ b/.github/workflows/post-release.yml @@ -72,7 +72,7 @@ jobs: path: release persist-credentials: false - name: Audit the release (run, jobs, files, Maven Central, signatures, provenance, bytecode) - uses: bernardladenthin/actions/java/audit-publication@630e3a93f4222f87da5de16cc48761068596f458 # audit-publication branch, pending a release + uses: bernardladenthin/actions/java/audit-publication@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: tag: ${{ env.TAG }} root: release diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index c82f15b16..04741885e 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -120,19 +120,19 @@ jobs: with: persist-credentials: false - name: Files equal their templates in bernardladenthin/actions - uses: bernardladenthin/actions/common/check-templates@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/check-templates@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 # This repository's own build checks build on actionlib (the setup step puts it on PYTHONPATH). - - uses: bernardladenthin/actions/common/setup@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + - uses: bernardladenthin/actions/common/setup@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build-check tests of this repository run: python3 -m unittest discover -s .github/buildcheck/tests -t .github - name: Release gate (every job gates both publish jobs, or release-gate-exemptions.txt says why) - uses: bernardladenthin/actions/common/check-release-gate@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/check-release-gate@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Run scripts parse (bash -n over every bash run script of the workflows and actions) - uses: bernardladenthin/actions/common/check-run-scripts@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/check-run-scripts@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Maven versions (warns where a dependency or plugin differs from a sibling repository) - uses: bernardladenthin/actions/java/check-maven-versions@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/check-maven-versions@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Publication wiring and release asset contract (dry run) - uses: bernardladenthin/actions/java/check-publication@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/check-publication@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 # --------------------------------------------------------------------------- # GPG signing-key preflight (standalone, no `needs:` -- runs in parallel at the @@ -148,7 +148,7 @@ jobs: runs-on: ubuntu-latest environment: maven-central steps: - - uses: bernardladenthin/actions/common/verify-gpg-key@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + - uses: bernardladenthin/actions/common/verify-gpg-key@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: private-key: ${{ secrets.GPG_PRIVATE_KEY }} passphrase: ${{ secrets.GPG_PASSPHRASE }} @@ -168,7 +168,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false - - uses: bernardladenthin/actions/java/verify-signing-key-gradle@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + - uses: bernardladenthin/actions/java/verify-signing-key-gradle@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: private-key: ${{ secrets.GPG_PRIVATE_KEY }} passphrase: ${{ secrets.GPG_PASSPHRASE }} @@ -272,7 +272,7 @@ jobs: print(f'{len(files)} workflow files, {bad} that do not parse') sys.exit(1 if bad else 0) PY - - uses: bernardladenthin/actions/common/setup@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + - uses: bernardladenthin/actions/common/setup@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Natives list check (everything that names a natives jar agrees with natives.csv) run: python3 .github/check-natives.py - name: Patch hunk check (every hunk of llama/patches/*.patch declares exactly the lines it carries) @@ -353,7 +353,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Download Linux x86_64 native library (reused, not rebuilt) uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: @@ -378,7 +378,7 @@ jobs: # Echo the crash logs into the job log (workspace/policies/ci-test-diagnostics.md 3.1). - name: Print crash logs (on failure) if: failure() - uses: bernardladenthin/actions/java/print-crash-logs@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/print-crash-logs@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: modules: llama-langchain4j - if: failure() @@ -410,7 +410,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Download Linux x86_64 native library (reused, not rebuilt) uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: @@ -441,7 +441,7 @@ jobs: done - name: Print crash logs (on failure) if: failure() - uses: bernardladenthin/actions/java/print-crash-logs@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/print-crash-logs@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: modules: llama-ffm - if: failure() @@ -534,7 +534,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Download Linux x86_64 native library (reused, not rebuilt) uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: @@ -561,7 +561,7 @@ jobs: # Echo the crash logs into the job log (workspace/policies/ci-test-diagnostics.md 3.1). - name: Print crash logs (on failure) if: failure() - uses: bernardladenthin/actions/java/print-crash-logs@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/print-crash-logs@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: modules: llama-atmosphere-agent - if: failure() @@ -713,7 +713,7 @@ jobs: name: webui-generated path: ${{ github.workspace }}/llama/webui-generated/ - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries shell: bash run: | @@ -750,7 +750,7 @@ jobs: name: webui-generated path: ${{ github.workspace }}/llama/webui-generated/ - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries shell: bash run: | @@ -839,7 +839,7 @@ jobs: # source directory, so the stamp the archive carries is what proves the patches are in. bash .github/verify-patches-applied.sh "$RUNNER_TEMP/llama-src" - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries from the archive shell: bash run: | @@ -874,7 +874,7 @@ jobs: name: webui-generated path: ${{ github.workspace }}/llama/webui-generated/ - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries shell: bash run: | @@ -911,7 +911,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Download shared WebUI assets uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: @@ -1055,7 +1055,7 @@ jobs: name: webui-generated path: ${{ github.workspace }}/llama/webui-generated/ - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries shell: bash run: | @@ -1324,7 +1324,7 @@ jobs: swap-storage: false - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6 with: java-version-file: .java-version @@ -1664,7 +1664,7 @@ jobs: distribution: 'temurin' java-version-file: .java-version - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Install sccache (shared compiler cache) if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != '' continue-on-error: true @@ -1701,7 +1701,7 @@ jobs: distribution: 'temurin' java-version-file: .java-version - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Install sccache (shared compiler cache) if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != '' continue-on-error: true @@ -1737,7 +1737,7 @@ jobs: name: webui-generated path: ${{ github.workspace }}/llama/webui-generated/ - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries shell: cmd run: | @@ -1796,7 +1796,7 @@ jobs: with: arch: x64 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Install sccache (shared compiler cache) if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != '' continue-on-error: true @@ -1846,7 +1846,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Download shared WebUI assets uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: @@ -2658,7 +2658,7 @@ jobs: distribution: 'temurin' java-version-file: .java-version - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries run: | mvn -q --no-transfer-progress -f llama/pom.xml compile @@ -2694,7 +2694,7 @@ jobs: echo "CC=gcc-14" >> "$GITHUB_ENV" echo "CXX=g++-14" >> "$GITHUB_ENV" - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries run: | mvn -q --no-transfer-progress -f llama/pom.xml compile @@ -2723,7 +2723,7 @@ jobs: with: arch: x64 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries shell: cmd run: | @@ -2750,7 +2750,7 @@ jobs: with: arch: arm64 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Build libraries shell: cmd run: | @@ -2780,7 +2780,7 @@ jobs: distribution: 'temurin' java-version-file: .java-version - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - name: Install sccache (shared compiler cache) if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != '' continue-on-error: true @@ -2809,7 +2809,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: name: natives-cpu-linux-x86-64 @@ -2863,7 +2863,7 @@ jobs: # Echo the crash logs into the job log (workspace/policies/ci-test-diagnostics.md 3.1). - name: Print crash logs (on failure) if: failure() - uses: bernardladenthin/actions/java/print-crash-logs@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/print-crash-logs@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: modules: llama - if: failure() @@ -3170,7 +3170,7 @@ jobs: # dylib shipped; the test-only macOS builds (macos-26-metal, macos-15-no-metal) are named # outside the glob for that reason. - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: pattern: "natives-*" @@ -3212,7 +3212,7 @@ jobs: # module-info.class and META-INF/versions/** are skipped unconditionally because a # classpath JVM never loads them. Kept byte-identical across all four sibling repos. - name: Verify Java 8 bytecode (no class newer than major 52) - uses: bernardladenthin/actions/java/verify-bytecode@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/verify-bytecode@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: max-major: "52" paths: | @@ -3378,7 +3378,7 @@ jobs: with: persist-credentials: false - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: name: llama-smoke-${{ matrix.target }} @@ -3393,7 +3393,7 @@ jobs: # script is bash-only, so the Linux rows check; the Windows sets hold the same jars. - name: Verify Java 8 bytecode (no class newer than major 52) if: runner.os == 'Linux' - uses: bernardladenthin/actions/java/verify-bytecode@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/verify-bytecode@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: max-major: "52" paths: | @@ -3454,7 +3454,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: name: llama-atmosphere-agent-jar @@ -3482,7 +3482,7 @@ jobs: # dumb terminal instead of failing (verified on 21.0.10). Only that package is waived, so # any other class above 65 still fails the gate. - name: Verify Java 21 bytecode (no class newer than major 65) - uses: bernardladenthin/actions/java/verify-bytecode@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/verify-bytecode@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: max-major: "65" allow: | @@ -3525,7 +3525,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Display CPU Info - uses: bernardladenthin/actions/common/print-host-info@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 with: name: llama-smoke-macos-aarch64 @@ -3591,7 +3591,7 @@ jobs: # a release is tagged on. - name: Require a SNAPSHOT version if: inputs.publish_to_central - uses: bernardladenthin/actions/java/check-release-ref@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/check-release-ref@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: snapshot @@ -3606,7 +3606,7 @@ jobs: persist-credentials: false # The tag names exactly the project version (v1.2.3 for 1.2.3), which is no SNAPSHOT. - name: Require the release version the tag names - uses: bernardladenthin/actions/java/check-release-ref@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/check-release-ref@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: release @@ -3713,7 +3713,7 @@ jobs: MAVEN_GPG_KEY_ID: ${{ secrets.GPG_KEY_ID }} - name: Finalize and secure the signed publication id: finalize - uses: bernardladenthin/actions/java/finalize-publication@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/finalize-publication@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: snapshot @@ -3729,7 +3729,7 @@ jobs: permissions: contents: write steps: - - uses: bernardladenthin/actions/java/publish-github@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + - uses: bernardladenthin/actions/java/publish-github@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: snapshot assets-complete: ${{ needs.publish-snapshot.outputs.assets_complete }} @@ -3834,7 +3834,7 @@ jobs: MAVEN_GPG_KEY_ID: ${{ secrets.GPG_KEY_ID }} - name: Finalize and secure the signed publication id: finalize - uses: bernardladenthin/actions/java/finalize-publication@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/finalize-publication@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: release @@ -3847,7 +3847,7 @@ jobs: permissions: contents: write steps: - - uses: bernardladenthin/actions/java/publish-github@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + - uses: bernardladenthin/actions/java/publish-github@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: release assets-complete: ${{ needs.publish-release.outputs.assets_complete }} @@ -3872,7 +3872,7 @@ jobs: with: persist-credentials: false - name: Transfer frozen artifacts (no rebuild, signing or validation polling) - uses: bernardladenthin/actions/java/publish-central@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/publish-central@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: snapshot assets-complete: ${{ needs.publish-snapshot.outputs.assets_complete }} @@ -3893,7 +3893,7 @@ jobs: with: persist-credentials: false - name: Transfer frozen artifacts (no rebuild, signing or validation polling) - uses: bernardladenthin/actions/java/publish-central@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/java/publish-central@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: release assets-complete: ${{ needs.publish-release.outputs.assets_complete }} @@ -3911,7 +3911,7 @@ jobs: id-token: write attestations: write steps: - - uses: bernardladenthin/actions/java/attest-publication@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + - uses: bernardladenthin/actions/java/attest-publication@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: snapshot assets-complete: ${{ needs.publish-snapshot.outputs.assets_complete }} @@ -3927,7 +3927,7 @@ jobs: id-token: write attestations: write steps: - - uses: bernardladenthin/actions/java/attest-publication@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + - uses: bernardladenthin/actions/java/attest-publication@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 with: channel: release assets-complete: ${{ needs.publish-release.outputs.assets_complete }} diff --git a/.github/workflows/reuse.yml b/.github/workflows/reuse.yml index ac8738a61..b467713cd 100644 --- a/.github/workflows/reuse.yml +++ b/.github/workflows/reuse.yml @@ -14,4 +14,4 @@ permissions: jobs: test: - uses: bernardladenthin/actions/.github/workflows/reuse.yml@23543c8554ca9de6792a1b8d8aae0bcd001d8508 # v1.0.3 + uses: bernardladenthin/actions/.github/workflows/reuse.yml@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0 diff --git a/CHANGELOG.md b/CHANGELOG.md index 859a3a249..3d6a4b73f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,10 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by ## [Unreleased] +### Changed +- **`bernardladenthin/actions` pinned at v1.1.0.** v1.1.0 adds `java/audit-publication`, which + `post-release.yml` uses (see Added); nothing changes for the actions already in use. + ### Fixed - **`cpu-linux-s390x` loads on glibc 2.28, like the other Linux CPU jars** (RHEL 8, Ubuntu 20.04, Debian 10 and later, z13 or newer). The 6.0.0 jar was cross-compiled against the diff --git a/TODO.md b/TODO.md index 7a28159a3..0227216d1 100644 --- a/TODO.md +++ b/TODO.md @@ -41,12 +41,12 @@ and footer, `SECURITY.md` on `6.x`) is the pull request `release/6.0.0`. token was refused (`Resource not accessible by integration`, 403), so the `snapshot` tag and notes still name `53f307ac` (May) and nothing was pruned. Presumably the workflow token may not move a ref across commits that change `.github/workflows/` (12 files between the two); - not verified. actions v1.0.3 (pinned) deletes and re-creates the tag on that 403 and points + not verified. actions v1.0.3 (and later; v1.1.0 is pinned) deletes and re-creates the tag on that 403 and points the release at it; whether GitHub accepts that shows only the next snapshot dispatch. `github-release` does not move a tag (the owner's `v6.0.0` exists already). - `central-snapshot`: `superseded()` died on a GitHub API `502` listing the jobs of an older run, in the re-run too: with 100 per page GitHub fails that listing for large runs, with 50 - it answered every try. actions v1.0.3 (pinned) lists 50 per page and repeats GitHub reads on + it answered every try. actions v1.0.3 (and later) lists 50 per page and repeats GitHub reads on 5xx. `central-release` does not call it. - [ ] **Owner: settings of bernardladenthin/actions** like the four repositories: a ruleset on `main` (pull request required, no force push, no deletion), Dependabot alerts and security updates, @@ -73,8 +73,8 @@ and footer, `SECURITY.md` on `6.x`) is the pull request `release/6.0.0`. (patches.sha256 recomputed), source and cache (73 logs, 68 show the cache off, none on) and fails android on exactly the known checker-qual finding; the consumer resolution, the server smoke and the WebUI check pass on the jars from Central (in Docker, JDK 21, not 8). Open: - - [ ] Move the `java/audit-publication` pin from the actions branch commit to the actions - release that carries it, before this branch is merged. + - [x] The `java/audit-publication` pin is the actions release that carries it: v1.1.0 + (`44390bd4`), like every other pin of the workflows. - [ ] Read the first real run: a `workflow_dispatch` with `v6.0.0` (expected: one android finding) and the first `workflow_run` after 6.1.0. Unverified on a runner: the Java 8 rows (Temurin 8 on Linux and Windows, Zulu 8 on macOS arm64), `smoke-native-macos.sh` @@ -85,7 +85,7 @@ and footer, `SECURITY.md` on `6.x`) is the pull request `release/6.0.0`. build stops at its license collector against a llama snapshot older than the natives license texts. Re-run its checks once the release is on Central. - [ ] After publication: the first snapshot dispatches of BitcoinAddressFinder and streambuffer on - actions v1.0.3 (none has run on `main` since the pin bumps), and the next one of this + actions v1.0.3 or later (none has run on `main` since the pin bumps), and the next one of this repository, which shows whether the re-created `snapshot` tag is accepted. ## Delivery roadmap — remaining work From 1443882220b3b48c23acea51afbaa1e03f30d109 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 11:26:48 +0100 Subject: [PATCH 09/13] TODO: srcmorph builds against llama 6.0.0 from Central (srcmorph#243) Verified locally from an empty local repository: 668 + 39 + 32 tests, BUILD SUCCESS. What stays open is srcmorph#243's CI. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AJGh28Gu8pLWYbTCqG1LZf --- TODO.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/TODO.md b/TODO.md index 0227216d1..d56443012 100644 --- a/TODO.md +++ b/TODO.md @@ -80,10 +80,10 @@ and footer, `SECURITY.md` on `6.x`) is the pull request `release/6.0.0`. rows (Temurin 8 on Linux and Windows, Zulu 8 on macOS arm64), `smoke-native-macos.sh` and `smoke-test-natives.ps1` on Java 8, `gh attestation verify` with the workflow token (`attestations: read`), and the keyserver fetch from a runner. -- [ ] After publication: srcmorph (#238) builds against `llama:6.0.0` from Central; its Build, - plugin IT, code style, CodeQL and SonarQube jobs are red until then, and a local srcmorph - build stops at its license collector against a llama snapshot older than the natives license - texts. Re-run its checks once the release is on Central. +- [ ] After publication: srcmorph builds against `llama:6.0.0` from Central -- locally verified + 2026-10-26 from an empty local repository (668 + 39 + 32 tests, BUILD SUCCESS); its TODO item + is closed in srcmorph#243. Open: that PR's CI, the first run of its Build, plugin IT, code + style, CodeQL and SonarQube jobs that can resolve the release. - [ ] After publication: the first snapshot dispatches of BitcoinAddressFinder and streambuffer on actions v1.0.3 or later (none has run on `main` since the pin bumps), and the next one of this repository, which shows whether the re-created `snapshot` tag is accepted. From c454baa59d57badb8ef6a7c219d0e7e6afeadc00 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 11:42:57 +0100 Subject: [PATCH 10/13] Windows guide: valid XML in the agent pom, cmd examples, and the agent's own flags Three defects in docs/windows-intel-core-ultra.md, two of them reported from the machine the guide is written for. 1. The agent pom did not parse. Its comment read "only needed for --model ... with --base-url ...", and a double hyphen is illegal inside an XML comment, so every editor and every parser flags the snippet. Reworded without one; both pom snippets are now checked with a real XML parser (xml.etree), not by eye. 2. The agent rejects the server's flags, which the guide did not say. It has its own parser (AgentOptions) and accepts only long forms -- --model, --ctx-size, --ngl/--gpu-layers -- so the server's -m fails with "unknown argument: -m" plus a usage block, which is what happened. Stated as an IMPORTANT note next to the first agent command, with the reason: unknown flags are refused rather than ignored, deliberately. 3. The examples are Windows batch now, not PowerShell, and each of the two steps opens with the JAVA_HOME it needs -- openjdk1.8.0_491 for the server, openjdk21.0.11 for the agent -- followed by PATH and the mvn call, so a step is one block to paste. `cd /d` for the drive-aware change, and the port test uses curl.exe (in PowerShell 5.1 `curl` is an alias for Invoke-WebRequest, so the bare name would behave differently). The cmd quoting is verified rather than assumed: `-Dexec.args="..."` was run through cmd.exe exactly as printed, and the whole argument arrived -- the server reported loading 'C:\models\does-not-exist.gguf' and accepted the remaining flags, failing only on the deliberately absent file. PowerShell needs `"-Dexec.args=..."` instead, which is why the blocks are bat. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- docs/windows-intel-core-ultra.md | 88 ++++++++++++++++++++------------ 1 file changed, 54 insertions(+), 34 deletions(-) diff --git a/docs/windows-intel-core-ultra.md b/docs/windows-intel-core-ultra.md index eb159704d..6ded0d7e3 100644 --- a/docs/windows-intel-core-ultra.md +++ b/docs/windows-intel-core-ultra.md @@ -51,16 +51,19 @@ you need. The agent is Java 21 bytecode; on a JDK 8 it fails in the compiler with `invalid target release: 21` and no flag helps. So the server runs on the 8, the agent on the 21, and `JAVA_HOME` decides which — -per window, because `exec:java` runs in Maven's own JVM: +set per window, because `exec:java` runs in Maven's own JVM. Both steps below open with exactly these +two lines, naming the JDK that step needs: -```powershell -# the window that starts the agent -$env:JAVA_HOME = 'C:\Program Files\Java\openjdk21.0.11' -mvn -v # must report: Java version: 21... +```bat +set "JAVA_HOME=C:\Program Files\Java\openjdk1.8.0_491" +set "PATH=%JAVA_HOME%\bin;%PATH%" +mvn -v ``` -Leave the machine-wide `JAVA_HOME` on the JDK 8 so the server keeps working in every other window. -Nothing else needs setting: Maven resolves the classpath from `~\.m2\repository` itself. +`mvn -v` must then report that JDK (`Java version: 1.8.0_491`, or `21.0.11` for the agent) — that +number decides, not `java -version`. The two `set` lines live and die with the window, so the +machine-wide `JAVA_HOME` can stay wherever it is. Nothing else needs setting: Maven resolves the +classpath from `%USERPROFILE%\.m2\repository` itself. ## Step 1 — two poms @@ -143,8 +146,8 @@ excluded because the agent's published pom pulls the library jars of all six des - + net.ladenthin llama @@ -176,11 +179,14 @@ excluded because the agent's published pom pulls the library jars of all six des ## Step 2 — pre-load everything, once Once per pom, with network. This downloads every artifact, every transitive dependency **and the -plugins** into `~\.m2\repository`: +plugins** into `%USERPROFILE%\.m2\repository`: -```powershell -cd C:\llama\server ; mvn dependency:go-offline -cd C:\llama\agent ; mvn dependency:go-offline +```bat +cd /d C:\llama\server +mvn dependency:go-offline + +cd /d C:\llama\agent +mvn dependency:go-offline ``` Either JDK does this — Maven only downloads here, it neither compiles nor loads the jars, so the @@ -197,9 +203,12 @@ what runs — the effective classpath still honours the exclusion. Put the GGUF anywhere, e.g. `C:\models\`. Any tool-capable instruct model works; a reasonable default here is Qwen3-4B-Instruct-2507-Q4_K_M (2.3 GB). -```powershell -cd C:\llama\server -mvn -o exec:java "-Dexec.args=-m C:\models\Qwen3-4B-Instruct-2507-Q4_K_M.gguf --host 127.0.0.1 --port 9931 -c 16384 -ngl 99 -t 8 -tb 10 --jinja" +```bat +set "JAVA_HOME=C:\Program Files\Java\openjdk1.8.0_491" +set "PATH=%JAVA_HOME%\bin;%PATH%" + +cd /d C:\llama\server +mvn -o exec:java -Dexec.args="-m C:\models\Qwen3-4B-Instruct-2507-Q4_K_M.gguf --host 127.0.0.1 --port 9931 -c 16384 -ngl 99 -t 8 -tb 10 --jinja" ``` This is the **real upstream `llama_server` running inside `libjllama`** over JNI, so every @@ -207,11 +216,13 @@ llama-server flag works verbatim and the bundled WebUI is served. `--jinja` enab template, which is what makes tool calling work — the agent in step 4 needs it. `-ngl 99` offloads to the iGPU through Vulkan; drop it to compare against the CPU. -Test the port from another window: +Test the port from another window (`curl.exe`, not `curl` — in PowerShell 5.1 `curl` is an alias for +`Invoke-WebRequest`): + +```bat +curl.exe http://127.0.0.1:9931/health -```powershell -Invoke-RestMethod http://127.0.0.1:9931/health -Invoke-RestMethod -Method Post http://127.0.0.1:9931/v1/chat/completions -ContentType application/json -Body '{"messages":[{"role":"user","content":"2+2=?"}],"max_tokens":16}' +curl.exe http://127.0.0.1:9931/v1/chat/completions -H "Content-Type: application/json" -d "{\"messages\":[{\"role\":\"user\",\"content\":\"2+2=?\"}],\"max_tokens\":16}" ``` and open in a browser for the WebUI. The OpenAI-compatible API lives on the @@ -220,23 +231,32 @@ same port under `/v1/…`. (To embed the server in your own Java 8 code instead, ## Step 4 — the Atmosphere agent on the JDK 21 -Leave the server running. In a new window, point `JAVA_HOME` at the 21 and start the agent against -that server — the model stays loaded once, in the server, and the agent drives it over HTTP: +Leave the server running. In a **new** window — this is where the other JDK goes — start the agent +against that server, so the model stays loaded once, in the server, and the agent drives it over +HTTP: -```powershell -$env:JAVA_HOME = 'C:\Program Files\Java\openjdk21.0.11' -cd C:\llama\agent -mvn -o -q exec:java "-Dexec.args=--base-url http://127.0.0.1:9931/v1 --workspace C:\path\to\project --allow-shell" +```bat +set "JAVA_HOME=C:\Program Files\Java\openjdk21.0.11" +set "PATH=%JAVA_HOME%\bin;%PATH%" + +cd /d C:\llama\agent +mvn -o -q exec:java -Dexec.args="--base-url http://127.0.0.1:9931/v1 --workspace C:\path\to\project --allow-shell" ``` A `you>` prompt appears. `/clear` drops the history, `/exit` quits, `--prompt "…"` runs a single turn and exits, `--plain` switches to a line-oriented console for piping or logging. +> [!IMPORTANT] +> **The agent's flags are not the server's.** The server takes llama.cpp's argv verbatim (`-m`, `-c`, +> `-ngl`), the agent has its own parser and accepts only the long forms — `--model`, `--ctx-size`, +> `--ngl`/`--gpu-layers`. Passing the server's `-m` to the agent fails with `unknown argument: -m` +> followed by its usage block. Flags it does not know are never ignored, which is deliberate. + For **Atmosphere's web console on its own port** instead of the terminal, add `--web`; it listens on `127.0.0.1:8787` and prints a URL with a fresh access token on every start: -```powershell -mvn -o -q exec:java "-Dexec.args=--base-url http://127.0.0.1:9931/v1 --workspace C:\path\to\project --allow-shell --web --web-port 8787 --web-token my-own-access-token-123" +```bat +mvn -o -q exec:java -Dexec.args="--base-url http://127.0.0.1:9931/v1 --workspace C:\path\to\project --allow-shell --web --web-port 8787 --web-token my-own-access-token-123" ``` `--web-port 0` takes any free port, and `--web-host 0.0.0.0` exists but warns loudly — tunnel over @@ -248,8 +268,8 @@ exclude each other. Without a running server the agent can load the GGUF itself (`--model` instead of `--base-url`), which is what the two natives dependencies in its pom are for: -```powershell -mvn -o -q exec:java "-Dexec.args=--model C:\models\Qwen3-4B-Instruct-2507-Q4_K_M.gguf --ngl 99 --ctx-size 16384 --workspace C:\path\to\project --allow-shell" +```bat +mvn -o -q exec:java -Dexec.args="--model C:\models\Qwen3-4B-Instruct-2507-Q4_K_M.gguf --ngl 99 --ctx-size 16384 --workspace C:\path\to\project --allow-shell" ``` > [!WARNING] @@ -282,8 +302,8 @@ load_backends_next_to_this_library: 3 backend(s) registered [RPC, Vulkan, CPU], The `ggml-cpu-.dll` line is the one that tells you whether this CPU got `alderlake`. A one-line build stamp per natives directory (llama.cpp tag, commit, patches, compiler) comes from: -```powershell -cd C:\llama\server +```bat +cd /d C:\llama\server mvn -o exec:java -Dexec.mainClass=net.ladenthin.llama.Diagnose ``` @@ -312,8 +332,8 @@ measured here**, the device choice is OpenVINO's own, and it needs the Intel Ope on `PATH`. To try it: add a fourth dependency with classifier `openvino-windows-x86-64` to the server pom, re-run `go-offline`, and name both modules explicitly. -```powershell -mvn -o exec:java "-Dexec.args=-m C:\models\Qwen3-4B-Instruct-2507-Q4_K_M.gguf --port 9931 -c 16384 -ngl 99 -lv 4" "-Dnet.ladenthin.llama.backend=vulkan,openvino" +```bat +mvn -o exec:java -Dnet.ladenthin.llama.backend=vulkan,openvino -Dexec.args="-m C:\models\Qwen3-4B-Instruct-2507-Q4_K_M.gguf --port 9931 -c 16384 -ngl 99 -lv 4" ``` Read the device list in the log before trusting any number: if a device named like OpenVINO's CPU From c3719f81bada60b005fe9521f0a0505c4fd69231 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 14:28:22 +0100 Subject: [PATCH 11/13] Measure the GPU device dedup on Windows: CUDA wins, by PCI id; Vulkan numbering moves The open half of TODO "Windows GPU modules on real hardware", measured on the published 6.0.0 jars (cpu, cuda13 and vulkan for windows-x86-64) on an RTX 3070 Laptop GPU with an AMD iGPU. Qwen2.5-1.5B Q4_K_M, -ngl 99 -lv 4, three requests per set, the set chosen with -Dnet.ladenthin.llama.backend. - llama.cpp deduplicates the GPU in device selection by PCI id ("skipping device Vulkan0 ... already using device CUDA0 ... with the same id"). The first registered backend, CUDA, wins, and all 29 layers go to CUDA0. - The iGPU is never drafted in. OpenCL drops the NVIDIA GPU as unsupported and registers no device. - cuda13, vulkan and both perform the same: ~190 t/s generation, ~2100-2500 t/s prompt. - Vulkan's device order depends on the module set. With Vulkan alone the RTX is Vulkan1; with CUDA loaded it is Vulkan0. A device name is therefore valid only for the set it was read from. The README says so next to the module list. docs/benchmarks/gpu-module-jars.md gains section 3 with the log lines and the table. TODO.md checks the item off except the tg128 comparison with the static build, which used another model. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AJGh28Gu8pLWYbTCqG1LZf --- README.md | 13 ++++++++ TODO.md | 28 +++++++++-------- docs/benchmarks/gpu-module-jars.md | 49 ++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index aba1932eb..20ce89b77 100644 --- a/README.md +++ b/README.md @@ -297,6 +297,19 @@ iGPU, the example being a Core Ultra 7 165U). > classpath; unset, every GPU module jar on the classpath is used except the opt-in `openvino` one > (see below). +> [!NOTE] +> **Two GPU modules that see the same GPU use it once.** With `cuda13-windows-x86-64` and +> `vulkan-windows-x86-64` on the classpath, ggml registers an NVIDIA GPU twice, as `CUDA0` and as a +> `Vulkan` device. llama.cpp then skips the second entry by its PCI id ("skipping device Vulkan0 … +> already using device CUDA0 … with the same id"). The first registered backend wins, which is CUDA, +> and the whole model goes to `CUDA0`. An integrated GPU next to a discrete one is not used either +> way. Measured on a published 6.0.0 set (an RTX 3070 Laptop GPU and an AMD iGPU), the three sets +> perform the same: [`docs/benchmarks/gpu-module-jars.md`](docs/benchmarks/gpu-module-jars.md), +> section 3. **The device numbering is not stable across module sets:** with only the Vulkan module, +> the RTX is `Vulkan1` and the iGPU `Vulkan0`; with CUDA loaded as well, Vulkan lists the RTX as +> `Vulkan0`. A `--device` / `setDevices(...)` name is only valid for the set of modules it was read +> from, so re-read the device list in the log after adding or removing a GPU jar. + > [!IMPORTANT] > **Two GPU jars need more than their runtime, and both were found by measurement rather than by > reading the code** ([`docs/benchmarks/gpu-module-jars.md`](docs/benchmarks/gpu-module-jars.md)). diff --git a/TODO.md b/TODO.md index d56443012..53cf0031a 100644 --- a/TODO.md +++ b/TODO.md @@ -209,18 +209,22 @@ Kept as a list because each entry names the machine it needs -- the harness is Also check whether CUDA, Vulkan and OpenCL deduplicate the same NVIDIA GPU by `device_id` and which backend wins. Linux CUDA on a Tesla T4 has already been measured; see [the GPU module measurements](docs/benchmarks/gpu-module-jars.md). - **Partly measured on the published 6.0.0 jars** (post-release check, `docs/post-release-check.md` - check 12; RTX 3070 Laptop + AMD iGPU, the three Windows x86-64 jars resolved from Central): the - loader line names both modules (`with GPU module(s) cuda13, vulkan`), ggml registers - `[CUDA, RPC, Vulkan, CPU]`, and the registry then lists **4 devices -- `CUDA0`, `Vulkan0`, - `Vulkan1`, `CPU`** -- with Vulkan's own enumeration naming device 0 as the same RTX 3070 that CUDA - reports. So the discrete GPU is registered **twice**, once per backend, and whatever `device_id` - dedup llama.cpp does happens in device *selection* rather than in registration. That selection - half is still open, and so is the tg128 comparison above: at the default verbosity threshold the - `load_tensors` / per-buffer lines are filtered out of the log (CLAUDE.md, patch `0014` -- llama - INFO maps to TRACE), so a run that answers it has to pass `-lv 4`. The earlier throughput pair on - this host (4 devices 92.0 t/s against cuda13-only 94.3 t/s, within noise) suggests the iGPU is not - drafted in, which is consistent with selection-time dedup but does not establish it. + **Measured 2026-10-26 on the published 6.0.0 jars** + ([`docs/benchmarks/gpu-module-jars.md`](docs/benchmarks/gpu-module-jars.md), section 3; RTX 3070 + Laptop + AMD iGPU, `-lv 4`): + - [x] Registration lists the discrete GPU twice (`CUDA0`, a `Vulkan` device). llama.cpp + **deduplicates it in device selection by PCI id** ("skipping device Vulkan0 … already using + device CUDA0 … with the same id"). The first registered backend wins (CUDA), and all 29 + layers go to `CUDA0`. + - [x] The iGPU is not drafted in, with Vulkan alone or with both modules. + - [x] OpenCL drops the NVIDIA GPU as unsupported (no device), so it takes no part. + - [x] CUDA, Vulkan and both are equal on Qwen2.5-1.5B Q4_K_M: ~190 t/s generation, ~2100–2500 + t/s prompt. + - [x] README sentence added next to the module list. The device numbering depends on the module + set (Vulkan alone: RTX = `Vulkan1`; with CUDA: RTX = `Vulkan0`), so a device name is valid + only for the set it was read from. + - [ ] Still open: the tg128 of the CUDA module against the static build's 265.6 t/s. That + figure used another model, so it needs the same model, `llama-bench`-style, in both builds. 2. **CUDA for Windows arm64 (`cuda13-windows-aarch64`) -- upstream builds it, we do not.** Their `windows-cuda` job is a matrix over `x64` and `arm64`, the latter through `cmake/arm64-windows-msvc-cuda` plus the arm64 `cudart`/`cublas` redistributables. This project has diff --git a/docs/benchmarks/gpu-module-jars.md b/docs/benchmarks/gpu-module-jars.md index d3443cf81..64bf76177 100644 --- a/docs/benchmarks/gpu-module-jars.md +++ b/docs/benchmarks/gpu-module-jars.md @@ -215,6 +215,55 @@ jars is "a module that cannot help is skipped, one that can help is used" — Op neither on a CPU-only host. The README says that the jar needs an Intel GPU/NPU *and* an OpenCL ICD loader. +## 3 · `cuda13` and `vulkan` together on one NVIDIA GPU: deduplicated, CUDA wins + +Measured **2026-10-26** on a Windows laptop with an NVIDIA RTX 3070 Laptop GPU (8 GB, driver 581.83, +CUDA 13.3 toolkit on `PATH`) and an AMD Radeon iGPU. The jars were the **published 6.0.0** set: +`llama`, `cpu-windows-x86-64`, `cuda13-windows-x86-64` and `vulkan-windows-x86-64`. The test ran the +embedded server on JDK 21, with `Qwen2.5-1.5B-Instruct-Q4_K_M` at `-c 4096 -ngl 99 -lv 4` and three +identical chat requests of 128 tokens each (`cache_prompt: false`). `-Dnet.ladenthin.llama.backend` +selected the set without swapping jars. + +The question came from the post-release check of 6.0.0. The registry lists the discrete GPU twice, +as `CUDA0` and as a Vulkan device. Does llama.cpp use it twice, or once, and through which backend? +At the default verbosity the answer was filtered out of the log; at `-lv 4` it is printed: + +``` +load_backends_next_to_this_library: 4 backend(s) registered [CUDA, RPC, Vulkan, CPU], 4 device(s): CUDA0 (CUDA), Vulkan0 (Vulkan), Vulkan1 (Vulkan), CPU (CPU) +llama_prepare_model_devices: skipping device Vulkan0 (NVIDIA GeForce RTX 3070 Laptop GPU) with id 0000:01:00.0 - already using device CUDA0 (NVIDIA GeForce RTX 3070 Laptop GPU) with the same id +llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3070 Laptop GPU) (0000:01:00.0) - 7116 MiB free +load_tensors: offloaded 29/29 layers to GPU +load_tensors: CUDA0 model buffer size = 934.70 MiB +``` + +| Set (`-Dnet.ladenthin.llama.backend`) | Device used | Prompt t/s (requests 2, 3) | Generation t/s (requests 1–3) | +|---|---|--:|--:| +| `cuda13` | `CUDA0` | 2357, 2370 | 186.5, 193.5, 187.7 | +| `vulkan` | `Vulkan1` (the RTX) | 2134, 2120 | 193.6, 189.3, 188.7 | +| both (unset) | `CUDA0`; `Vulkan0` skipped | 2571, 2426 / 2231, 2013 | 185.9, 194.6, 190.9 / 179.7, 190.8, 192.6 | + +The first request of each server is left out of the prompt column: it includes warm-up (950–1400 t/s). +The two "both" rows are two separate server runs. + +- **Deduplication happens in device selection, by PCI id, and the first registered backend wins.** + CUDA registers before Vulkan, so the model goes to `CUDA0`. No VRAM is spent twice, and nothing is + split between two handles of the same GPU. +- **The iGPU is not drafted in.** Only the discrete GPU is used, with Vulkan alone as well as with + both modules. That agrees with the earlier throughput pair on this host: 4 devices at 92.0 t/s + against cuda13 alone at 94.3 t/s. +- **CUDA and Vulkan are equal on this model** (~190 t/s generation, within noise). The 1.5B model is + small; a larger one may separate them. The tg128 comparison against the static build's 265.6 t/s + in TODO.md used a different model and is not answered by this table. +- **OpenCL does not take the NVIDIA GPU at all.** With `opencl-windows-x86-64` added as a third + module, ggml's OpenCL backend selects the "NVIDIA CUDA" platform, then logs `unsupported GPU + 'NVIDIA GeForce RTX 3070 Laptop GPU'` and `drop unsupported device`. It registers as a backend with + no device: 5 backends, still 4 devices. The model goes to `CUDA0` as before (generation 182–190 + t/s), because ggml's OpenCL backend targets Qualcomm Adreno and Intel GPUs. +- **Vulkan's device order depends on the modules loaded.** With Vulkan alone, `ggml_vulkan: 0` is the + AMD iGPU and `1` the RTX. With CUDA loaded as well, `0` is the RTX and `1` the iGPU. A device name + such as `Vulkan1` therefore names a different GPU depending on which module jars are on the + classpath; the README says so next to the module list. + ## What these runs did *not* test * **No Intel GPU or NPU was available**, so nothing here says what OpenVINO does on the hardware it From 4a805d2562a788149c93143b461ccf36219f2792 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 14:43:58 +0100 Subject: [PATCH 12/13] TODO: s390x verified on real IBM Z with the CI-built jar, clang faster than GCC 13 From the first green Publish run (40309665294) and cfarm191 (z16): - The CI checks pass: 694/694 under qemu, 99.62 % sccache hits, GLIBC_2.27 / GLIBCXX_3.4.25 / CXXABI_1.3.11, all three RHEL 8 rows. - The CI-built cpu-linux-s390x jar loads natively (Clang 18.1.3). - Real big-endian inference: Qwen3-0.6B converted with gguf_convert_endian.py answers "Paris." through the embedded server. - Speed against the 6.0.0 GCC 13 jar, three alternating rounds on the shared host: clang ahead in every round (generation 0.62-1.87 against 0.31-0.93 t/s). New open item: s390x runs scalar in both builds (GGML_VXE off). ggml's CPU variants for z15-z17 would use the vector unit without raising the z13 floor. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AJGh28Gu8pLWYbTCqG1LZf --- TODO.md | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/TODO.md b/TODO.md index 53cf0031a..5feb14859 100644 --- a/TODO.md +++ b/TODO.md @@ -534,15 +534,30 @@ loads on RHEL 8.10 s390x (UBI 8 OpenJDK 21 runtime under qemu) where the 6.0.0 j what 6.0.0 shipped), [Unreleased] records the fix. - [x] `smoke-glibc-floor`: the three Linux CPU jars are loaded on RHEL 8.10 in every run, s390x under qemu; `check-natives.py` holds the rows to `floor-set` (derived from `GLIBC_CEILING`). -- [ ] **Read the first Publish run of this branch** -- none has run yet: `build-linux-s390x` (apt - `clang-18` / `binutils-s390x-linux-gnu`, the `docker pull` of the s390x image on the runner, - sccache wrapping clang: hit rate on the second run), its `verify-native-deps.py` step, - `package` (the `llama-glibc-floor` upload) and the three `smoke-glibc-floor` rows -- the s390x - row needs qemu-user-static's binfmt entry to reach Docker on the runner (it did locally, with - Docker Desktop's own entry), the aarch64 row runs natively on `ubuntu-24.04-arm`. -- [ ] Unmeasured: the speed of the clang build against the former GCC 13 build on real s390x (both - scalar, `-march=z13`). No model-backed check runs on s390x at all (a GGUF would have to be - big-endian), so only `llama-bench` on cfarm191 with a converted model could say. +- [x] **The first Publish run of this branch** (40309665294, `c454baa5`) is green on every s390x job, + after the sysroot permission fix: + - `build-linux-s390x`: the sysroot came out at 289 MB with glibc 2.28; sccache wrapped clang + (260 requests, 99.62 % hits on the second run); the C++ suite passed 694/694 under qemu; + `verify-native-deps.py` reads `GLIBC_2.27`, `GLIBCXX_3.4.25`, `CXXABI_1.3.11`. + - All three `smoke-glibc-floor` rows passed, including s390x under qemu (the runner's binfmt + entry reached Docker). +- [x] **The CI-built jar on real IBM Z** (cfarm191, z16, Debian 13; 2026-10-26): + - `NativeLoadSmoke` loads `llama-6.1.0-SNAPSHOT-cpu-linux-s390x.jar` from that run: linked + b11887, `compiler=Clang 18.1.3`, the same patch hash as 6.0.0. + - **Real big-endian inference**, which no CI job does: Qwen3-0.6B Q4_K_M, converted with + llama.cpp's `gguf_convert_endian.py`, served by `ServerLauncher` on JDK 21, answers "Paris." + with `system_fingerprint` b11887-6050b467c. + - **Speed against the 6.0.0 GCC 13 jar**, same model, prompt and `-t 8`, three alternating + rounds on the shared host: generation 0.62–1.87 t/s (clang) against 0.31–0.93 t/s (GCC), + prompt 2.0–6.5 against 1.9–2.8 t/s. Clang was ahead in every round. Absolute numbers are + noisy and low. +- [ ] **s390x runs scalar.** Both the GCC and the clang build have `GGML_NATIVE=OFF`, so `GGML_VXE` is + off: no vector code for any IBM Z. ggml's `GGML_CPU_ALL_VARIANTS` has s390x variants for + z15–z17 (`-march=z15/z16/arch15` with VXE2, NNPA on z16+) that a `GGML_BACKEND_DL` build picks + at run time, as the x86 jars do. Moving s390x from the static single library to the modular + build with those variants would keep the z13 floor (a scalar baseline variant) and use the + vector unit where it exists. To check first: whether clang 18 builds the z15–z17 variants, and + what they gain on cfarm191. - [ ] **Owner:** cfarm192 refused the connection with "Host key verification failed" on 2026-10-26 (cfarm191 worked): its host key changed. Compare the new fingerprint with the farm's published one before accepting it; nothing was changed in `known_hosts`. From d75e9ddd35fce49c84b56031889d5bfda9572921 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 26 Oct 2026 16:15:42 +0100 Subject: [PATCH 13/13] Windows guide: the agent's web console next to the console agent `--web`, `--acp` and `--prompt` exclude each other (AgentOptions.parse), so one agent process serves exactly one front end. The guide said "instead of the terminal" and left the next question unanswered: how to have both. Both is a second agent process against the same server, and it costs nothing extra, because the model lives in the server and each agent is only an HTTP client of it. The block is self-contained like every other step's, with the JDK 21 JAVA_HOME it needs. What the paragraph adds besides the command: the token is part of the printed URL and random on every start unless --web-token pins it, so the address is worth bookmarking only with one; the ssh -L form for reaching it from another machine; one conversation per process, so the two front ends share the model and not the history; and where the console pages come from (atmosphere-spring-boot-starter, carried for those resources alone), which is why --web works out of step 2's offline cache. And the caveat the doc owed the reader: the server of step 3 has one slot, so two agents serialise. --parallel 2 gives each its own slot and splits -c 16384 into 8192 per slot, because the per-slot context is n_ctx_seq, which is n_ctx / n_seq_max unless --kv-unified makes the slots share one pool (src/llama-context.cpp at b11887; kv_unified defaults to false). Read in the pinned source, not assumed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- docs/windows-intel-core-ultra.md | 41 +++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/docs/windows-intel-core-ultra.md b/docs/windows-intel-core-ultra.md index 6ded0d7e3..c52171a98 100644 --- a/docs/windows-intel-core-ultra.md +++ b/docs/windows-intel-core-ultra.md @@ -252,18 +252,43 @@ and exits, `--plain` switches to a line-oriented console for piping or logging. > `--ngl`/`--gpu-layers`. Passing the server's `-m` to the agent fails with `unknown argument: -m` > followed by its usage block. Flags it does not know are never ignored, which is deliberate. -For **Atmosphere's web console on its own port** instead of the terminal, add `--web`; it listens on -`127.0.0.1:8787` and prints a URL with a fresh access token on every start: +**Atmosphere's web console** is `--web`. It is not an addition to the terminal but a replacement for +it: `--web`, `--acp` and `--prompt` exclude each other, so one agent process serves exactly one +front end. To have both, start a **second** agent process against the same server — the model stays +loaded once, in the server, and each agent is only an HTTP client of it. A third window, the JDK 21 +again: ```bat -mvn -o -q exec:java -Dexec.args="--base-url http://127.0.0.1:9931/v1 --workspace C:\path\to\project --allow-shell --web --web-port 8787 --web-token my-own-access-token-123" +set "JAVA_HOME=C:\Program Files\Java\openjdk21.0.11" +set "PATH=%JAVA_HOME%\bin;%PATH%" + +cd /d C:\llama\agent +mvn -o -q exec:java -Dexec.args="--base-url http://127.0.0.1:9931/v1 --workspace C:\path\to\project --allow-shell --web --web-port 8787" ``` -`--web-port 0` takes any free port, and `--web-host 0.0.0.0` exists but warns loudly — tunnel over -SSH rather than binding to the network. One conversation is served: every tab holding the token -shares it. For an IDE, `--acp` speaks the [Agent Client -Protocol](https://agentclientprotocol.com) on stdin/stdout instead; `--web`, `--acp` and `--prompt` -exclude each other. +It prints `Open in a browser: http://127.0.0.1:8787/?token=`. **The token is part of the +URL** and is a new random one on every start, unless `--web-token` pins it (16 or more characters of +`A-Z a-z 0-9 _ -`) — which is what makes the address worth bookmarking. `--web-port 0` takes any +free port. Binding is loopback only, and `--web-host 0.0.0.0` exists but warns loudly: tunnel with +`ssh -L 8787:127.0.0.1:8787 @` and open the same address on the other machine +instead. One conversation per process — every tab holding the token shares it, and a message sent +while another is running replaces it, as typing over a running turn does on the console. +`/mode auto` in the browser stops the approval prompts for the session. The console agent in the +other window keeps its **own** history: the two share the model, not the conversation. + +The console's pages come from `atmosphere-spring-boot-starter`, which the agent pom carries for +those resources alone (every transitive Spring dependency excluded), so Step 2 already cached them +and `--web` works offline. If they were missing, the agent says so on stderr and still serves the +endpoint. + +> [!NOTE] +> The server of Step 3 runs with one slot, so two agents **serialise**: the second waits for the +> first to finish its turn. `--parallel 2` on the server gives each its own slot, and that splits +> `-c 16384` into 8192 per slot — so it is not free. (`--kv-unified` makes the slots share one pool +> at the full size instead, which trades the guarantee for the capacity.) + +For an IDE, `--acp` speaks the [Agent Client Protocol](https://agentclientprotocol.com) on +stdin/stdout instead. Without a running server the agent can load the GGUF itself (`--model` instead of `--base-url`), which is what the two natives dependencies in its pom are for: