diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bda6aba..8b2021a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,8 +45,11 @@ the first two items break every existing consumer at upgrade. `OpenAiServerConfig.Builder.port(int)`) where a fixed port matters, and point clients that were configured for `http://localhost:8080` at the new port. 5. **Platform floors:** Windows 10 or newer (Universal CRT), macOS 15.0, glibc 2.28 on Linux - (RHEL 8, Ubuntu 20.04, Debian 10 and later), Android API 28; the ROCm jars need a ROCm 10 - runtime, the CUDA jars CUDA 13. 32-bit Windows and the `msvc-windows-x86-64` jar no longer ship. + x86-64 and aarch64 (RHEL 8, Ubuntu 20.04, Debian 10 and later), Android API 28. The s390x jar is + the exception: glibc 2.38 and the libstdc++ of GCC 13 (Ubuntu 24.04, SLES 15 SP6, Debian 13; not + RHEL 8/9, not Ubuntu 22.04), because it is cross-compiled on `ubuntu-latest`. The ROCm jars + need a ROCm 10 runtime, the CUDA jars CUDA 13. 32-bit Windows and the `msvc-windows-x86-64` jar + no longer ship. 6. **Rejected requests** come back as `InvalidRequestException` (a `LlamaException`, so existing handlers still match) with the bare reason as the message; the OpenAI-compatible server answers `400` for them. diff --git a/README.md b/README.md index b05bd729..304dc064 100644 --- a/README.md +++ b/README.md @@ -259,7 +259,7 @@ platforms you target, e.g. `cpu-linux-x86-64`. |---|---|---|---| | `cpu-linux-x86-64` | CPU | Linux x86-64 | A JDK 8+ JVM; glibc ≥ 2.28 (manylinux_2_28: RHEL 8, Ubuntu 20.04, Debian 10 and later). Ships one CPU backend module per instruction-set level (x86-64 baseline, SSE4.2, AVX, AVX2, AVX-512, AVX-VNNI, AMX), of which ggml loads the best for the running CPU at start-up — a CPU without AVX2 works, and an AVX-512/AMX machine uses its kernels. | | `cpu-linux-aarch64` | CPU | Linux aarch64 | glibc ≥ 2.28 (RHEL 8, Ubuntu 20.04, Debian 10, Amazon Linux 2023 and later) — built in the manylinux_2_28 image on an arm64 runner. Ships one CPU backend module per ARM feature level (armv8.0 up to armv9.2: dotprod, fp16, SVE, i8mm, SVE2, SME), chosen at start-up like the x86-64 ones. | -| `cpu-linux-s390x` | CPU | Linux s390x (IBM Z, big-endian) | A JDK 8+ JVM. | +| `cpu-linux-s390x` | CPU | Linux s390x (IBM Z, big-endian) | A JDK 8+ JVM; **glibc ≥ 2.38 and the libstdc++ of GCC 13** (`GLIBCXX_3.4.32`): Ubuntu 24.04, SLES 15 SP6, Debian 13 and later. **Not RHEL 8 or 9, not Ubuntu 22.04** -- the load fails there with `GLIBC_2.38 not found`. Cross-compiled on `ubuntu-latest`, not in a manylinux image, so the 2.28 floor of the other Linux CPU jars does not apply. | | `cpu-windows-x86-64` | CPU | Windows x86-64 | A JDK 8+ JVM; **Windows 10 or newer**. Ships one CPU backend module per instruction-set level (as the Linux x86-64 jar: x86-64 baseline, SSE4.2, AVX, AVX2, AVX-512, AVX-VNNI, AMX), of which ggml loads the best for the running CPU at start-up. Built with clang, hybrid CRT: the STL and vcruntime are static (no `msvcp140.dll` / `vcruntime140.dll`, no VC++ redistributable), the Universal CRT is the OS one (`ucrtbase.dll`), so Microsoft's UCRT security updates apply. | | `cpu-windows-aarch64` | CPU | Windows on ARM (Snapdragon X / Surface) | A JDK 8+ JVM; **Windows 10 or newer**. Built natively on `windows-11-arm` with `clang-cl`, same hybrid CRT as the x86-64 jar; one CPU backend module (ggml builds no ARM variant set for Windows), so the Vulkan and OpenCL module jars can join it. | | `metal-macos-aarch64` | Metal + CPU | macOS aarch64 (Apple silicon) | A JDK 8+ JVM. | diff --git a/TODO.md b/TODO.md index 0b2f53a2..9b37091c 100644 --- a/TODO.md +++ b/TODO.md @@ -485,6 +485,62 @@ ROCm anyway. So this needs a real Radeon machine, or a deliberate decision to sh of a green link plus `verify-native-deps.py`'s dependency allowlist. Do not quietly treat "the job is green" as the measurement. +### Linux s390x: glibc 2.38 floor (after the release) + +**Measured 2026-10-25** on the CI artifacts of Publish runs 40217818548 and 40227501358. + +What each Linux CPU jar references: + +| jar | GLIBC | GLIBCXX | built with | +|---|---|---|---| +| `cpu-linux-x86-64` | 2.27 | 3.4.25 | GCC 14.2.1, manylinux_2_28 | +| `cpu-linux-aarch64` | 2.27 | 3.4.25 | GCC 14.2.1, manylinux_2_28 | +| `cpu-linux-s390x` | **2.38** | **3.4.32** | GCC 13.3.0, Ubuntu 24.04 cross toolchain | + +The 2.38 symbols are `__isoc23_strtol` / `strtoul` / `sscanf` and their relatives, and `fmod`. The +2.34 symbols are `pthread_*` and `dl*`. All of them are only the newest symbol versions of the +cross sysroot (glibc 2.39); no code needs them. + +The load was tested as a consumer does it: `llama` + `cpu-linux-s390x` from `llama-jars` on the +classpath, then `LlamaLoader.initialize()`, in Docker with s390x emulated by qemu. + +- **Fails** on RHEL 8.10 (glibc 2.28) with `UnsatisfiedLinkError … GLIBC_2.38 not found`. `ldd` + also lists `GLIBCXX_3.4.26`..`3.4.32` and `CXXABI_1.3.13`, so libstdc++ blocks it as well. +- **Fails** on RHEL 9 (2.34) and Ubuntu 22.04 (2.35). +- **Loads** on Ubuntu 24.04. +- **`ldd` clean** on SLES 15 SP6 (BCI, glibc 2.38), Debian 13 and Ubuntu 25.04. +- **Control:** `cpu-linux-x86-64` and `cpu-linux-aarch64` load on RHEL 8. + +**Not a decision but a side effect.** `GLIBC_CEILING` in `.github/buildcheck/nativedeps.py` holds +only the manylinux directories. Every other Linux build "inherits the floor of the ubuntu runner"; +CLAUDE.md justifies that for GPU modules only. `1559fe4c` never mentions s390x. The CI test cannot +see the gap: its `ctest` runs under qemu against the same 2.39 sysroot. The README and the +CHANGELOG now state the real floor. + +Follow-ups, none of them before the release (no build change now): + +- [ ] **Pin `build-linux-s390x` to `runs-on: ubuntu-24.04`** instead of `ubuntu-latest`. The output + stays identical (24.04 is `ubuntu-latest` today), but a runner image change can no longer + raise the floor to ~2.42 without anyone noticing. +- [ ] **`"Linux/s390x/cpu": (2, 38)` in `GLIBC_CEILING`.** This freezes today's floor, so + `verify-native-deps.py` fails loudly on any further rise instead of passing it. Extend + `test_nativedeps.py` with it. Optionally add a GLIBCXX ceiling too; the check reads GLIBC + only today. +- [ ] **The real fix: floor 2.28 like the other Linux jars.** Then set `(2, 28)`, and put back the + README row and CHANGELOG item 5. + - **Candidates** (measure them, do not guess): + - (a) build in `quay.io/pypa/manylinux_2_28_s390x` (gcc-toolset-14) as aarch64 does. No + GitHub s390x runner exists: either emulate (slow) or check whether IBM's runner offer for + open-source projects applies; + - (b) cross-compile against a glibc 2.28 s390x sysroot (AlmaLinux 8) with a newer cross GCC. + - **libstdc++:** either way it needs the gcc-toolset treatment (newer parts linked statically, + `GLIBCXX` ≤ 3.4.25). + - **Acceptance:** on real LinuxONE hardware, cfarm191/192 (`workspacePrivate/cfarm`), with + RHEL 8 in Docker as the quick check. +- **Ruled out:** `ubuntu-22.04` as the runner (glibc 2.35 still misses RHEL 8/9, and its cross GCC 11 + cannot build llama.cpp, which needs GCC >= 12). `-static-libstdc++` alone is ruled out too: it + fixes GLIBCXX but not GLIBC. + ### Hardware and test-fixture coverage - **Real arm64 Android / Adreno OpenCL device:** run the AAR + Kotlin facade and LLM Service app