diff --git a/CLAUDE.md b/CLAUDE.md index e8fe53145..10a45e026 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1307,7 +1307,7 @@ Current patches: | `0014-common-log-callback-sink.patch` | **Gives `common_log` a callback sink: `common_log_set_callback(log, cb, user_data)` (`common/log.{h,cpp}`).** This is what `LlamaModel.setLogger` hooks. Before it, the Java logger was a `llama_log_set()` callback, which has two holes, both found while chasing `slot print_timing` lines interleaving with the Atmosphere agent's streamed answer: **(a)** every model load runs `common_init()`, which re-points `llama_log_set()` at `common_log_default_callback` (`common.cpp:394`), so `setLogger(…)` *before* `new LlamaModel(…)` silently lost the callback; **(b)** the server's own `SRV_*`/`SLT_*` macros are `LOG_INF` and write straight into `common_log`, which `llama_log_set()` never carried, so the per-request `slot …`/`srv …` lines could not be routed to Java at all (the reason `LlamaModelTest#testLogText/JSON` sat `@Disabled` for years). `common_log` upstream offers file, colors, prefix, timestamps, verbosity and JSONL but no hook. The patch adds one: while a callback is set, the worker thread hands every entry to it **instead of** printing to stdout/stderr (a `--log-file` still receives them); the callback gets the bare formatted message (no prefix/timestamp/colors) with the `ggml_log_callback` signature; swapping the callback pauses the worker first, so queued entries reach the *previous* sink (which is what makes `setLogger(format, null)` a synchronous drain). With the sink, `common_init()`'s `llama_log_set()` reset is harmless — it points at the default callback that feeds `common_log`, i.e. exactly the path into the sink — so the ordering problem (a) disappears without any re-install logic, and the `srv`/`slot` lines (b) arrive because they are `common_log` entries. `jllama.cpp`'s `setLogger` therefore sets `common_log_set_callback(common_log_main(), trampoline)` **plus** `llama_log_set(common_log_default_callback)` (so llama/ggml lines feed `common_log` even before the first load). Two consequences to know: the callback runs on `common_log`'s **worker thread**, a plain `std::thread` llama.cpp re-creates on every pause/resume and never attaches to the JVM — the trampoline attaches it **once per worker thread** as a daemon, named `jllama-log-worker`, and a `thread_local` guard detaches it when the thread exits (`get_jni_env_attaching`; `common_log::pause()` joins the worker, a normal exit on which TLS destructors run, and `JNI_OnUnload` pauses it before anything else). Until 5.2.x it attached and detached per line, which created a `java.lang.Thread` object per log line (13 lines of a failed load, 13 `Thread` objects -- `LlamaLoggerTest` pins the single attach now). Each call runs inside a `PushLocalFrame`/`PopLocalFrame` pair, because a thread that stays attached and has no Java frame would otherwise accumulate local references, `setLogger` must call `common_log_set_callback` **outside** `g_log_mutex`, because the pause joins the worker, which needs that mutex to read the callback, and `setLogger` callers are serialized by a **separate** `g_set_logger_mutex`: two unserialized swaps race on the worker's `std::thread` (one joins it while the other assigns a fresh thread over the still-joinable object = `std::terminate`, the whole JVM), which `LlamaLoggerTest#concurrentSetLoggerCallsDoNotRaceOnTheLogWorker` reproduced before the mutex existed. Two caveats the Javadoc carries: a caller must not hold a lock the *previous* callback needs (the drain runs it on the worker while the caller waits), and the verbosity threshold is process-wide and reset by **every** load (`common_params_parse` ends with `common_log_set_verbosity_thold(params.verbosity)`, default 3), so a load without `-lv` puts it back to 3 — a review assumed the opposite, and `LlamaLoggerTest#verbosityThresholdIsProcessWideAndEveryLoadSetsIt` now pins the measured behaviour. And the verbosity threshold applies *before* the sink: at the default (`3`) the Java logger sees errors, warnings and the server's INFO lines, while llama/ggml INFO lines (`common_log_get_verbosity` maps them to TRACE = 4) arrive only from `setLogVerbosity(4)` on — the same filtering the console gets, and a behaviour change for consumers who captured the unfiltered `llama_log_set()` stream before. **Runnable guards:** `src/test/cpp/test_common_log_callback.cpp` (6 tests over a private `common_log_init()` instance: delivery, bare text under prefix+timestamps, clear, swap-drains-to-old-sink, file kept, levels pass through) links the function on every platform, so a bump that drops the patch reds `C++ Tests` at link time; `LlamaLoggerTest` (model-free, needs only `libjllama`: a logger set before a deliberately failing load on a non-GGUF file sees the `srv … loading model` INFO line and llama's ERROR line, in TEXT and JSON) and the re-enabled `LlamaModelTest#testLogText/testLogJSON` plus `#testLoggerSetBeforeLoadSurvivesTheLoad` (vocab-only load) cover the Java side. Upstream-submittable ("common : add a callback sink to common_log for embedding hosts"); **not yet filed upstream**. Touches only `common/log.{h,cpp}`, which no other patch touches. **Refreshed at the b11401 bump** (#29895 adds a `colors` member next to the sink's members and a `get_colors()` next to `set_callback()`; only context moved, the `+`/`-` lines are unchanged, and upstream still has no hook -- it added `common_log_get_colors` only). **On a bump, check whether upstream added a hook of its own (grep `callback` in `common/log.h`) and, if so, DROP this patch and port `setLogger` to theirs rather than refreshing it.** | | `0015-rpc-embeddable-client-and-stoppable-server.patch` | **Makes ggml-rpc usable inside a JVM** (see "RPC backend" below). Upstream treats every client-side RPC problem as `GGML_ABORT` — a malformed endpoint, a server that is not running, a failed handshake — which in a JVM kills the application over a typo in `--rpc`. **(1)** The *registration* path reports failure instead: `rpc_dispatcher::start()` returns `false`, `try_get_dispatcher()` `nullptr`, `ggml_backend_rpc_get_device_count()` `0`, `ggml_backend_rpc_add_server()` `nullptr`, and `add_rpc_devices()` (`common/arg.cpp`) throws `std::invalid_argument` naming the server where it used to register `nullptr` and silently run without it. Every path *after* registration keeps the original contract through `get_dispatcher()`, which still aborts — a server that vanishes mid-inference stays fatal (TODO). **(2)** `add_server()`'s cache hit is re-checked (a stopped server used to pass registration and abort on the first tensor upload), and `ggml_backend_rpc_get_device_memory()` of a gone server returns 0/0 instead of aborting — RPC devices stay in ggml's process-wide registry forever, and `common_init()` queries the memory of **every** registered device at `-lv 4`, so a stopped server would otherwise take down an unrelated later load. **(3)** The server becomes stoppable: `ggml_backend_rpc_stop_server()` (disconnects the client being served, wakes `accept()` with a loopback connection — the one portable way) and `ggml_backend_rpc_server_listening()`; the loop now reaches its cleanup and frees its backends on every early return, but deliberately **not** `rpc_transport_shutdown()`, whose `WSACleanup()` would invalidate every other RPC socket in the process. **(4)** transport: `MSG_NOSIGNAL`/`SO_NOSIGPIPE`, `socket_t::shutdown()`, and the fds `connect()`/`create_server()`/`accept()` leaked on their error paths are closed. Touches `ggml/include/ggml-rpc.h`, `ggml/src/ggml-rpc/{ggml-rpc.cpp,transport.cpp,transport.h}` and one function in `common/arg.cpp` (`add_rpc_devices`, away from `0001`'s `common_params_parse` hunks). **Runnable guard: `src/test/cpp/test_rpc.cpp`** — links both new functions (a bump that drops the patch fails `C++ Tests` at link time on every platform) and drives registration, stop and the memory query over loopback. Upstream-submittable; **not yet filed upstream**. On a bump, check whether upstream grew its own stop function or non-aborting registration (grep `stop_server` / `GGML_ABORT("Failed to connect` in `ggml-rpc.cpp`). **Refreshed at the b11450 bump** (#26610, `-sm tensor` over RPC, ~800 lines in `ggml-rpc.cpp`, protocol 7 → 8): three hunks lost their context -- new dispatcher members beside `start()`, and new `get_proc_address` names before `return NULL` (the two new names now sit after `ggml_backend_comm_allreduce_tensor`); the `+`/`-` lines are unchanged. The server-to-server comm #26610 adds (a listener on `0.0.0.0`, an `accept()` a stop does not wake) is on file in `TODO.md`. | | `0017-ggml-cpu-x86-prefetch-byte-offset-and-msvc-type.patch` | **Makes `GGML_CPU_ALL_VARIANTS=ON` buildable with plain clang on Windows, and fixes a prefetch address that is wrong everywhere.** The four `_mm_prefetch` calls in `ggml/src/ggml-cpu/arch/x86/quants.c` (the SSSE3-without-AVX branch of `ggml_vec_dot_q4_0_q8_0`, and the only `_mm_prefetch` calls in the whole ggml tree) carry two independent defects; both are fixed by routing the address through one new `static inline` helper in the new header `ggml/src/ggml-cpu/arch/x86/prefetch.h`, which is included relatively, so **no CMake change is needed**. **(1) Type -- fatal only on a current clang:** MSVC declares the intrinsic as `void _mm_prefetch(char const *, int)`; clang's own `xmmintrin.h` defines a casting macro but only `#ifndef _MSC_VER`, so on a `*-windows-msvc` target clang leaves the declaration to Microsoft and a typed pointer is `-Wincompatible-pointer-types`. **clang 22 promoted that to an error by default** ([llvm-project #157364](https://github.com/llvm/llvm-project/pull/157364); the [22.1.0 release notes](https://releases.llvm.org/22.1.0/tools/clang/docs/ReleaseNotes.html) add that `-Wno-error=incompatible-pointer-types` downgrades it) -- *not* clang 16, which promoted only `-Wincompatible-FUNCTION-pointer-types`; an earlier version of this row said otherwise. Measured on the real file at `-msse4.2`, unpatched → patched: clang **20.1.8** 4 warnings → 0/0, clang **22.1.3** 4 errors → 0/0, clang **23.1.3** 4 errors → 0/0. Plain clang is exactly what `cmake/x64-windows-llvm.cmake` selects -- and plain clang is the only way to get all **14** CPU variants, because `clang-cl` makes CMake's `MSVC` true and the `if (NOT MSVC)` block in `ggml/src/CMakeLists.txt` then drops `zen4`, `cooperlake`, `sapphirerapids`, `ivybridge` and `piledriver`. **(2) Arithmetic:** `&x[ib] + sizeof(block_q4_0)` adds `sizeof()` *elements*, not bytes -- +324 and +1156 instead of +18 and +34 (the sizes `ggml-common.h` pins by `static_assert`), and +648 instead of +36 for the `2 *` pair. That half is wrong on every platform, including where the call compiles. **Measured:** no throughput difference (see the test-table row), so this is a correctness and portability fix, not a performance one. **Upstream-submittable and meant to be submitted.** **Why upstream's own build is green is measured, not an open question:** their `windows-cpu` release job builds exactly this with plain clang, `GGML_CPU_ALL_VARIANTS=ON` and no suppressing flag (`CMAKE_ARGS` carries only build-target toggles), and ships `ggml-cpu-sse42.dll` -- because the `windows-2025-vs2026` image has **LLVM 20.1.8** on `PATH`, where these are warnings. That is narrower than it looks: the same image also ships **clang 22.1.3** inside Visual Studio 2026 (`VC/Tools/Llvm/x64`), which *fails* on the unpatched file -- upstream reaches the working compiler only because `C:\Program Files/LLVM/bin` comes first on `PATH`. It is **not** a header difference: all three MSVC toolsets on that image (14.29.30133, 14.44.35207, 14.51.36231) declare it identically as `extern void _mm_prefetch(char const*_A, int _Sel);` -- that was the first hypothesis, and a one-off probe workflow (deleted again once it had answered) disproved it by printing the declaration from each toolset on the image. So this patch is **ahead of** upstream rather than diverging: their build breaks the moment the image ships a clang that makes the diagnostic fatal, and until then they compile four warnings and a prefetch address 18x/34x too far. **Runnable guard: `src/test/cpp/test_prefetch.cpp`.** On a bump, run `git grep -n _mm_prefetch ggml/src/ggml-cpu/arch/x86/quants.c` at the new tag: if the call sites already cast **before** the addition, **drop this patch rather than refreshing it**. | -| `0016-model-kolibri1.patch` | **Adds Aleph Alpha's Kolibri-1 (architecture `kolibri1`) ahead of upstream** -- a 78B German/English reasoning MoE (3.46B active) that upstream does not support yet ([ggml-org/llama.cpp#29922](https://github.com/ggml-org/llama.cpp/issues/29922), no PR at the time). A **temporary carry, not a fix**: drop it the moment upstream registers the architecture (`git grep -n kolibri src/llama-arch.cpp` at the new tag), and keep `test_kolibri1.cpp` -- its numerical comparisons must then pass against upstream's implementation, while its GGUF-format rows follow whatever format upstream's converter fixes (see the test-table row). **Architecture** (Aleph Alpha's vLLM reference, `aleph_alpha_inference/kolibri1.py` at `049a6a7`): GQA with per-head q/k RMSNorm; `layer_types` interleaves sliding-window layers (NEOX RoPE) with full-attention layers that have **no** positional encoding; sandwich norms around attention and MoE; every layer routed MoE + one ungated shared expert; the router selects top-k on **`logits + expert_bias`** and weights by the **unbiased `sigmoid(logits)`** -- not DeepSeek-V3's sigmoid router (selection on `sigmoid(logits) + bias`), which picks other experts as soon as the bias is non-zero. The graph therefore builds the selection itself and passes it to `build_moe_ffn` (`selected_experts_in`), leaving upstream's shared MoE code untouched. **Derived from two community ports, which are incompatible with each other's GGUFs** -- Eliasfpv28's AFMoE-based one (base b11378, `expert_gating_func` 2, pre-tokenizer `qwen2`) and the Qwen3-MoE-based one behind Hob-forge's GGUFs (base b11381, as carried in [mjwsolo/localcode#101](https://github.com/mjwsolo/localcode/pull/101): gating 5, pre-tokenizer `kolibri1`, rejects gating 2). This patch **loads both dialects**: gating 2, 5 or absent (all mean the one router), `kolibri1` mapped to the Qwen2 pre-tokenizer in `llama-vocab.cpp`, `expert_weights_norm` / `expert_shared_count` / `output.weight` optional. Every source, with license, is listed in the patch header; `REUSE.toml` annotates the file `MIT AND Apache-2.0` (Eliasfpv28's additions are Apache-2.0). Touches `src/llama-arch.{h,cpp}`, `src/llama-model.cpp` (mapping + NEOX rope list), `src/llama-vocab.cpp` (one name), `src/models/models.h` and the new `src/models/kolibri1.cpp`; no converter. **Runnable guard: `src/test/cpp/test_kolibri1.cpp`** (writes tiny random GGUFs in both dialects and compares every logit with an independent double-precision reference). **Not verified here:** the real 78B model (HuggingFace is unreachable from the sessions this was written in, and the smallest GGUF is 28.6 GB) and GPU backends -- the community ports report real-model checks (top-1 67/67 against a float64 reference for Q8_0, tool calls, German text). | +| `0016-model-kolibri1.patch` | **Adds Aleph Alpha's Kolibri-1 (architecture `kolibri1`) ahead of upstream** -- a 78B German/English reasoning MoE (3.46B active) that upstream does not support yet ([ggml-org/llama.cpp#29922](https://github.com/ggml-org/llama.cpp/issues/29922), no PR at the time). A **temporary carry, not a fix**: drop it the moment upstream registers the architecture (`git grep -n kolibri src/llama-arch.cpp` at the new tag), and keep `test_kolibri1.cpp` -- its numerical comparisons must then pass against upstream's implementation, while its GGUF-format rows follow whatever format upstream's converter fixes (see the test-table row). **Architecture** (Aleph Alpha's vLLM reference, `aleph_alpha_inference/kolibri1.py` at `049a6a7`): GQA with per-head q/k RMSNorm; `layer_types` interleaves sliding-window layers (NEOX RoPE) with full-attention layers that have **no** positional encoding; sandwich norms around attention and MoE; every layer routed MoE + one ungated shared expert; the router selects top-k on **`logits + expert_bias`** and weights by the **unbiased `sigmoid(logits)`** -- not DeepSeek-V3's sigmoid router (selection on `sigmoid(logits) + bias`), which picks other experts as soon as the bias is non-zero. The graph therefore builds the selection itself and passes it to `build_moe_ffn` (`selected_experts_in`), leaving upstream's shared MoE code untouched. **Derived from two community ports, which are incompatible with each other's GGUFs** -- Eliasfpv28's AFMoE-based one (base b11378, `expert_gating_func` 2, pre-tokenizer `qwen2`) and the Qwen3-MoE-based one behind Hob-forge's GGUFs (base b11381, as carried in [mjwsolo/localcode#101](https://github.com/mjwsolo/localcode/pull/101): gating 5, pre-tokenizer `kolibri1`, rejects gating 2). This patch **loads both dialects**: gating 2, 5 or absent (all mean the one router), `kolibri1` mapped to the Qwen2 pre-tokenizer in `llama-vocab.cpp`, `expert_weights_norm` / `expert_shared_count` / `output.weight` optional. Every source, with license, is listed in the patch header; `REUSE.toml` annotates the file `MIT AND Apache-2.0` (Eliasfpv28's additions are Apache-2.0). Touches `src/llama-arch.{h,cpp}`, `src/llama-model.cpp` (mapping + NEOX rope list), `src/llama-vocab.cpp` (one name), `src/models/models.h` and the new `src/models/kolibri1.cpp`; no converter. **Runnable guard: `src/test/cpp/test_kolibri1.cpp`** (writes tiny random GGUFs in both dialects and compares every logit with an independent double-precision reference). **Verified against the real 78B model on 2026-10-11** (rented 16-vCPU / 123 GB instance, both published dialects fetched on the machine): both load, and a chat turn through the model's own template answers correctly in each -- `content` `"Berlin."` for a German question and `"51"` for 17x3, with the chain of thought in `reasoning_content` and `finish_reason` `stop`. **The two dialects tokenize identically** (`[452 22090 493 1678 2459 5513 46]`, round-tripping to the input sentence), which is the direct evidence that mapping `kolibri1` to the Qwen2 pre-tokenizer is right. Q4_K_M (44 GB) gave 13.4 t/s and Q3_K_S (31 GB) 14.6 t/s of generation on 16 cores. **Two traps for whoever repeats this:** a *raw* completion on this instruct model degenerates into repetition whatever the patch does, so measure through the chat endpoint; and `chatCompleteText` returns only `content`, which is **empty** when a reasoning model spends its token budget inside the thinking block -- use `handleChatCompletions` and a budget in the hundreds. **Still not verified:** GPU backends -- the community ports report real-model checks (top-1 67/67 against a float64 reference for Q8_0, tool calls, German text). | | `0018-ggml-amx-free-the-aligned-buffer.patch` | **The AMX weight buffer is freed with `free()` but allocated with `_aligned_malloc()` on Windows -- `STATUS_HEAP_CORRUPTION` (0xC0000374) the moment a model is unloaded.** `ggml_backend_amx_buffer_type_alloc_buffer()` (`ggml/src/ggml-cpu/amx/amx.cpp`) allocates with `ggml_aligned_malloc()`, which is `_aligned_malloc()` under `_MSC_VER` (plain clang on a `*-windows-msvc` target defines it too); `ggml_backend_amx_buffer_free_buffer()` called `free()` on it. Fine on Linux (`posix_memalign` + `free`), fatal on Windows, and only there. The patch frees with `ggml_aligned_free(buffer->context, buffer->size)`, exactly what `ggml_backend_cpu_buffer_free_buffer()` does; on Linux that is `free()`, so nothing changes there. **Reached by** the `sapphirerapids` CPU module (the only variant built with `AMX_TILE AMX_INT8`, which `ggml_backend_load_best` picks on Sapphire/Emerald/Granite Rapids; `ggml_amx_init()` is `return true` on `_WIN32`) and a model whose weights have AMX kernels (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS -- i.e. every CI model). **Found by the second full run of the modular natives** (38069595101): `Java Tests Windows 2025 x86_64 (clang)` landed on an `Intel(R) Xeon(R) 6973P-C` runner (Granite Rapids), loaded `ggml-cpu-sapphirerapids.dll`, ran `GrammarExample`'s two generations fine and died with exit code -1073740940 at `LlamaModel.close()`; the first run (38065017595) had the same job green on an AMD EPYC 7763 (no AVX-512, `haswell`). **The Windows runners are a CPU lottery**, so which CPU module a Windows job exercises differs per run; before the modular natives the Windows jar was a static Haswell-level build and this code never ran. **Of the six callers of `ggml_aligned_malloc` in the tree this was the only mismatched pair** (`ggml-backend.cpp`, `ggml-cpu.c` twice, `ggml-openvino.cpp`, `ggml-zdnn.cpp` and `ggml.c` all pair correctly), and `ggml-backend.cpp:2464` is the line the fix copies. **Reproduced deliberately rather than waiting for the lottery again**: on a rented Granite Rapids machine, one CPU module staged at a time with the same model and the DLLs CI itself built, `sapphirerapids` crashed **6 of 6** runs -- including at **one** thread, which rules out a data race -- while `cooperlake`, `icelake` and `haswell` were clean 6 of 6, and on Linux with the same CPU all six exit 0 with byte-identical output. The crash lands *after* the inference prints its timings, in cleanup: the signature of a heap corrupted earlier and detected at `free`. With the patch applied the same matrix on the same machine is **clean in all twelve runs**. **The guard is a configure-time check, not a `jllama_test` case**: the AMX code exists only in that one variant of a `GGML_CPU_ALL_VARIANTS` build and executes only on AMX hardware, which the static test build, the local agent's Zen 3 and the cloud sandboxes lack -- so **`llama/cmake/check-aligned-free-pairing.cmake`** fails the configure on **every** platform when a ggml source calls `ggml_aligned_malloc` and never `ggml_aligned_free` (verified red against the unpatched source, naming `amx.cpp`, and green with this patch). A run-time test cannot replace it: without an AMX CPU there is nothing to observe, which is exactly how this shipped. What exercises it: the Windows x86-64 Java test job and the `windows-x86-64` smoke row whenever GitHub hands them an Intel AMX runner; the patch itself is held by `check-patches.py`, applies at b11538, and the patched file compiles with the variant's flags (GCC 13, `-mamx-tile -mamx-int8 -mavx512vnni ...`). Upstream master still carried the `free()` on 2026-10-10; **not yet filed upstream** (nothing before the release). **Drop-check on every bump:** `grep -n "free(buffer->context)" ggml/src/ggml-cpu/amx/amx.cpp` at the new tag -- gone means upstream fixed it, drop the patch. | **Dropped patches** (`0004`, `0005`, `0009`, `0010`, `0011`, `0013` -- upstream fixed each defect diff --git a/TODO.md b/TODO.md index ced4ea6c4..b374b36dd 100644 --- a/TODO.md +++ b/TODO.md @@ -441,137 +441,19 @@ machine are the **interactive agent** (console behaviour needs a human at a real below; it is additionally backed by a GCC compile of the patched file with the variant's own flags, `-mamx-tile -mamx-int8 -mavx512vnni`). What stays open is the original question, the *throughput* gain of `sapphirerapids` and `zen4` over `haswell`, which needs the same hardware but no longer blocks anything. -- **Kolibri-1 on the real 78B model.** `patches/0016` is verified only against tiny synthesized GGUFs - (`test_kolibri1.cpp`); CLAUDE.md's own row says "Not verified here: the real 78B model ... and GPU - backends". **The RAM objection is measured and wrong for the small quants:** the smallest published - GGUF of each dialect is **Q2_K at 26.7 GB** (gating 5, pre-tokenizer `kolibri1`) and **Q3_K_S at - 31.5 GB** (gating 2, `qwen2`), both of which fit in 63 GB without paging -- only Q6_K (59.8 GB) does - not. What is open is the run, on the published set: both dialects must load, a German prompt must - come back coherent (the router selects on `logits + expert_bias` but weights by the *unbiased* - `sigmoid(logits)`, and a DeepSeek-V3-style router produces fluent-looking nonsense instead), a - thinking block, a tool call through the chat endpoint with `--jinja`, tokens/s and peak RAM. A load - error or gibberish is a finding **against** `0016`, not something to work around. - **In flight since 2026-10-11:** a single rented run on an `r7i.4xlarge` (16 vCPU, 128 GB RAM, - ~1.06 USD/h, dead-man switch at 150 min) loads **both** dialects in one boot, downloading each GGUF - straight from HuggingFace on the instance -- `Hob-forge/Kolibri-1-GGUF` **Q4_K_M (44.2 GB)** rather - than its Q2_K, because the question is whether the German answer is *coherent* and a 78B MoE at - Q2_K would make that ambiguous, and `Eliasfpv28/Kolibri-1-Q3_K_S-GGUF` **Q3_K_S (31.5 GB)** for the - gating-2 dialect. Hob-forge publishes Q2_K through Q8_0, so quant choice is free; 128 GB fits even - Q6_K (59.8 GB), which no local machine does. The probe prints the architecture, expert and gating - lines from the load at verbosity 4, then the generated **text** of two raw completions and one chat - turn through the model's own template, each guarded separately so one failure still leaves the rest - measured. - -### Compatibility matrix on rented hardware (one measurement per generation) - -The project ships **14 x86-64 CPU modules and 8 aarch64 ones**, of which `ggml_backend_load_best` -keeps exactly one -- the best for the running CPU. So a module is only ever exercised *as the chosen -backend* on a machine whose CPU prefers it, and no amount of running the suite on one machine or one -CI runner changes that. Measured coverage today is **8 of 14 x86 levels** (six on a Zen 3 -workstation, `skylakex` and `cascadelake` added by a Cascade Lake machine) and **0 of 8 aarch64 -levels**. - -Rented cloud instances can close most of that, one short run each, and it is a *per-generation -measurement* -- not something CI should repeat. Each family maps to exactly one module: - -| instance family | CPU | module it exercises | -|---|---|---| -| `m8i` | Granite Rapids | **`sapphirerapids`** (AMX) | -| `m7i` / `c7i` | Sapphire Rapids | `sapphirerapids` (AMX) | -| `m6i` / `c6i` | Ice Lake-SP | `icelake` | -| `m7a` / `c7a` | EPYC Genoa (Zen 4) | **`zen4`** | -| `m5n` / `c5n` | Cascade Lake | `cascadelake` | -| `m5` / `c5` | Skylake-SP | `skylakex` | -| `m6a` / `c6a` | EPYC Milan (Zen 3) | `haswell` | -| `m4` and older | Broadwell / Ivy Bridge | `haswell`, `ivybridge` | -| **`m7g` / `c8g` (Graviton)** | **aarch64** | **done: 6 of the 8 run on a Graviton 4** (`armv9.2_*` need SME, which no instance has) | - -**Four levels stay unreachable there** and are only ever *load*-tested: `piledriver` (no -Bulldozer-era instance exists), `cannonlake` (barely shipped), `alderlake` (consumer Intel) and -`cooperlake`. - -**Graviton was the biggest gain per euro, and it has been taken**: 6 of the 8 aarch64 modules are now -measured with real inference on a Graviton 4 (`docs/local-test-plan.md` has the table). -`armv9.2_1`/`armv9.2_2` need **SME**, which no AWS instance has, so they stay unmeasured. Two things -came out of it: the two SME modules decline *cleanly* (no CPU backend, `LlamaException`, not a -crash), and **ggml picks a module that is 37 % slower at prompt processing than a lower one** -- -`armv8.6_2` at 198 t/s against `armv8.2_2` at 271 t/s on the same machine and model. That is a -scoring question for upstream and invisible to anyone who only runs what the CPU picks. - -GPU backends, same idea, with two that do not work out: - -| backend | rentable | -|---|---| -| CUDA, Vulkan, OpenCL | yes -- T4 / A10G / L4 class instances are enough | -| OpenVINO | yes (CPU) | -| **ROCm** | **no** -- the one AMD-GPU instance family is not an officially supported ROCm target | -| **SYCL** | **no** -- needs an Intel GPU, which is not offered | - -**How to run these, and the invariants that came out of doing it, are in -[`docs/rented-hardware-test-plan.md`](docs/rented-hardware-test-plan.md)** -- including the one that -costs days when missed: a GPU instance family starts at a **zero vCPU quota** on a fresh account, and -raising it is a support request measured in hours to days, so ask before the run is due. The large -accelerator families (8-GPU training instances) prove nothing here that a single small GPU does not, -and cost two orders of magnitude more. - -**The AMX row is closed.** It was a suspected defect, turned out to be one line of upstream ggml -(the AMX buffer released with `free()` instead of `ggml_aligned_free`), and is fixed as -`patches/0018` -- proven on a rented Granite Rapids machine (`sapphirerapids` 6/6 crash against 0/6 -for the other modules, Linux unaffected) and verified there again with the patch (12/12 clean). The -full record is in the patch header, `CLAUDE.md`'s patch table, `CHANGELOG.md` and the module table in -`docs/local-test-plan.md`; `llama/cmake/check-aligned-free-pairing.cmake` is the guard. **What stays -open is only the original question** -- the *throughput* gain of `sapphirerapids` and `zen4` over -`haswell` -- plus one rule worth keeping: a green Windows job proves nothing unless its log names an -AMX CPU, which is why `print-host-info.sh` now runs in every job that touches native code. - -### AIX on POWER -- an open porting question, not a test - -Not supported and never built. Worth a line because the access exists (an AIX 7.3 POWER8 machine on -the GCC Compile Farm) and because it would be the **second big-endian platform** after s390x -- -which is exactly the coverage s390x was added for, so the endian-sensitive surface of *our* layer is -already guarded by `ctest` under qemu there. - -What makes it a port rather than a run: AIX is big-endian POWER with **XCOFF**, not ELF, so the -loader's assumptions (`$ORIGIN`-style run paths, `dlopen` semantics, `verify-native-deps.py`'s ELF -reader) do not carry over; upstream llama.cpp does not target AIX; and a JVM there means IBM Semeru, -not Temurin. **Nothing is scheduled**: the useful first step is a plain `cmake` build of the static -library on that machine to see how far it gets and what breaks, with the result written down -- -before anything is promised about a natives jar. - -### GraalVM Native Image -- evaluated, parked until a consumer asks - -The evaluation is written down in -[`docs/graalvm-native-image-evaluation.md`](docs/graalvm-native-image-evaluation.md): what the -library needs from Native Image (a closed JNI surface, resource or sidecar natives, no reflection of -its own), the two decisions a real attempt has to take, the cost (a second build matrix and metadata -that rots silently without an image-build-plus-smoke job), and the one measurement that decides it: -the share of JVM start-up in a cold one-token run, which needs a model and therefore the local agent -("Needs a machine" above). Nothing to do here until that number or a consumer request exists. - -### macOS model-backed server smoke (optional depth, not scheduled) - -`smoke-natives-macos` verifies the dylib inside the published `metal-macos-aarch64` jar (code -signature + a real JVM load and JNI round-trip, plus the HTTPS round over the 1 MB model it downloads; -green as `smoke-fatjar-macos` in CI since run 37985786772). A full model-backed macOS server smoke as -Linux and Windows run would need `verify-model-cache` plus a cache restore and -`smoke-test-natives.sh` against the `macos-aarch64` smoke set. Worth doing only if a macOS-specific -*inference* regression ever appears; the load-time failure class is covered. - -### Test-coverage gaps found by the b10679 mutation audit (PR #403) - -> **Update.** The `IdleSleepWakeIntegrationTest` added to close the `wake_and_post` gap immediately -> found a real JVM crash (SIGSEGV on all six CI platforms) — see the CHANGELOG "Fixed" entry. Both -> facets are fixed in that PR via the `wake_server()` choke point. This is the clearest evidence for -> the floor on executed tests that `.github/verify-test-counts.sh` enforces since: the defect had been -> reachable from public API for as long as `--sleep-idle-seconds` has existed, and nothing ran that path. - -A mutation pass over the branch applied 27 mutations and 26 went red on the test that claims them, -so no test here passes with its subject deleted. What it did find is code with **no runnable guard**. -Two of the three were closed in that PR (a model-free `jsonSchemaToGrammar` test in -`NativeLibraryLoadSmokeTest`, and `IdleSleepWakeIntegrationTest` for the `wake_and_post` path); the -third was `patches/0010`'s `(int)` cast, reachable only from a model-gated Java test, and it went -away with the patch itself at the b11080 bump. This is what remains. - +- **Kolibri-1 on the real 78B model: done, `patches/0016` holds.** Measured 2026-10-11 on a rented + 16-vCPU / 123 GB instance, both published dialects fetched from Hugging Face on the machine at + 80-94 MB/s. **Both load and both answer correctly** through a chat turn on the model's own template: + `content` `"Berlin."` for a German question and `"51"` for 17x3, the chain of thought in + `reasoning_content`, `finish_reason` `stop`. Generation 13.4 t/s for Q4_K_M (44 GB) and 14.6 for + Q3_K_S (31 GB). **The strongest single piece of evidence is the tokenizer**: both dialects produce + the *same* token ids (`[452 22090 493 1678 2459 5513 46]`) and round-trip to the input sentence, so + mapping `kolibri1` to the Qwen2 pre-tokenizer is demonstrated rather than assumed -- and a broken + router would not have produced two correct answers in two quantizations. What is still unverified is + **GPU backends** (the patch's own caveat) -- the first run of this measurement was wrong in a way + worth keeping on file: it used a raw completion at temperature 0.0, which loops on an instruct model + regardless, and read `chatCompleteText`, which returns only `content` and is therefore *empty* when a + reasoning model spends its budget inside the thinking block. Both failures looked like the patch. - **`TestConstantsTest.theShippedModelConstantsGoThroughTheResolver` is vacuous when the fixture is absent.** Mutating `MODEL_PATH = resolveModelPath("models/…")` to the bare literal leaves the test green with `models/` empty, and only goes red once the GGUF actually exists. In CI that is the