From 5108b4d29c24db0064749d29c4a5f57e966d0110 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 20:27:10 +0000 Subject: [PATCH 1/3] Upgrade llama.cpp from b11815 to b11820 Five upstream commits (d77256999 = b11820). One compile break, in jllama_engine.cpp: format_prompt_rerank() takes the decoded images of the query and the document since #29625 ("server : support typed content input for RERANK endpoint"); engine::rerank passes two empty lists, which is the old text-only prompt. The rerank route reads the query and each document as a string or {"text": string or [strings], "image": ...} now. OaiRerankSupport follows the text half: an object query is accepted, a text array concatenated, and an item with an image is a 400 naming NativeServer instead of being ranked by its text alone (an object document's other keys used to be ignored); video/fps/max_frames get upstream's message. Six new cases in OaiRerankSupportTest, red before the change. readTopN's Javadoc says that upstream refuses a negative top_n too now. Also: Nemotron with a wider Q projection, the domyn-small pre-tokenizer and a tokenizer.ggml.metaspace.prepend_scheme key (#23343); posix_memalign in the Vulkan buffers (#30828); two CI-only commits. All 12 patches apply unchanged (check-patches.py: 88 hunks, 0 problems). The drop-checks of 0002, 0003 and 0021, whose file server-context.cpp is in the range, still find the defects. The server request fields, their bounds and the response keys are identical. Verified: fresh configure and build, ctest 686/686, NativeLibraryLoadSmokeTest 7/7 against the b11820 library, OaiRerankSupportTest 30/30, OpenAiCompatServerHttpTest 81/81, spotless and spotbugs clean. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VBFFnrhtvXiUeyxjmUc4R2 --- CHANGELOG.md | 14 +++- CLAUDE.md | 10 +-- README.md | 4 +- TODO.md | 12 ++- docs/history/llama-cpp-breaking-changes.md | 2 + llama/CMakeLists.txt | 4 +- llama/src/main/cpp/jllama_engine.cpp | 8 +- .../llama/server/OaiRerankSupport.java | 79 +++++++++++++++---- .../llama/value/LlamaCppVersion.java | 8 +- .../llama/server/OaiRerankSupportTest.java | 53 +++++++++++++ 10 files changed, 157 insertions(+), 37 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a2696dd..7adfd456 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -136,7 +136,7 @@ the first two items break every existing consumer at upgrade. fails the stream through `onError`. ### Added -- **llama.cpp b11815** (from b11639, in eight steps on the branch: b11760, b11761, b11799, b11800, b11803, b11809, b11814, b11815). What the +- **llama.cpp b11820** (from b11639, in nine steps: b11760, b11761, b11799, b11800, b11803, b11809, b11814, b11815, b11820). What the range brings to a consumer: `POST /tts` (llama.cpp b11781, #26603, speech from a TTS-capable model through the embedded server) and the realtime transcription WebSocket `/v1/realtime` (b11780, #30636), both served by `NativeServer` -- classic and attach mode -- and not by the Java transport or `LlamaModel` yet; a @@ -157,6 +157,12 @@ the first two items break every existing consumer at upgrade. pre-tokenizer (#30960), KleidiAI v1.32.0 with SVE Q8_0 kernels (#30808, not built here) -- nothing here changed. The b11815 step (2026-10-22, one commit): the SYCL `CONV_2D` kernel accepts the channel-fastest CWHN layout instead of aborting (#30363), which reaches the two SYCL module jars. + The b11820 step (2026-10-22): llama.cpp's rerank route takes typed items (#29625) -- the query and each + document a string or `{"text": ..., "image": ...}`, the images ranked through the projector -- which + `NativeServer` serves; `OpenAiCompatServer` reads the text half (see "Reranking" under Fixed) and + `LlamaModel.rerank` stays text-only. Also Nemotron models with more Q heads than `n_embd / head_dim`, + the `domyn-small` pre-tokenizer and a `tokenizer.ggml.metaspace.prepend_scheme` GGUF key (#23343), and + `posix_memalign` in the Vulkan buffers (#30828). Two patches dropped (`0016`, `0019`), `0007` refreshed. - **`RouterModel.Status.INCOMPLETE`** (llama.cpp b11714): a router download that never finished is listed as `"incomplete"` without a worker; it was `UNKNOWN` here. @@ -1136,7 +1142,11 @@ the first two items break every existing consumer at upgrade. negative one is `400` in both spellings; `"documents": null` next to `texts` is served from `texts`; and a native answer that is not a JSON array is a `500` with the reason instead of an empty result list. `LlamaModel.rerank(String, String...)` returns its map in the order of the - documents instead of hash order. + documents instead of hash order. The query and each document are read as llama.cpp reads them since + b11820 (#29625): a string or `{"text": string or [strings]}` (an object query is accepted, a text array + concatenated), and an item with an `image` -- which upstream ranks through the projector -- is `400` + instead of being ranked by its text alone; `video` / `fps` / `max_frames` are refused with upstream's + message. `NativeServer` serves upstream's route, images included. - **Java API.** - **`InferenceParameters.withMessages(List)` keeps `tool_calls` and `tool_call_id`**, which only `ChatRequest` added, so a tool round built through it sent an assistant turn without diff --git a/CLAUDE.md b/CLAUDE.md index fa49f879..0bd3fad0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b11815** +Current llama.cpp pinned version: **b11820** ## Natives jars: library jars and GPU module jars (`.github/natives.csv`) @@ -213,7 +213,7 @@ measurements behind the Windows step are in `docs/handover/local-agent-report-b1 a `llama-common`/`cpp-httplib` that came out shared (one more file to ship). On the static platforms (macOS, s390x) all of it stays inside `libjllama`. **Every ggml default llama.cpp's own CMakeLists changes before its `add_subdirectory(ggml)` is repeated** -- `GGML_LLAMAFILE_DEFAULT` - and `GGML_CUDA_GRAPHS_DEFAULT` at b11815 (unchanged since b11639) -- because adding ggml first skips that block; re-check with + and `GGML_CUDA_GRAPHS_DEFAULT` at b11820 (unchanged since b11639) -- because adding ggml first skips that block; re-check with `grep -n _DEFAULT CMakeLists.txt` at the new tag on every bump. The second one was missing once: ggml's own default for CUDA graphs is OFF, and a CUDA module built without it lost two thirds of its token generation (265.6 -> 88.8 tg128 on an RTX 3070, prompt processing -20%; Vulkan, which @@ -1183,7 +1183,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b11815 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b11820 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -1223,7 +1223,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`JLLAMA_GIT_TAG b11815`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`JLLAMA_GIT_TAG b11820`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -2671,7 +2671,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `JLLAMA_GIT_TAG b11815`. +llama.cpp is fetched via CMake FetchContent, pinned to `JLLAMA_GIT_TAG b11820`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.18.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 92a28d7d..25ed8abe 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b11815](https://img.shields.io/badge/llama.cpp-%23b11815-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11815) +[![llama.cpp b11820](https://img.shields.io/badge/llama.cpp-%23b11820-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11820) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.1%20%40NullMarked-25A162)](https://jspecify.dev) @@ -1015,7 +1015,7 @@ built-in WebUI" below). It serves: | `POST /v1/chat/completions` | `LlamaModel.streamChatCompletion` (streaming SSE) / `chatComplete` (blocking) | | `POST /v1/completions` | `LlamaModel.streamCompletionsOai` (streaming SSE) / `handleCompletionsOai` (blocking) | | `POST /v1/embeddings` (requires `--embedding`) | `LlamaModel.handleEmbeddings` | -| `POST /v1/rerank` (requires `--reranking`) | `LlamaModel.handleRerank` (reshaped to `results`/`data` with `usage`; a request with TEI's `texts` is answered as TEI's bare `[{index, score[, text]}]` array, `text` on `return_text`, as upstream) | +| `POST /v1/rerank` (requires `--reranking`) | `LlamaModel.handleRerank` (reshaped to `results`/`data` with `usage`; a request with TEI's `texts` is answered as TEI's bare `[{index, score[, text]}]` array, `text` on `return_text`, as upstream; the query and each document are a string or `{"text": string or [strings]}`, as upstream reads them since llama.cpp b11820, and one with an `image` -- which upstream ranks through the projector -- is a `400` rather than its text alone; `NativeServer` serves upstream's route with images) | | `POST /infill` | `LlamaModel.handleInfill` (fill-in-the-middle autocomplete) | | `GET /v1/models` | the configured model id | | `GET /metrics` | native server and per-slot token/cache counters (JSON) | diff --git a/TODO.md b/TODO.md index a175992b..d8bf92be 100644 --- a/TODO.md +++ b/TODO.md @@ -575,8 +575,8 @@ stays a no-op -- callers repeat the stop until the serve returns, as `RpcServer` The bump itself is done (b11639 -> b11760 -> b11761 -> b11799 -> b11800, b11803 on 2026-10-21: Flint, SenseNova U1 and a jinja fix, nothing to wire; b11809 on 2026-10-22: LoRA on prism.hadamard folded weights, an OpenCL ARGMAX kernel, nothing to wire; b11814 on 2026-10-22: the `tokenizer.ggml.clean_spaces` key and the Pixtral -pre-tokenizer change, nothing to wire; b11815 on 2026-10-22: a SYCL CONV_2D layout fix, nothing to wire; patches `0016` and `0019` dropped, `0007` refreshed; -`docs/history/llama-cpp-breaking-changes.md` has the rows). Four things the range brings that this +pre-tokenizer change, nothing to wire; b11815 on 2026-10-22: a SYCL CONV_2D layout fix, nothing to wire; b11820 on 2026-10-22: typed rerank items, `OpenAiCompatServer` reads their text and refuses images; patches `0016` and `0019` dropped, `0007` refreshed; +`docs/history/llama-cpp-breaking-changes.md` has the rows). Five things the range brings that this library does not expose, each a decision rather than an omission: - **Server-side TTS through `LlamaModel` / the Java transport.** `POST /tts` (b11781, #26603) runs @@ -586,6 +586,14 @@ library does not expose, each a decision rather than an omission: `/v1/audio/speech`-shaped route on `OpenAiCompatServer` would make it reachable without the native frontend; `TextToSpeech` (its own context over `mtmd_helper::gen_audio`) stays the one-shot path. Not before the release. +- **Multimodal rerank through `LlamaModel` / the Java transport.** Since b11820 (#29625) upstream's rerank + route takes `{"text": ..., "image": ...}` items and splices the images through `process_mtmd_prompt` + (`format_prompt_rerank(..., query_files, doc_files, ...)`, `handle_media()` for the decoding); `NativeServer` + serves it. `engine::rerank` passes empty file lists, `jllama_rerank` and `LlamaModel.rerank` take strings, + and `OpenAiCompatServer` refuses an `image` with a `400` naming `NativeServer` rather than ranking the text + alone. Wiring it means items in the C API (text plus data URIs, as `jllama_completion`'s multimodal prompt + already carries) and a typed overload on `LlamaModel`; it needs a multimodal reranker in the CI model set to + be tested. Not before the release. - **Realtime transcription** (`/v1/realtime`, WebSocket, b11780, #30636: `mtmd_audio_stream`, `SERVER_TASK_TYPE_TRANSCRIBE_RT`, `SERVER_CONTROL_ACTION_AUDIO_*`). Served by `NativeServer`; the JDK's `com.sun.net.httpserver` has no WebSocket, so the Java transport cannot follow without a dependency. Not diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 341eaec5..11d6bfc6 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -911,3 +911,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11809–b11814 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `d88040c70`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems). The range touches two files under `tests/` (`tests/CMakeLists.txt`, `tests/test-tokenizer-0.cpp`), neither of them a `0001` target (its `tests/` hunks are `test-arg-parser.cpp`, `test-export-graph-ops.cpp` and `test-thread-safety.cpp`), and no other patch-target file, so the drop-check answers of the b11803 row stand unchanged. | | b11814–b11815 | `ggml/src/ggml-sycl/conv2d.cpp`, `ggml/src/ggml-sycl/ggml-sycl.cpp`, `docs/ops.md`, `docs/ops/SYCL.csv` (one commit, `7170cad4d` = b11815, 2026-10-22) | **One SYCL kernel fix, no source change here.** `CONV_2D` on the SYCL backend (#30363) supports the channel-fastest CWHN layout (input `[N][H][W][C]`, kernel `[KH][KW][IC][OC]`) next to the contiguous WHCN one -- the kernel is templated on the layout, `supports_op` accepts a non-contiguous input or kernel whose channel stride is the element size, and the former `GGML_ASSERT(p[6] == 0)` ("no cwhn layout support") is gone; the regenerated `docs/ops/SYCL.csv` (7,793 insertions, 6,047 deletions, 2.3 MB of diff) is what pushed this one-commit chunk over the runbook's 100 KiB threshold on its own, so it is a chunk of its own. The SYCL module jars are built in CI only (`sycl-linux-x86-64`, `sycl-windows-x86-64`); the change reaches a consumer running a convolution (a vision projector, Whisper-style audio input) on an Intel GPU with a CWHN tensor, which used to abort. Nothing in `common/`, `include/`, `tools/server/`, `tools/mtmd/`, `src/`, the root `CMakeLists.txt` (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` unchanged) or upstream's `.github/`; the server request-field set, its bounds and the response keys are identical (`tools/server/` is not in the range). **Verified at b11815**: `jllama` and `jllama_test` built from the pin in a fresh tree, `ctest` **686/686** (665 `TEST` macros), `NativeLibraryLoadSmokeTest` against the b11815 library. | | b11814–b11815 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `7170cad4d`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems; and `git apply --check` + `git apply` in filename order on a pristine b11815 worktree, every one clean, the 48 paths the patches touch equal to the `diff --git` headers). **Across the whole b11803..b11815 range no patch-target file changed**: the 48 touched paths are the same blobs at both tags (`git rev-parse :`), and the intersection with `git diff --name-only b11803..b11815` (59 files) is empty. The drop-checks, re-read at b11815, give the b11803 answers: `0001` -- `argv = utf8.ptrs.data()` at `common/arg.cpp:1288` inside the count guard, no `common_params_parse_main` anywhere in the tree (75 `common_params_parse(` call sites at both tags, none new); `0002` -- `params_base.load_progress_callback = load_progress_callback;` unguarded at `server-context.cpp:1283`; `0006`/`0007` -- `tools/server/server.cpp` is the same blob, the helper body still byte-identical to the route-table block it replaces; `0008` -- `get_server_exec_path()` at `server-models.cpp:575`, no `LLAMA_SERVER_WORKER_CMD`; `0014` -- `common/log.h` names only `common_log_default_callback`; `0015` -- `GGML_ABORT("Failed to connect` at `ggml-rpc.cpp:650`, no `stop_server`, `RPC_PROTO` 8/0/1 and `GGML_OP_COUNT` 102 at both tags; `0017` -- the four `_mm_prefetch` calls of `quants.c` with the cast still after the addition; `0018` -- `free(buffer->context)` still at `amx.cpp:46`; `0020` -- `ggml_gallocr_needs_realloc()` still compares only `n_leafs`; `0021` -- the `else` branch of the per-request LoRA block still takes `params_base.lora_adapters` without the clear, `init()` still starts a slot without the server's adapters. Root `CMakeLists.txt` the same blob (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` `ON` at lines 167 / 171); `.github/workflows/release.yml`, `vendor/cpp-httplib/CMakeLists.txt`, `ggml/CMakeLists.txt`, `ggml/src/CMakeLists.txt`, `tools/server/CMakeLists.txt` and `tools/mtmd/CMakeLists.txt` byte-identical, so every `KEEP IN SYNC WITH UPSTREAM` pin of `publish.yml` stays (ROCm 10.0.0, Vulkan 1.4.357.0, DLE 2026.1.4 + oneDNN 2026.0.2 + Level Zero 1.33.1, CCCL v3.4.3, OpenVINO 2026.4.1, BoringSSL 0.20261020.0). | +| b11815–b11820 | `tools/server/server-common.{cpp,h}`, `tools/server/server-context.{cpp,h}`, `tools/server/README.md`, `src/llama-vocab.{cpp,h}`, `src/llama-arch.{cpp,h}`, `src/models/nemotron.cpp`, `ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp`, `conversion/`, `gguf-py/`, `ci/run.sh`, `tools/server/tests/conftest.py` (5 commits, `d77256999` = b11820, 2026-10-22) | **One compile break, in `jllama_engine.cpp`: `format_prompt_rerank` gained two parameters** (#29625, "server : support typed content input for RERANK endpoint"): `const std::vector & query_files, const std::vector & doc_files` between `doc` and `init_opt`, the decoded images of the query and the document, which it splices through `process_mtmd_prompt` behind one `get_media_marker()` per file (and refuses with `std::invalid_argument` -- "multimodal rerank input requires --mmproj" -- without a projector). `engine::rerank` passes two empty lists, so its prompt is the text-only one as before (the `files.empty()` branch is the old code). The route itself moved out of the lambda into `server_routes::handle_rerank_impl` and reads the query and each document as a string or `{"text": string or [strings], "image": url or data URI or raw base64, or a list of them}` (texts concatenated, one marker per image; `video` / `fps` / `max_frames` refused); a non-object, non-string item, an empty `documents` and a negative `top_n` are `400`s now -- the last one is what `OaiRerankSupport.readTopN` already did, its Javadoc says so now. `handle_media()` (url / file / data URI into `raw_buffer`s) is declared in `server-common.h` for it. **`OpenAiCompatServer` follows the text half**: `OaiRerankSupport` reads the query and the documents with upstream's item rule (a `text` array concatenated, an object query accepted), and refuses an `image` with a `400` that names `NativeServer` instead of ranking the text alone -- before this, an object document's other keys were ignored, which was harmless while upstream knew no other key and would now answer another question than the one asked; `video` / `fps` / `max_frames` get upstream's own message (6 new cases in `OaiRerankSupportTest`, red before the change). `NativeServer` serves upstream's route, images included; `LlamaModel.rerank` / `handleRerank` and `jllama_rerank` stay text-only. Nemotron (#23343): the Q projection is `n_embd_head_k * n_head` wide instead of `n_embd` (a model with more Q heads than `n_embd / head_dim`), the pre-tokenizer `domyn-small` (`LLAMA_VOCAB_PRE_TYPE_DOMYN_SMALL` = 63), and a GGUF key `tokenizer.ggml.metaspace.prepend_scheme` (`never` / `first` / `always`, read only when present, so an existing GGUF tokenizes as before; `first` on a BPE vocab puts U+2581 before the first fragment only and drops the leading space of the first rendered piece, `always` is refused for BPE) -- the model saver does not write the key, so a GGUF `LlamaTrainer` writes from such a model falls back to `never`. Vulkan (#30828): `portable_aligned_malloc` uses `posix_memalign` instead of `std::aligned_alloc` off Windows (the Vulkan module jars; `check-aligned-free-pairing.cmake` is about `ggml_aligned_malloc`, which this is not, and passes). Two CI-only commits (`ci/run.sh`, `conftest.py`). The server request-field set, its bounds and the response keys are identical (`server-schema.cpp` and `server-task.cpp` are not in the range); nothing in `common/`, `include/`, `tools/mtmd/`, the root `CMakeLists.txt` (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` unchanged) or upstream's `.github/`. **Verified at b11820**: `jllama` and `jllama_test` built from the pin in a fresh tree, `ctest` **686/686** (665 `TEST` macros), `NativeLibraryLoadSmokeTest` against the b11820 library, `OaiRerankSupportTest` 30/30 and `OpenAiCompatServerHttpTest` 81/81. | +| b11815–b11820 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `d77256999`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems). Two patch-target files are in the range, `tools/server/server-context.{cpp,h}` (`0002`, `0003`, `0021`); the commit touches only the rerank route there (the lambda body replaced by a call, `handle_rerank_impl` added after `handle_embeddings_impl`, one declaration in the header), away from every hunk. Drop-checks re-read at b11820: `0002` -- `params_base.load_progress_callback = load_progress_callback;` still unguarded at `server-context.cpp:1283`; `0003` -- no `get_slot_prompt_similarity` in `server-context.{cpp,h}`; `0021` -- the `else` branch of the per-request LoRA block still takes `params_base.lora_adapters` without the clear (line 1939), and `init()` still leaves `slot.lora` empty. The other patch-target files are not in the range, so the b11815 answers stand. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index fd17d956..35e34b89 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -347,8 +347,8 @@ endif() # hash, every other consumer keeps the number, and the pin check after FetchContent_MakeAvailable # asserts that the fetched commit's count IS the number. Clear it once the tag exists; # `llama-next-version.sh` says when, and `llama-next-version.sh --resolve b` prints the hash. -set(JLLAMA_GIT_TAG b11815) -set(JLLAMA_GIT_COMMIT "7170cad4d02f7907517613931840cbb2367399a3") # the commit b11815 names; kept after the tag exists +set(JLLAMA_GIT_TAG b11820) +set(JLLAMA_GIT_COMMIT "d77256999f211f8c0a5b12d13564f48d685b59e7") # the commit b11820 names; kept after the tag exists if(NOT JLLAMA_GIT_TAG MATCHES "^b[0-9]+$") message(FATAL_ERROR "JLLAMA_GIT_TAG must be a llama.cpp release number b, got '${JLLAMA_GIT_TAG}'") endif() diff --git a/llama/src/main/cpp/jllama_engine.cpp b/llama/src/main/cpp/jllama_engine.cpp index 663ecb29..7d6cfb15 100644 --- a/llama/src/main/cpp/jllama_engine.cpp +++ b/llama/src/main/cpp/jllama_engine.cpp @@ -1028,10 +1028,10 @@ std::string rerank(jllama_context *jctx, const std::string &query, const std::ve std::vector tasks; tasks.reserve(documents.size()); for (size_t i = 0; i < documents.size(); i++) { - tasks.push_back(indexed_token_task( - rd, SERVER_TASK_TYPE_RERANK, - format_prompt_rerank(model, jctx->vocab, nullptr, query, documents[i], mtmd_helper_init_opt_default()), - static_cast(i), TASK_RESPONSE_TYPE_NONE)); + tasks.push_back(indexed_token_task(rd, SERVER_TASK_TYPE_RERANK, + format_prompt_rerank(model, jctx->vocab, nullptr, query, documents[i], {}, + {}, mtmd_helper_init_opt_default()), + static_cast(i), TASK_RESPONSE_TYPE_NONE)); } wake_and_post(rd, std::move(tasks)); auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); diff --git a/llama/src/main/java/net/ladenthin/llama/server/OaiRerankSupport.java b/llama/src/main/java/net/ladenthin/llama/server/OaiRerankSupport.java index 27ffd89b..c4776dc1 100644 --- a/llama/src/main/java/net/ladenthin/llama/server/OaiRerankSupport.java +++ b/llama/src/main/java/net/ladenthin/llama/server/OaiRerankSupport.java @@ -39,23 +39,26 @@ final class OaiRerankSupport { private OaiRerankSupport() {} /** - * Read the required {@code query} string. + * Read the required {@code query}: a string, or an object whose {@code "text"} is a string or an + * array of strings (concatenated), the item shape llama.cpp's rerank route reads since b11820 (#29625). * * @param request the parsed rerank request * @return the query text - * @throws IllegalArgumentException if {@code query} is missing or not a string + * @throws IllegalArgumentException if {@code query} is missing or not such an item, or carries media + * (see {@link #readItem}) */ static String readQuery(JsonNode request) { JsonNode query = request.path("query"); - if (!query.isTextual()) { - throw new IllegalArgumentException("'query' must be a string"); + if (query.isMissingNode() || query.isNull()) { + throw new IllegalArgumentException("'query' must be provided"); } - return query.asText(); + return readItem(query, "query"); } /** * Read the required {@code documents} array. Each entry may be a plain string or an object carrying a - * {@code "text"} string (the Cohere/Jina document shape). + * {@code "text"} string or array of strings (the Cohere/Jina document shape, read as {@link #readItem} + * reads it). * * @param request the parsed rerank request * @return the documents, in request order @@ -71,17 +74,61 @@ static String[] readDocuments(JsonNode request) { } List out = new ArrayList<>(documents.size()); for (JsonNode document : documents) { - if (document.isTextual()) { - out.add(document.asText()); - } else if (document.isObject() && document.path("text").isTextual()) { - out.add(document.path("text").asText()); - } else { - throw new IllegalArgumentException("each document must be a string or an object with a 'text' string"); - } + out.add(readItem(document, "document")); } return out.toArray(new String[0]); } + /** + * Read one rerank input item as llama.cpp's {@code handle_rerank_impl} reads it (b11820, #29625): a + * string, or an object whose {@code "text"} is a string or an array of strings, concatenated. + * + *

Two differences, both refusals rather than another answer. An {@code "image"} upstream ranks + * through the multimodal projector, which this transport does not drive ({@link + * net.ladenthin.llama.LlamaModel}'s rerank takes text), so it is refused: ranking the text alone would + * answer another question than the one asked. {@link NativeServer} serves llama.cpp's own route. And + * an object without a {@code "text"} is refused, where upstream would rank an empty text. {@code + * "video"}, {@code "fps"} and {@code "max_frames"} are refused with upstream's own message. + * + * @param item the query or one document + * @param what {@code "query"} or {@code "document"}, for the message + * @return the item's text + * @throws IllegalArgumentException if the item has none of these shapes or carries media + */ + private static String readItem(JsonNode item, String what) { + if (item.isTextual()) { + return item.asText(); + } + final String shape = ("query".equals(what) ? "'query'" : "each " + what) + + " must be a string or an object with a 'text' string"; + if (!item.isObject()) { + throw new IllegalArgumentException(shape); + } + if (item.has("video") || item.has("fps") || item.has("max_frames")) { + throw new IllegalArgumentException( + "video/fps/max_frames are not supported for rerank input (in the " + what + ")"); + } + if (item.has("image")) { + throw new IllegalArgumentException("an 'image' in a rerank " + what + + " is ranked by llama.cpp's own server only (NativeServer); this transport ranks text"); + } + final JsonNode text = item.path("text"); + if (text.isTextual()) { + return text.asText(); + } + if (!text.isArray()) { + throw new IllegalArgumentException(shape); + } + final StringBuilder out = new StringBuilder(); + for (JsonNode part : text) { + if (!part.isTextual()) { + throw new IllegalArgumentException("'text' array elements must be strings (in the " + what + ")"); + } + out.append(part.asText()); + } + return out.toString(); + } + /** * Read the optional {@code top_n} cap. {@code 0} keeps no result, as upstream's {@code * format_response_rerank} does ({@code min(top_n, n)} entries). @@ -95,9 +142,9 @@ static String[] readDocuments(JsonNode request) { * @param request the parsed rerank request * @return the requested cap, or {@code -1} when absent, not a number or outside the {@code int} * range - * @throws IllegalArgumentException if {@code top_n} is negative, integer or float (upstream resizes - * its result vector to {@code min(top_n, n)}, a huge size for a negative value, and fails inside - * the server) + * @throws IllegalArgumentException if {@code top_n} is negative, integer or float (upstream refuses + * it too since b11820, #29625, with {@code "top_n" must be >= 0}; before, it resized its result + * vector to {@code min(top_n, n)}, a huge size for a negative value, and failed inside the server) */ static int readTopN(JsonNode request) { JsonNode topN = request.path("top_n"); diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 810775ae..a635f79f 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -9,13 +9,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11815"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11820"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b11815-"} — call + * plus the resolved upstream commit, e.g. {@code "b11820-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -23,14 +23,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11815"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11820"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b11815"; + public static final String LLAMA_CPP_VERSION = "b11820"; // Constants holder — not instantiable. private LlamaCppVersion() {} diff --git a/llama/src/test/java/net/ladenthin/llama/server/OaiRerankSupportTest.java b/llama/src/test/java/net/ladenthin/llama/server/OaiRerankSupportTest.java index c3a3b5cc..9c2d38f4 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/OaiRerankSupportTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/OaiRerankSupportTest.java @@ -68,6 +68,59 @@ public void readDocumentsThrowsOnUnsupportedEntry() throws IOException { assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readDocuments(request)); } + @Test + public void readQueryAcceptsATextObjectLikeUpstream() throws IOException { + // llama.cpp b11820 (#29625) reads the query like a document: a string or {"text": ...}. + assertThat(OaiRerankSupport.readQuery(read("{\"query\":{\"text\":\"hello\"}}")), is("hello")); + assertThat(OaiRerankSupport.readQuery(read("{\"query\":{\"text\":[\"hel\",\"lo\"]}}")), is("hello")); + } + + @Test + public void readDocumentsConcatenatesATextArrayLikeUpstream() throws IOException { + String[] docs = OaiRerankSupport.readDocuments(read("{\"documents\":[{\"text\":[\"a\",\"b\"]},\"c\"]}")); + assertThat(docs.length, is(2)); + assertThat(docs[0], is("ab")); + assertThat(docs[1], is("c")); + } + + @Test + public void readDocumentsRefusesATextArrayWithANonString() throws IOException { + JsonNode request = read("{\"documents\":[{\"text\":[\"a\",1]}]}"); + IllegalArgumentException e = + assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readDocuments(request)); + assertThat(e.getMessage(), containsString("'text' array elements must be strings")); + } + + @Test + public void anImageInADocumentIsRefusedRatherThanDropped() throws IOException { + // Upstream ranks {"text", "image"} items through the projector since b11820; this transport ranks + // text only, so ranking the text alone would answer another question than the one asked. + JsonNode request = read("{\"documents\":[\"a\",{\"text\":\"x\",\"image\":\"data:image/png;base64,AA==\"}]}"); + IllegalArgumentException e = + assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readDocuments(request)); + assertThat(e.getMessage(), containsString("'image'")); + assertThat(e.getMessage(), containsString("NativeServer")); + } + + @Test + public void anImageInTheQueryIsRefusedRatherThanDropped() throws IOException { + JsonNode request = read("{\"query\":{\"text\":\"q\",\"image\":[\"u\"]},\"documents\":[\"a\"]}"); + IllegalArgumentException e = + assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readQuery(request)); + assertThat(e.getMessage(), containsString("'image'")); + assertThat(e.getMessage(), containsString("NativeServer")); + } + + @Test + public void videoIsRefusedLikeUpstream() throws IOException { + for (String key : new String[] {"video", "fps", "max_frames"}) { + JsonNode request = read("{\"documents\":[{\"text\":\"x\",\"" + key + "\":1}]}"); + IllegalArgumentException e = + assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readDocuments(request)); + assertThat(key, e.getMessage(), containsString("video/fps/max_frames are not supported")); + } + } + @Test public void readTopNReturnsValueOrNegativeOne() throws IOException { assertThat(OaiRerankSupport.readTopN(read("{\"top_n\":3}")), is(3)); From f6159b5ead1fe1a8cd177875809a3a8e4774b38d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 20:33:43 +0000 Subject: [PATCH 2/3] Upgrade llama.cpp from b11820 to b11827 Seven upstream commits (970b51ce7 = b11827), no source change here: SYCL multi-column mat-vec on the Intel XMX engines for short batches (#29864, GGML_SYCL_ENABLE_XMX_ESIMD), an OpenCL CONV_2D_DW kernel (#30900), the MiniCPM-OCR 1.0 projector in mtmd (#30979), DFlash drafter fixes for qwen4exp (#30927) and glm5-next (#30932), and --mlock locking the parts of a split host buffer (a CUDA pinned host buffer above 4 GB, skipped before). mtmd.h and mtmd-helper.h are not in the range, nor is anything in common/, include/ or tools/server/. All 12 patches apply unchanged (check-patches.py: 88 hunks, 0 problems); no patch-target file is in the range. Upstream's build files and .github/ are unchanged across b11815..b11827, so the toolchain pins of publish.yml stay. Verified: fresh configure and build, ctest 686/686, NativeLibraryLoadSmokeTest 7/7 against the b11827 library. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VBFFnrhtvXiUeyxjmUc4R2 --- CHANGELOG.md | 8 ++++++-- CLAUDE.md | 10 +++++----- README.md | 2 +- TODO.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 4 ++-- .../net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 7 files changed, 21 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7adfd456..df1680f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -136,7 +136,7 @@ the first two items break every existing consumer at upgrade. fails the stream through `onError`. ### Added -- **llama.cpp b11820** (from b11639, in nine steps: b11760, b11761, b11799, b11800, b11803, b11809, b11814, b11815, b11820). What the +- **llama.cpp b11827** (from b11639, in ten steps: b11760, b11761, b11799, b11800, b11803, b11809, b11814, b11815, b11820, b11827). What the range brings to a consumer: `POST /tts` (llama.cpp b11781, #26603, speech from a TTS-capable model through the embedded server) and the realtime transcription WebSocket `/v1/realtime` (b11780, #30636), both served by `NativeServer` -- classic and attach mode -- and not by the Java transport or `LlamaModel` yet; a @@ -162,7 +162,11 @@ the first two items break every existing consumer at upgrade. `NativeServer` serves; `OpenAiCompatServer` reads the text half (see "Reranking" under Fixed) and `LlamaModel.rerank` stays text-only. Also Nemotron models with more Q heads than `n_embd / head_dim`, the `domyn-small` pre-tokenizer and a `tokenizer.ggml.metaspace.prepend_scheme` GGUF key (#23343), and - `posix_memalign` in the Vulkan buffers (#30828). + `posix_memalign` in the Vulkan buffers (#30828). The b11827 step (2026-10-22): `--mlock` locks the + parts of a split host buffer too (a CUDA pinned host buffer above 4 GB, which it skipped before), SYCL + mat-vec kernels on the Intel XMX engines for short batches (#29864, `GGML_SYCL_ENABLE_XMX_ESIMD`), an + OpenCL depthwise `CONV_2D_DW` (#30900), the MiniCPM-OCR 1.0 projector (#30979) and DFlash drafter + fixes for qwen4exp and glm5-next -- nothing here changed. Two patches dropped (`0016`, `0019`), `0007` refreshed. - **`RouterModel.Status.INCOMPLETE`** (llama.cpp b11714): a router download that never finished is listed as `"incomplete"` without a worker; it was `UNKNOWN` here. diff --git a/CLAUDE.md b/CLAUDE.md index 0bd3fad0..6118aacd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b11820** +Current llama.cpp pinned version: **b11827** ## Natives jars: library jars and GPU module jars (`.github/natives.csv`) @@ -213,7 +213,7 @@ measurements behind the Windows step are in `docs/handover/local-agent-report-b1 a `llama-common`/`cpp-httplib` that came out shared (one more file to ship). On the static platforms (macOS, s390x) all of it stays inside `libjllama`. **Every ggml default llama.cpp's own CMakeLists changes before its `add_subdirectory(ggml)` is repeated** -- `GGML_LLAMAFILE_DEFAULT` - and `GGML_CUDA_GRAPHS_DEFAULT` at b11820 (unchanged since b11639) -- because adding ggml first skips that block; re-check with + and `GGML_CUDA_GRAPHS_DEFAULT` at b11827 (unchanged since b11639) -- because adding ggml first skips that block; re-check with `grep -n _DEFAULT CMakeLists.txt` at the new tag on every bump. The second one was missing once: ggml's own default for CUDA graphs is OFF, and a CUDA module built without it lost two thirds of its token generation (265.6 -> 88.8 tg128 on an RTX 3070, prompt processing -20%; Vulkan, which @@ -1183,7 +1183,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b11820 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b11827 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -1223,7 +1223,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`JLLAMA_GIT_TAG b11820`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`JLLAMA_GIT_TAG b11827`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -2671,7 +2671,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `JLLAMA_GIT_TAG b11820`. +llama.cpp is fetched via CMake FetchContent, pinned to `JLLAMA_GIT_TAG b11827`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.18.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 25ed8abe..8b3c0846 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b11820](https://img.shields.io/badge/llama.cpp-%23b11820-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11820) +[![llama.cpp b11827](https://img.shields.io/badge/llama.cpp-%23b11827-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11827) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.1%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/TODO.md b/TODO.md index d8bf92be..89fe4c27 100644 --- a/TODO.md +++ b/TODO.md @@ -575,7 +575,7 @@ stays a no-op -- callers repeat the stop until the serve returns, as `RpcServer` The bump itself is done (b11639 -> b11760 -> b11761 -> b11799 -> b11800, b11803 on 2026-10-21: Flint, SenseNova U1 and a jinja fix, nothing to wire; b11809 on 2026-10-22: LoRA on prism.hadamard folded weights, an OpenCL ARGMAX kernel, nothing to wire; b11814 on 2026-10-22: the `tokenizer.ggml.clean_spaces` key and the Pixtral -pre-tokenizer change, nothing to wire; b11815 on 2026-10-22: a SYCL CONV_2D layout fix, nothing to wire; b11820 on 2026-10-22: typed rerank items, `OpenAiCompatServer` reads their text and refuses images; patches `0016` and `0019` dropped, `0007` refreshed; +pre-tokenizer change, nothing to wire; b11815 on 2026-10-22: a SYCL CONV_2D layout fix, nothing to wire; b11820 on 2026-10-22: typed rerank items, `OpenAiCompatServer` reads their text and refuses images; b11827 on 2026-10-22: SYCL XMX mat-vec, OpenCL depthwise conv, MiniCPM-OCR, the `--mlock` split-buffer fix, nothing to wire; patches `0016` and `0019` dropped, `0007` refreshed; `docs/history/llama-cpp-breaking-changes.md` has the rows). Five things the range brings that this library does not expose, each a decision rather than an omission: diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 11d6bfc6..de719fd6 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -913,3 +913,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11814–b11815 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `7170cad4d`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems; and `git apply --check` + `git apply` in filename order on a pristine b11815 worktree, every one clean, the 48 paths the patches touch equal to the `diff --git` headers). **Across the whole b11803..b11815 range no patch-target file changed**: the 48 touched paths are the same blobs at both tags (`git rev-parse :`), and the intersection with `git diff --name-only b11803..b11815` (59 files) is empty. The drop-checks, re-read at b11815, give the b11803 answers: `0001` -- `argv = utf8.ptrs.data()` at `common/arg.cpp:1288` inside the count guard, no `common_params_parse_main` anywhere in the tree (75 `common_params_parse(` call sites at both tags, none new); `0002` -- `params_base.load_progress_callback = load_progress_callback;` unguarded at `server-context.cpp:1283`; `0006`/`0007` -- `tools/server/server.cpp` is the same blob, the helper body still byte-identical to the route-table block it replaces; `0008` -- `get_server_exec_path()` at `server-models.cpp:575`, no `LLAMA_SERVER_WORKER_CMD`; `0014` -- `common/log.h` names only `common_log_default_callback`; `0015` -- `GGML_ABORT("Failed to connect` at `ggml-rpc.cpp:650`, no `stop_server`, `RPC_PROTO` 8/0/1 and `GGML_OP_COUNT` 102 at both tags; `0017` -- the four `_mm_prefetch` calls of `quants.c` with the cast still after the addition; `0018` -- `free(buffer->context)` still at `amx.cpp:46`; `0020` -- `ggml_gallocr_needs_realloc()` still compares only `n_leafs`; `0021` -- the `else` branch of the per-request LoRA block still takes `params_base.lora_adapters` without the clear, `init()` still starts a slot without the server's adapters. Root `CMakeLists.txt` the same blob (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` `ON` at lines 167 / 171); `.github/workflows/release.yml`, `vendor/cpp-httplib/CMakeLists.txt`, `ggml/CMakeLists.txt`, `ggml/src/CMakeLists.txt`, `tools/server/CMakeLists.txt` and `tools/mtmd/CMakeLists.txt` byte-identical, so every `KEEP IN SYNC WITH UPSTREAM` pin of `publish.yml` stays (ROCm 10.0.0, Vulkan 1.4.357.0, DLE 2026.1.4 + oneDNN 2026.0.2 + Level Zero 1.33.1, CCCL v3.4.3, OpenVINO 2026.4.1, BoringSSL 0.20261020.0). | | b11815–b11820 | `tools/server/server-common.{cpp,h}`, `tools/server/server-context.{cpp,h}`, `tools/server/README.md`, `src/llama-vocab.{cpp,h}`, `src/llama-arch.{cpp,h}`, `src/models/nemotron.cpp`, `ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp`, `conversion/`, `gguf-py/`, `ci/run.sh`, `tools/server/tests/conftest.py` (5 commits, `d77256999` = b11820, 2026-10-22) | **One compile break, in `jllama_engine.cpp`: `format_prompt_rerank` gained two parameters** (#29625, "server : support typed content input for RERANK endpoint"): `const std::vector & query_files, const std::vector & doc_files` between `doc` and `init_opt`, the decoded images of the query and the document, which it splices through `process_mtmd_prompt` behind one `get_media_marker()` per file (and refuses with `std::invalid_argument` -- "multimodal rerank input requires --mmproj" -- without a projector). `engine::rerank` passes two empty lists, so its prompt is the text-only one as before (the `files.empty()` branch is the old code). The route itself moved out of the lambda into `server_routes::handle_rerank_impl` and reads the query and each document as a string or `{"text": string or [strings], "image": url or data URI or raw base64, or a list of them}` (texts concatenated, one marker per image; `video` / `fps` / `max_frames` refused); a non-object, non-string item, an empty `documents` and a negative `top_n` are `400`s now -- the last one is what `OaiRerankSupport.readTopN` already did, its Javadoc says so now. `handle_media()` (url / file / data URI into `raw_buffer`s) is declared in `server-common.h` for it. **`OpenAiCompatServer` follows the text half**: `OaiRerankSupport` reads the query and the documents with upstream's item rule (a `text` array concatenated, an object query accepted), and refuses an `image` with a `400` that names `NativeServer` instead of ranking the text alone -- before this, an object document's other keys were ignored, which was harmless while upstream knew no other key and would now answer another question than the one asked; `video` / `fps` / `max_frames` get upstream's own message (6 new cases in `OaiRerankSupportTest`, red before the change). `NativeServer` serves upstream's route, images included; `LlamaModel.rerank` / `handleRerank` and `jllama_rerank` stay text-only. Nemotron (#23343): the Q projection is `n_embd_head_k * n_head` wide instead of `n_embd` (a model with more Q heads than `n_embd / head_dim`), the pre-tokenizer `domyn-small` (`LLAMA_VOCAB_PRE_TYPE_DOMYN_SMALL` = 63), and a GGUF key `tokenizer.ggml.metaspace.prepend_scheme` (`never` / `first` / `always`, read only when present, so an existing GGUF tokenizes as before; `first` on a BPE vocab puts U+2581 before the first fragment only and drops the leading space of the first rendered piece, `always` is refused for BPE) -- the model saver does not write the key, so a GGUF `LlamaTrainer` writes from such a model falls back to `never`. Vulkan (#30828): `portable_aligned_malloc` uses `posix_memalign` instead of `std::aligned_alloc` off Windows (the Vulkan module jars; `check-aligned-free-pairing.cmake` is about `ggml_aligned_malloc`, which this is not, and passes). Two CI-only commits (`ci/run.sh`, `conftest.py`). The server request-field set, its bounds and the response keys are identical (`server-schema.cpp` and `server-task.cpp` are not in the range); nothing in `common/`, `include/`, `tools/mtmd/`, the root `CMakeLists.txt` (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` unchanged) or upstream's `.github/`. **Verified at b11820**: `jllama` and `jllama_test` built from the pin in a fresh tree, `ctest` **686/686** (665 `TEST` macros), `NativeLibraryLoadSmokeTest` against the b11820 library, `OaiRerankSupportTest` 30/30 and `OpenAiCompatServerHttpTest` 81/81. | | b11815–b11820 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `d77256999`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems). Two patch-target files are in the range, `tools/server/server-context.{cpp,h}` (`0002`, `0003`, `0021`); the commit touches only the rerank route there (the lambda body replaced by a call, `handle_rerank_impl` added after `handle_embeddings_impl`, one declaration in the header), away from every hunk. Drop-checks re-read at b11820: `0002` -- `params_base.load_progress_callback = load_progress_callback;` still unguarded at `server-context.cpp:1283`; `0003` -- no `get_slot_prompt_similarity` in `server-context.{cpp,h}`; `0021` -- the `else` branch of the per-request LoRA block still takes `params_base.lora_adapters` without the clear (line 1939), and `init()` still leaves `slot.lora` empty. The other patch-target files are not in the range, so the b11815 answers stand. | +| b11820–b11827 | `ggml/src/ggml-sycl/` (`common.hpp`, `esimd.hpp`, `fusion.cpp`, `ggml-sycl.cpp`, `mmvq.{cpp,hpp}`), `ggml/src/ggml-opencl/` (`CMakeLists.txt`, `ggml-opencl.cpp`, `kernels/conv2_dw.cl`), `src/llama-model.cpp`, `src/llama-memory-hybrid-idx.cpp`, `src/models/{qwen4exp,glm5-next,clef}.cpp`, `tools/mtmd/{clip-impl.h,clip.cpp,mtmd-image.h,mtmd.cpp}`, `conversion/`, `gguf-py/`, `tests/test-backend-ops.cpp`, `docs/backend/SYCL.md` (7 commits, `970b51ce7` = b11827, 2026-10-22 .. 2026-10-23) | **No source change here.** SYCL multi-column mat-vec on the Intel XMX engines (#29864): ESIMD DPAS kernels for reordered Q2_K..Q6_K and Q8_0 weights up to `GGML_SYCL_XMX_MAX_COLS` (default 80) activation columns -- speculative decoding and short prompt batches -- plus a fused gate/up GLU kernel up to `GGML_SYCL_XMX_GLU_MAX_COLS` (default 16), switched by `GGML_SYCL_ENABLE_XMX_ESIMD` (default 1); compiled in where `icpx` finds `` (`GGML_SYCL_MMVQ_HAS_XMX` in `common.hpp`), i.e. in both SYCL module jars, whose DLE 2026.1.4 toolchain is unchanged. OpenCL `CONV_2D_DW` (#30900, F32 and F16 kernels in `conv2_dw.cl`, added to the embedded kernel list -- the three OpenCL module jars). `--mlock` (`36cbf415f`, "Fix !30793"): the buffers holding the tensors are locked one by one, so the parts of a split host buffer -- since #30449 a CUDA pinned host buffer above 4 GB is a multi-buffer without a base, which the old check skipped -- are locked too, and a no-alloc load locks nothing. DFlash drafters on qwen4exp (#30927) read each layer input as the mean of its hyper-connection streams, built only for the layers `embeddings_layer_inp` asks for; glm5-next's k-pool inputs are one 1-D tensor instead of a 2-D one with views (#30932). MiniCPM-OCR 1.0 (#30979): projector `minicpm_ocr_1_0`, the MiniCPM-V preprocessor with the slice aligned to `patch_size * n_merge`; `tools/mtmd/mtmd.h` and `mtmd-helper.h` are not in the range, the two changed headers are internal. `clef.cpp` includes ``. Nothing in `common/`, `include/`, `tools/server/`, the root `CMakeLists.txt` (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` unchanged) or upstream's `.github/`; the server request-field set, its bounds and the response keys are identical (`tools/server/` is not in the range). **Verified at b11827**: `jllama` and `jllama_test` built from the pin in a fresh tree, `ctest` **686/686** (665 `TEST` macros), `NativeLibraryLoadSmokeTest` 7/7 against the b11827 library. | +| b11820–b11827 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `970b51ce7`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems). No patch-target file is in the range (the intersection of the 48 paths the patches touch with `git diff --name-only b11820..b11827`, 23 files, is empty), so the drop-check answers of the b11820 row stand. Across b11815..b11827 the root `CMakeLists.txt`, `.github/`, `vendor/`, `ggml/CMakeLists.txt`, `ggml/src/CMakeLists.txt`, `tools/server/CMakeLists.txt` and `tools/mtmd/CMakeLists.txt` are unchanged, so every `KEEP IN SYNC WITH UPSTREAM` pin of `publish.yml` stays (ROCm 10.0.0, Vulkan 1.4.357.0, DLE 2026.1.4 + oneDNN 2026.0.2 + Level Zero 1.33.1, CCCL v3.4.3, OpenVINO 2026.4.1, BoringSSL 0.20261020.0). | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 35e34b89..c4fefa58 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -347,8 +347,8 @@ endif() # hash, every other consumer keeps the number, and the pin check after FetchContent_MakeAvailable # asserts that the fetched commit's count IS the number. Clear it once the tag exists; # `llama-next-version.sh` says when, and `llama-next-version.sh --resolve b` prints the hash. -set(JLLAMA_GIT_TAG b11820) -set(JLLAMA_GIT_COMMIT "d77256999f211f8c0a5b12d13564f48d685b59e7") # the commit b11820 names; kept after the tag exists +set(JLLAMA_GIT_TAG b11827) +set(JLLAMA_GIT_COMMIT "970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc") # the commit b11827 names; kept after the tag exists if(NOT JLLAMA_GIT_TAG MATCHES "^b[0-9]+$") message(FATAL_ERROR "JLLAMA_GIT_TAG must be a llama.cpp release number b, got '${JLLAMA_GIT_TAG}'") endif() diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index a635f79f..ebfaf7ed 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -9,13 +9,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11820"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11827"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b11820-"} — call + * plus the resolved upstream commit, e.g. {@code "b11827-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -23,14 +23,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11820"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11827"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b11820"; + public static final String LLAMA_CPP_VERSION = "b11827"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 04e3cea8ba5cb8f6d891bcb40e9f9536cde97b75 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 20:50:19 +0000 Subject: [PATCH 3/3] Read rerank items exactly as upstream does, and correct the bump docs The review of the b11820/b11827 commits found that OaiRerankSupport.readItem refused an item whose "image" was null or [] and one without a "text". Upstream's parse_item ranks both by their text (an empty text for the second). An object document with "image": null was ranked here before 5108b4d, so for documents this was a regression of that commit. readItem now follows parse_item step by step: video/fps/max_frames refused, a text string or array of strings concatenated (missing or null is empty text, another type or a non-string element is upstream's message), a null or empty image ignored, an image of another type or a list starting with a non-string refused with upstream's message. A string image or a list of them stays the one deliberate difference: a 400 naming NativeServer instead of ranking the text alone. Five new cases in OaiRerankSupportTest, four of them red before the change; the existing text-array case now expects upstream's quoting. Docs: the CHANGELOG's b11827 sentence called the glm5-next commit a DFlash fix (it removes graph inputs, #30932); TODO.md named a jllama_completion that does not exist and data URIs where the raw prompt carries base64; the b11820 history row said three 400s were new where only the negative top_n is, missed one comment line upstream adds in handle_embeddings_impl, and left convert_hf_to_gguf_update.py out of the file list. Verified: OaiRerankSupportTest 35/35, OpenAiCompatServerHttpTest 81/81, spotless and spotbugs clean. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VBFFnrhtvXiUeyxjmUc4R2 --- CHANGELOG.md | 11 +-- TODO.md | 6 +- docs/history/llama-cpp-breaking-changes.md | 4 +- .../llama/server/OaiRerankSupport.java | 68 +++++++++++-------- .../llama/server/OaiRerankSupportTest.java | 50 +++++++++++++- 5 files changed, 99 insertions(+), 40 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index df1680f7..5a23a96b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -165,8 +165,8 @@ the first two items break every existing consumer at upgrade. `posix_memalign` in the Vulkan buffers (#30828). The b11827 step (2026-10-22): `--mlock` locks the parts of a split host buffer too (a CUDA pinned host buffer above 4 GB, which it skipped before), SYCL mat-vec kernels on the Intel XMX engines for short batches (#29864, `GGML_SYCL_ENABLE_XMX_ESIMD`), an - OpenCL depthwise `CONV_2D_DW` (#30900), the MiniCPM-OCR 1.0 projector (#30979) and DFlash drafter - fixes for qwen4exp and glm5-next -- nothing here changed. + OpenCL depthwise `CONV_2D_DW` (#30900), the MiniCPM-OCR 1.0 projector (#30979), a DFlash drafter + fix for qwen4exp (#30927) and fewer graph inputs for glm5-next (#30932) -- nothing here changed. Two patches dropped (`0016`, `0019`), `0007` refreshed. - **`RouterModel.Status.INCOMPLETE`** (llama.cpp b11714): a router download that never finished is listed as `"incomplete"` without a worker; it was `UNKNOWN` here. @@ -1148,9 +1148,10 @@ the first two items break every existing consumer at upgrade. empty result list. `LlamaModel.rerank(String, String...)` returns its map in the order of the documents instead of hash order. The query and each document are read as llama.cpp reads them since b11820 (#29625): a string or `{"text": string or [strings]}` (an object query is accepted, a text array - concatenated), and an item with an `image` -- which upstream ranks through the projector -- is `400` - instead of being ranked by its text alone; `video` / `fps` / `max_frames` are refused with upstream's - message. `NativeServer` serves upstream's route, images included. + concatenated, a missing or `null` `text` empty), and an item with an `image` -- which upstream ranks + through the projector -- is `400` instead of being ranked by its text alone (a `null` or empty `image` + is none, there as here); `video` / `fps` / `max_frames` are refused, as upstream refuses them. + `NativeServer` serves upstream's route, images included. - **Java API.** - **`InferenceParameters.withMessages(List)` keeps `tool_calls` and `tool_call_id`**, which only `ChatRequest` added, so a tool round built through it sent an assistant turn without diff --git a/TODO.md b/TODO.md index 89fe4c27..021aa047 100644 --- a/TODO.md +++ b/TODO.md @@ -591,9 +591,9 @@ library does not expose, each a decision rather than an omission: (`format_prompt_rerank(..., query_files, doc_files, ...)`, `handle_media()` for the decoding); `NativeServer` serves it. `engine::rerank` passes empty file lists, `jllama_rerank` and `LlamaModel.rerank` take strings, and `OpenAiCompatServer` refuses an `image` with a `400` naming `NativeServer` rather than ranking the text - alone. Wiring it means items in the C API (text plus data URIs, as `jllama_completion`'s multimodal prompt - already carries) and a typed overload on `LlamaModel`; it needs a multimodal reranker in the CI model set to - be tested. Not before the release. + alone. Wiring it means items in the C API (text plus base64 media, as the multimodal prompt of + `jllama_completions` -- `multimodal_data` -- already carries) and a typed overload on `LlamaModel`; it + needs a multimodal reranker in the CI model set to be tested. Not before the release. - **Realtime transcription** (`/v1/realtime`, WebSocket, b11780, #30636: `mtmd_audio_stream`, `SERVER_TASK_TYPE_TRANSCRIBE_RT`, `SERVER_CONTROL_ACTION_AUDIO_*`). Served by `NativeServer`; the JDK's `com.sun.net.httpserver` has no WebSocket, so the Java transport cannot follow without a dependency. Not diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index de719fd6..846beb5c 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -911,7 +911,7 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11809–b11814 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `d88040c70`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems). The range touches two files under `tests/` (`tests/CMakeLists.txt`, `tests/test-tokenizer-0.cpp`), neither of them a `0001` target (its `tests/` hunks are `test-arg-parser.cpp`, `test-export-graph-ops.cpp` and `test-thread-safety.cpp`), and no other patch-target file, so the drop-check answers of the b11803 row stand unchanged. | | b11814–b11815 | `ggml/src/ggml-sycl/conv2d.cpp`, `ggml/src/ggml-sycl/ggml-sycl.cpp`, `docs/ops.md`, `docs/ops/SYCL.csv` (one commit, `7170cad4d` = b11815, 2026-10-22) | **One SYCL kernel fix, no source change here.** `CONV_2D` on the SYCL backend (#30363) supports the channel-fastest CWHN layout (input `[N][H][W][C]`, kernel `[KH][KW][IC][OC]`) next to the contiguous WHCN one -- the kernel is templated on the layout, `supports_op` accepts a non-contiguous input or kernel whose channel stride is the element size, and the former `GGML_ASSERT(p[6] == 0)` ("no cwhn layout support") is gone; the regenerated `docs/ops/SYCL.csv` (7,793 insertions, 6,047 deletions, 2.3 MB of diff) is what pushed this one-commit chunk over the runbook's 100 KiB threshold on its own, so it is a chunk of its own. The SYCL module jars are built in CI only (`sycl-linux-x86-64`, `sycl-windows-x86-64`); the change reaches a consumer running a convolution (a vision projector, Whisper-style audio input) on an Intel GPU with a CWHN tensor, which used to abort. Nothing in `common/`, `include/`, `tools/server/`, `tools/mtmd/`, `src/`, the root `CMakeLists.txt` (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` unchanged) or upstream's `.github/`; the server request-field set, its bounds and the response keys are identical (`tools/server/` is not in the range). **Verified at b11815**: `jllama` and `jllama_test` built from the pin in a fresh tree, `ctest` **686/686** (665 `TEST` macros), `NativeLibraryLoadSmokeTest` against the b11815 library. | | b11814–b11815 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `7170cad4d`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems; and `git apply --check` + `git apply` in filename order on a pristine b11815 worktree, every one clean, the 48 paths the patches touch equal to the `diff --git` headers). **Across the whole b11803..b11815 range no patch-target file changed**: the 48 touched paths are the same blobs at both tags (`git rev-parse :`), and the intersection with `git diff --name-only b11803..b11815` (59 files) is empty. The drop-checks, re-read at b11815, give the b11803 answers: `0001` -- `argv = utf8.ptrs.data()` at `common/arg.cpp:1288` inside the count guard, no `common_params_parse_main` anywhere in the tree (75 `common_params_parse(` call sites at both tags, none new); `0002` -- `params_base.load_progress_callback = load_progress_callback;` unguarded at `server-context.cpp:1283`; `0006`/`0007` -- `tools/server/server.cpp` is the same blob, the helper body still byte-identical to the route-table block it replaces; `0008` -- `get_server_exec_path()` at `server-models.cpp:575`, no `LLAMA_SERVER_WORKER_CMD`; `0014` -- `common/log.h` names only `common_log_default_callback`; `0015` -- `GGML_ABORT("Failed to connect` at `ggml-rpc.cpp:650`, no `stop_server`, `RPC_PROTO` 8/0/1 and `GGML_OP_COUNT` 102 at both tags; `0017` -- the four `_mm_prefetch` calls of `quants.c` with the cast still after the addition; `0018` -- `free(buffer->context)` still at `amx.cpp:46`; `0020` -- `ggml_gallocr_needs_realloc()` still compares only `n_leafs`; `0021` -- the `else` branch of the per-request LoRA block still takes `params_base.lora_adapters` without the clear, `init()` still starts a slot without the server's adapters. Root `CMakeLists.txt` the same blob (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` `ON` at lines 167 / 171); `.github/workflows/release.yml`, `vendor/cpp-httplib/CMakeLists.txt`, `ggml/CMakeLists.txt`, `ggml/src/CMakeLists.txt`, `tools/server/CMakeLists.txt` and `tools/mtmd/CMakeLists.txt` byte-identical, so every `KEEP IN SYNC WITH UPSTREAM` pin of `publish.yml` stays (ROCm 10.0.0, Vulkan 1.4.357.0, DLE 2026.1.4 + oneDNN 2026.0.2 + Level Zero 1.33.1, CCCL v3.4.3, OpenVINO 2026.4.1, BoringSSL 0.20261020.0). | -| b11815–b11820 | `tools/server/server-common.{cpp,h}`, `tools/server/server-context.{cpp,h}`, `tools/server/README.md`, `src/llama-vocab.{cpp,h}`, `src/llama-arch.{cpp,h}`, `src/models/nemotron.cpp`, `ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp`, `conversion/`, `gguf-py/`, `ci/run.sh`, `tools/server/tests/conftest.py` (5 commits, `d77256999` = b11820, 2026-10-22) | **One compile break, in `jllama_engine.cpp`: `format_prompt_rerank` gained two parameters** (#29625, "server : support typed content input for RERANK endpoint"): `const std::vector & query_files, const std::vector & doc_files` between `doc` and `init_opt`, the decoded images of the query and the document, which it splices through `process_mtmd_prompt` behind one `get_media_marker()` per file (and refuses with `std::invalid_argument` -- "multimodal rerank input requires --mmproj" -- without a projector). `engine::rerank` passes two empty lists, so its prompt is the text-only one as before (the `files.empty()` branch is the old code). The route itself moved out of the lambda into `server_routes::handle_rerank_impl` and reads the query and each document as a string or `{"text": string or [strings], "image": url or data URI or raw base64, or a list of them}` (texts concatenated, one marker per image; `video` / `fps` / `max_frames` refused); a non-object, non-string item, an empty `documents` and a negative `top_n` are `400`s now -- the last one is what `OaiRerankSupport.readTopN` already did, its Javadoc says so now. `handle_media()` (url / file / data URI into `raw_buffer`s) is declared in `server-common.h` for it. **`OpenAiCompatServer` follows the text half**: `OaiRerankSupport` reads the query and the documents with upstream's item rule (a `text` array concatenated, an object query accepted), and refuses an `image` with a `400` that names `NativeServer` instead of ranking the text alone -- before this, an object document's other keys were ignored, which was harmless while upstream knew no other key and would now answer another question than the one asked; `video` / `fps` / `max_frames` get upstream's own message (6 new cases in `OaiRerankSupportTest`, red before the change). `NativeServer` serves upstream's route, images included; `LlamaModel.rerank` / `handleRerank` and `jllama_rerank` stay text-only. Nemotron (#23343): the Q projection is `n_embd_head_k * n_head` wide instead of `n_embd` (a model with more Q heads than `n_embd / head_dim`), the pre-tokenizer `domyn-small` (`LLAMA_VOCAB_PRE_TYPE_DOMYN_SMALL` = 63), and a GGUF key `tokenizer.ggml.metaspace.prepend_scheme` (`never` / `first` / `always`, read only when present, so an existing GGUF tokenizes as before; `first` on a BPE vocab puts U+2581 before the first fragment only and drops the leading space of the first rendered piece, `always` is refused for BPE) -- the model saver does not write the key, so a GGUF `LlamaTrainer` writes from such a model falls back to `never`. Vulkan (#30828): `portable_aligned_malloc` uses `posix_memalign` instead of `std::aligned_alloc` off Windows (the Vulkan module jars; `check-aligned-free-pairing.cmake` is about `ggml_aligned_malloc`, which this is not, and passes). Two CI-only commits (`ci/run.sh`, `conftest.py`). The server request-field set, its bounds and the response keys are identical (`server-schema.cpp` and `server-task.cpp` are not in the range); nothing in `common/`, `include/`, `tools/mtmd/`, the root `CMakeLists.txt` (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` unchanged) or upstream's `.github/`. **Verified at b11820**: `jllama` and `jllama_test` built from the pin in a fresh tree, `ctest` **686/686** (665 `TEST` macros), `NativeLibraryLoadSmokeTest` against the b11820 library, `OaiRerankSupportTest` 30/30 and `OpenAiCompatServerHttpTest` 81/81. | -| b11815–b11820 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `d77256999`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems). Two patch-target files are in the range, `tools/server/server-context.{cpp,h}` (`0002`, `0003`, `0021`); the commit touches only the rerank route there (the lambda body replaced by a call, `handle_rerank_impl` added after `handle_embeddings_impl`, one declaration in the header), away from every hunk. Drop-checks re-read at b11820: `0002` -- `params_base.load_progress_callback = load_progress_callback;` still unguarded at `server-context.cpp:1283`; `0003` -- no `get_slot_prompt_similarity` in `server-context.{cpp,h}`; `0021` -- the `else` branch of the per-request LoRA block still takes `params_base.lora_adapters` without the clear (line 1939), and `init()` still leaves `slot.lora` empty. The other patch-target files are not in the range, so the b11815 answers stand. | +| b11815–b11820 | `tools/server/server-common.{cpp,h}`, `tools/server/server-context.{cpp,h}`, `tools/server/README.md`, `src/llama-vocab.{cpp,h}`, `src/llama-arch.{cpp,h}`, `src/models/nemotron.cpp`, `ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp`, `conversion/`, `gguf-py/`, `convert_hf_to_gguf_update.py`, `ci/run.sh`, `tools/server/tests/conftest.py` (5 commits, `d77256999` = b11820, 2026-10-22) | **One compile break, in `jllama_engine.cpp`: `format_prompt_rerank` gained two parameters** (#29625, "server : support typed content input for RERANK endpoint"): `const std::vector & query_files, const std::vector & doc_files` between `doc` and `init_opt`, the decoded images of the query and the document, which it splices through `process_mtmd_prompt` behind one `get_media_marker()` per file (and refuses with `std::invalid_argument` -- "multimodal rerank input requires --mmproj" -- without a projector). `engine::rerank` passes two empty lists, so its prompt is the text-only one as before (the `files.empty()` branch is the old code). The route itself moved out of the lambda into `server_routes::handle_rerank_impl` and reads the query and each document as a string or `{"text": string or [strings], "image": url or data URI or raw base64, or a list of them}` (texts concatenated, one marker per image; `video` / `fps` / `max_frames` refused); a negative `top_n` is a `400` now (before, `format_response_rerank` resized its result to a huge size and threw, a `500`) -- what `OaiRerankSupport.readTopN` already did, its Javadoc says so now; a non-string, non-object item and an empty `documents` stay `400`s with new messages, and an object item, a `400` before (`documents` was read as a string array, a non-string query refused), is read now. `handle_media()` (url / file / data URI into `raw_buffer`s) is declared in `server-common.h` for it. **`OpenAiCompatServer` follows the text half**: `OaiRerankSupport` reads the query and the documents with upstream's item rule (a `text` array concatenated, a missing or `null` `text` empty, an object query accepted, a `null` or empty `image` none), and refuses an `image` with a `400` that names `NativeServer` instead of ranking the text alone -- before this, an object document's other keys were ignored, which was harmless while upstream knew no other key and would now answer another question than the one asked; `video` / `fps` / `max_frames` are refused as upstream refuses them (11 new cases in `OaiRerankSupportTest`, red before the change but the one that pins the refusal of an image list). The first form of this change (`5108b4d`) refused an item whose `image` was `null` or `[]` -- which used to be ranked here, and is by upstream -- and one without a `text`, which upstream ranks as empty text; the branch's own review caught both and the following commit corrects them, with five of the eleven cases. `NativeServer` serves upstream's route, images included; `LlamaModel.rerank` / `handleRerank` and `jllama_rerank` stay text-only. Nemotron (#23343): the Q projection is `n_embd_head_k * n_head` wide instead of `n_embd` (a model with more Q heads than `n_embd / head_dim`), the pre-tokenizer `domyn-small` (`LLAMA_VOCAB_PRE_TYPE_DOMYN_SMALL` = 63), and a GGUF key `tokenizer.ggml.metaspace.prepend_scheme` (`never` / `first` / `always`, read only when present, so an existing GGUF tokenizes as before; `first` on a BPE vocab puts U+2581 before the first fragment only and drops the leading space of the first rendered piece, `always` is refused for BPE) -- the model saver does not write the key, so a GGUF `LlamaTrainer` writes from such a model falls back to `never`. Vulkan (#30828): `portable_aligned_malloc` uses `posix_memalign` instead of `std::aligned_alloc` off Windows (the Vulkan module jars; `check-aligned-free-pairing.cmake` is about `ggml_aligned_malloc`, which this is not, and passes). Two CI-only commits (`ci/run.sh`, `conftest.py`). The server request-field set, its bounds and the response keys are identical (`server-schema.cpp` and `server-task.cpp` are not in the range); nothing in `common/`, `include/`, `tools/mtmd/`, the root `CMakeLists.txt` (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` unchanged) or upstream's `.github/`. **Verified at b11820**: `jllama` and `jllama_test` built from the pin in a fresh tree, `ctest` **686/686** (665 `TEST` macros), `NativeLibraryLoadSmokeTest` against the b11820 library, `OaiRerankSupportTest` 30/30 and `OpenAiCompatServerHttpTest` 81/81. | +| b11815–b11820 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `d77256999`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems). Two patch-target files are in the range, `tools/server/server-context.{cpp,h}` (`0002`, `0003`, `0021`); the commit touches only the rerank route there (the lambda body replaced by a call, `handle_rerank_impl` added after `handle_embeddings_impl`, one declaration in the header) plus one comment line inside `handle_embeddings_impl`, away from every hunk. Drop-checks re-read at b11820: `0002` -- `params_base.load_progress_callback = load_progress_callback;` still unguarded at `server-context.cpp:1283`; `0003` -- no `get_slot_prompt_similarity` in `server-context.{cpp,h}`; `0021` -- the `else` branch of the per-request LoRA block still takes `params_base.lora_adapters` without the clear (line 1939), and `init()` still leaves `slot.lora` empty. The other patch-target files are not in the range, so the b11815 answers stand. | | b11820–b11827 | `ggml/src/ggml-sycl/` (`common.hpp`, `esimd.hpp`, `fusion.cpp`, `ggml-sycl.cpp`, `mmvq.{cpp,hpp}`), `ggml/src/ggml-opencl/` (`CMakeLists.txt`, `ggml-opencl.cpp`, `kernels/conv2_dw.cl`), `src/llama-model.cpp`, `src/llama-memory-hybrid-idx.cpp`, `src/models/{qwen4exp,glm5-next,clef}.cpp`, `tools/mtmd/{clip-impl.h,clip.cpp,mtmd-image.h,mtmd.cpp}`, `conversion/`, `gguf-py/`, `tests/test-backend-ops.cpp`, `docs/backend/SYCL.md` (7 commits, `970b51ce7` = b11827, 2026-10-22 .. 2026-10-23) | **No source change here.** SYCL multi-column mat-vec on the Intel XMX engines (#29864): ESIMD DPAS kernels for reordered Q2_K..Q6_K and Q8_0 weights up to `GGML_SYCL_XMX_MAX_COLS` (default 80) activation columns -- speculative decoding and short prompt batches -- plus a fused gate/up GLU kernel up to `GGML_SYCL_XMX_GLU_MAX_COLS` (default 16), switched by `GGML_SYCL_ENABLE_XMX_ESIMD` (default 1); compiled in where `icpx` finds `` (`GGML_SYCL_MMVQ_HAS_XMX` in `common.hpp`), i.e. in both SYCL module jars, whose DLE 2026.1.4 toolchain is unchanged. OpenCL `CONV_2D_DW` (#30900, F32 and F16 kernels in `conv2_dw.cl`, added to the embedded kernel list -- the three OpenCL module jars). `--mlock` (`36cbf415f`, "Fix !30793"): the buffers holding the tensors are locked one by one, so the parts of a split host buffer -- since #30449 a CUDA pinned host buffer above 4 GB is a multi-buffer without a base, which the old check skipped -- are locked too, and a no-alloc load locks nothing. DFlash drafters on qwen4exp (#30927) read each layer input as the mean of its hyper-connection streams, built only for the layers `embeddings_layer_inp` asks for; glm5-next's k-pool inputs are one 1-D tensor instead of a 2-D one with views (#30932). MiniCPM-OCR 1.0 (#30979): projector `minicpm_ocr_1_0`, the MiniCPM-V preprocessor with the slice aligned to `patch_size * n_merge`; `tools/mtmd/mtmd.h` and `mtmd-helper.h` are not in the range, the two changed headers are internal. `clef.cpp` includes ``. Nothing in `common/`, `include/`, `tools/server/`, the root `CMakeLists.txt` (`GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` unchanged) or upstream's `.github/`; the server request-field set, its bounds and the response keys are identical (`tools/server/` is not in the range). **Verified at b11827**: `jllama` and `jllama_test` built from the pin in a fresh tree, `ctest` **686/686** (665 `TEST` macros), `NativeLibraryLoadSmokeTest` 7/7 against the b11827 library. | | b11820–b11827 | patches | **All 12 apply unchanged** (the fail-loud applier on a fresh build directory at `970b51ce7`; `jllama-build-info` lists 0001, 0002, 0003, 0006, 0007, 0008, 0014, 0015, 0017, 0018, 0020, 0021; `check-patches.py`: 12 patches, 88 hunks, 0 problems). No patch-target file is in the range (the intersection of the 48 paths the patches touch with `git diff --name-only b11820..b11827`, 23 files, is empty), so the drop-check answers of the b11820 row stand. Across b11815..b11827 the root `CMakeLists.txt`, `.github/`, `vendor/`, `ggml/CMakeLists.txt`, `ggml/src/CMakeLists.txt`, `tools/server/CMakeLists.txt` and `tools/mtmd/CMakeLists.txt` are unchanged, so every `KEEP IN SYNC WITH UPSTREAM` pin of `publish.yml` stays (ROCm 10.0.0, Vulkan 1.4.357.0, DLE 2026.1.4 + oneDNN 2026.0.2 + Level Zero 1.33.1, CCCL v3.4.3, OpenVINO 2026.4.1, BoringSSL 0.20261020.0). | diff --git a/llama/src/main/java/net/ladenthin/llama/server/OaiRerankSupport.java b/llama/src/main/java/net/ladenthin/llama/server/OaiRerankSupport.java index c4776dc1..cab85764 100644 --- a/llama/src/main/java/net/ladenthin/llama/server/OaiRerankSupport.java +++ b/llama/src/main/java/net/ladenthin/llama/server/OaiRerankSupport.java @@ -40,12 +40,12 @@ private OaiRerankSupport() {} /** * Read the required {@code query}: a string, or an object whose {@code "text"} is a string or an - * array of strings (concatenated), the item shape llama.cpp's rerank route reads since b11820 (#29625). + * array of strings (concatenated), the item shape llama.cpp's rerank route reads since b11820 (#29625; + * see {@link #readItem}). * * @param request the parsed rerank request * @return the query text - * @throws IllegalArgumentException if {@code query} is missing or not such an item, or carries media - * (see {@link #readItem}) + * @throws IllegalArgumentException if {@code query} is missing or not such an item, or carries an image */ static String readQuery(JsonNode request) { JsonNode query = request.path("query"); @@ -81,50 +81,60 @@ static String[] readDocuments(JsonNode request) { /** * Read one rerank input item as llama.cpp's {@code handle_rerank_impl} reads it (b11820, #29625): a - * string, or an object whose {@code "text"} is a string or an array of strings, concatenated. + * string, or an object whose {@code "text"} is a string or an array of strings (concatenated) and is + * empty text when absent or {@code null}; {@code "video"}, {@code "fps"} and {@code "max_frames"} are + * refused, as upstream refuses them. * - *

Two differences, both refusals rather than another answer. An {@code "image"} upstream ranks - * through the multimodal projector, which this transport does not drive ({@link - * net.ladenthin.llama.LlamaModel}'s rerank takes text), so it is refused: ranking the text alone would - * answer another question than the one asked. {@link NativeServer} serves llama.cpp's own route. And - * an object without a {@code "text"} is refused, where upstream would rank an empty text. {@code - * "video"}, {@code "fps"} and {@code "max_frames"} are refused with upstream's own message. + *

One difference, a refusal rather than another answer: an {@code "image"} (a string, or an array + * with a string) upstream ranks through the multimodal projector, which this transport does not drive + * ({@link net.ladenthin.llama.LlamaModel}'s rerank takes text), so it is refused -- ranking the text + * alone would answer another question than the one asked. {@link NativeServer} serves llama.cpp's own + * route. An {@code "image"} that is {@code null} or an empty array carries no image there either, and + * is ignored here as there. * * @param item the query or one document * @param what {@code "query"} or {@code "document"}, for the message * @return the item's text - * @throws IllegalArgumentException if the item has none of these shapes or carries media + * @throws IllegalArgumentException if the item has none of these shapes or carries an image */ private static String readItem(JsonNode item, String what) { if (item.isTextual()) { return item.asText(); } - final String shape = ("query".equals(what) ? "'query'" : "each " + what) - + " must be a string or an object with a 'text' string"; if (!item.isObject()) { - throw new IllegalArgumentException(shape); + throw new IllegalArgumentException(("query".equals(what) ? "'query'" : "each " + what) + + " must be a string or an object with a 'text' string"); } + final String in = " (in the " + what + ")"; if (item.has("video") || item.has("fps") || item.has("max_frames")) { - throw new IllegalArgumentException( - "video/fps/max_frames are not supported for rerank input (in the " + what + ")"); - } - if (item.has("image")) { - throw new IllegalArgumentException("an 'image' in a rerank " + what - + " is ranked by llama.cpp's own server only (NativeServer); this transport ranks text"); + throw new IllegalArgumentException("video/fps/max_frames are not supported for rerank input" + in); } + final StringBuilder out = new StringBuilder(); final JsonNode text = item.path("text"); if (text.isTextual()) { - return text.asText(); + out.append(text.asText()); + } else if (text.isArray()) { + for (JsonNode part : text) { + if (!part.isTextual()) { + throw new IllegalArgumentException("\"text\" array elements must be strings" + in); + } + out.append(part.asText()); + } + } else if (!text.isMissingNode() && !text.isNull()) { + throw new IllegalArgumentException("\"text\" must be a string or an array of strings" + in); } - if (!text.isArray()) { - throw new IllegalArgumentException(shape); + // upstream decodes the images in order, so the first element decides which refusal comes first + final JsonNode image = item.path("image"); + final boolean list = image.isArray() && image.size() > 0; + if (list && !image.get(0).isTextual()) { + throw new IllegalArgumentException("\"image\" array elements must be strings" + in); } - final StringBuilder out = new StringBuilder(); - for (JsonNode part : text) { - if (!part.isTextual()) { - throw new IllegalArgumentException("'text' array elements must be strings (in the " + what + ")"); - } - out.append(part.asText()); + if (image.isTextual() || list) { + throw new IllegalArgumentException("an 'image' in a rerank " + what + + " is ranked by llama.cpp's own server only (NativeServer); this transport ranks text"); + } + if (!image.isArray() && !image.isMissingNode() && !image.isNull()) { + throw new IllegalArgumentException("\"image\" must be a string or an array of strings" + in); } return out.toString(); } diff --git a/llama/src/test/java/net/ladenthin/llama/server/OaiRerankSupportTest.java b/llama/src/test/java/net/ladenthin/llama/server/OaiRerankSupportTest.java index 9c2d38f4..f7c33290 100644 --- a/llama/src/test/java/net/ladenthin/llama/server/OaiRerankSupportTest.java +++ b/llama/src/test/java/net/ladenthin/llama/server/OaiRerankSupportTest.java @@ -88,7 +88,7 @@ public void readDocumentsRefusesATextArrayWithANonString() throws IOException { JsonNode request = read("{\"documents\":[{\"text\":[\"a\",1]}]}"); IllegalArgumentException e = assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readDocuments(request)); - assertThat(e.getMessage(), containsString("'text' array elements must be strings")); + assertThat(e.getMessage(), containsString("\"text\" array elements must be strings")); } @Test @@ -111,6 +111,54 @@ public void anImageInTheQueryIsRefusedRatherThanDropped() throws IOException { assertThat(e.getMessage(), containsString("NativeServer")); } + @Test + public void aNullOrEmptyImageIsNoImageLikeUpstream() throws IOException { + // Upstream's parse_item adds a file only for a string or a string element; null and [] add none, + // so the item is ranked by its text. A serializer that writes absent fields as null must not + // make the request fail. + String[] docs = OaiRerankSupport.readDocuments( + read("{\"documents\":[{\"text\":\"x\",\"image\":null},{\"text\":\"y\",\"image\":[]}]}")); + assertThat(docs.length, is(2)); + assertThat(docs[0], is("x")); + assertThat(docs[1], is("y")); + assertThat(OaiRerankSupport.readQuery(read("{\"query\":{\"text\":\"q\",\"image\":null}}")), is("q")); + } + + @Test + public void aMissingOrNullTextIsEmptyTextLikeUpstream() throws IOException { + String[] docs = OaiRerankSupport.readDocuments(read("{\"documents\":[{},{\"text\":null}]}")); + assertThat(docs.length, is(2)); + assertThat(docs[0], is("")); + assertThat(docs[1], is("")); + } + + @Test + public void aTextOfAnotherTypeIsRefusedLikeUpstream() throws IOException { + JsonNode request = read("{\"documents\":[{\"text\":5}]}"); + IllegalArgumentException e = + assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readDocuments(request)); + assertThat(e.getMessage(), containsString("\"text\" must be a string or an array of strings")); + } + + @Test + public void anImageOfAnotherTypeIsRefusedLikeUpstream() throws IOException { + JsonNode number = read("{\"documents\":[{\"text\":\"x\",\"image\":5}]}"); + IllegalArgumentException e = + assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readDocuments(number)); + assertThat(e.getMessage(), containsString("\"image\" must be a string or an array of strings")); + JsonNode element = read("{\"documents\":[{\"text\":\"x\",\"image\":[5]}]}"); + e = assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readDocuments(element)); + assertThat(e.getMessage(), containsString("\"image\" array elements must be strings")); + } + + @Test + public void anImageListIsRefusedRatherThanDropped() throws IOException { + JsonNode request = read("{\"documents\":[{\"text\":\"x\",\"image\":[\"data:image/png;base64,AA==\"]}]}"); + IllegalArgumentException e = + assertThrows(IllegalArgumentException.class, () -> OaiRerankSupport.readDocuments(request)); + assertThat(e.getMessage(), containsString("NativeServer")); + } + @Test public void videoIsRefusedLikeUpstream() throws IOException { for (String key : new String[] {"video", "fps", "max_frames"}) {