From be2b42688ad802f17b6d5750677260aa2a36385d Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Fri, 25 Sep 2026 22:58:10 -0300 Subject: [PATCH] Vulkan: carry ggml-org#28243 (Qwen3.8-Flash-Next MTP) on our fork branch third_party/llama.cpp-vulkan moves from ggml-org v0.5.0 to 1bit-MONSTER/llama.cpp 1bit/vulkan-upstream (62484fba): the same release plus ggml-org#28243, which adds Qwen3.8-Flash-Next's NextN/MTP draft head and fixes -md loading the target model instead of the draft file. Reviewed line by line before pinning. bump-llama-vulkan.yml now rebases the carried commits onto each new release (a commit upstream has merged drops out as empty), tags the old tip vulkan-upstream-, and pins the result. docs/vulkan.md lists what we carry; NOTICE credits the PR's authors. Verified on Strix Halo with llama-server from the branch: Qwen3-0.6B unchanged, Qwen3.8-27B + MTP unchanged (35.0/28.1/32.2 tok/s, same draft acceptance). Flash-Next + MTP is still to run here: the box had no room for its 104 GiB. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/bump-llama-vulkan.yml | 80 ++++++++++++++++++++----- .gitmodules | 3 +- NOTICE | 3 +- docs/vulkan.md | 32 +++++++++- third_party/llama.cpp-vulkan | 2 +- 5 files changed, 100 insertions(+), 20 deletions(-) diff --git a/.github/workflows/bump-llama-vulkan.yml b/.github/workflows/bump-llama-vulkan.yml index ab1f625a..6af77315 100644 --- a/.github/workflows/bump-llama-vulkan.yml +++ b/.github/workflows/bump-llama-vulkan.yml @@ -14,15 +14,20 @@ # limitations under the License. # # Keep third_party/llama.cpp-vulkan on upstream llama.cpp's latest release -# (docs/vulkan.md): when ggml-org publishes a new release, open a PR here moving -# the submodule. Releases, not master: each is a commit upstream's CI built and -# tested. GitHub-hosted CI has no GPU; run the checks in the PR body on Strix -# Halo before merging. third_party/llama.cpp (HRX) follows AMD's tested pair -# instead (bump-hrx.yml). +# (docs/vulkan.md). The submodule is 1bit-MONSTER/llama.cpp 1bit/vulkan-upstream: +# ggml-org's release plus the few upstream PRs we carry until upstream merges them +# (Qwen3.8-Flash-Next's MTP head, ggml-org#28243). When ggml-org publishes a new +# release, this workflow rebases those commits onto it (a commit upstream has since +# merged becomes empty and drops out), tags the old tip so pinned commits stay +# reachable, and opens a PR here moving the submodule. Releases, not master: each +# is a commit upstream's CI built and tested. GitHub-hosted CI has no GPU; run the +# checks in the PR body on Strix Halo before merging. third_party/llama.cpp (HRX) +# follows AMD's tested pair instead (bump-hrx.yml). # -# Uses the secret HRX_BUMP_TOKEN (Contents and Pull requests read/write on -# 1bit-MONSTER/engine): a PR opened with the default GITHUB_TOKEN would not -# start CI. +# Uses the secret HRX_BUMP_TOKEN (Contents read/write on 1bit-MONSTER/llama.cpp +# and 1bit-MONSTER/engine, Pull requests read/write on 1bit-MONSTER/engine): a PR +# opened with the default GITHUB_TOKEN would not start CI, and that token cannot +# push to the fork. name: bump-llama-vulkan on: @@ -63,10 +68,52 @@ jobs: tag=$(gh api repos/ggml-org/llama.cpp/releases/latest --jq .tag_name) upstream=$(gh api "repos/ggml-org/llama.cpp/commits/${tag}" --jq .sha) ours=$(git ls-tree HEAD third_party/llama.cpp-vulkan | awk '{print $3}') - echo "upstream ${tag} ${upstream}, ours ${ours}" + # the release our pin sits on: where our commits branch off ggml-org's history + ours_base=$(gh api "repos/ggml-org/llama.cpp/compare/${upstream}...${ours}" --jq .merge_base_commit.sha) + echo "upstream ${tag} ${upstream}, ours ${ours} on ${ours_base}" { - echo "tag=$tag"; echo "upstream=$upstream"; echo "ours=$ours" - if [ "$upstream" = "$ours" ]; then echo "changed=false"; else echo "changed=true"; fi + echo "tag=$tag"; echo "upstream=$upstream"; echo "ours=$ours"; echo "ours_base=$ours_base" + if [ "$upstream" = "$ours_base" ]; then echo "changed=false"; else echo "changed=true"; fi + } >> "$GITHUB_OUTPUT" + + - name: Rebase our commits onto the release + id: fork + if: steps.pins.outputs.changed == 'true' + env: + GH_TOKEN: ${{ secrets.HRX_BUMP_TOKEN }} + TAG: ${{ steps.pins.outputs.tag }} + UPSTREAM: ${{ steps.pins.outputs.upstream }} + OURS: ${{ steps.pins.outputs.ours }} + OURS_BASE: ${{ steps.pins.outputs.ours_base }} + run: | + set -euo pipefail + # master mirrors ggml-org, so the fork holds the release's objects + gh api -X POST repos/1bit-MONSTER/llama.cpp/merge-upstream -f branch=master > /dev/null + git init -q fork && cd fork + git remote add fork "https://x-access-token:${GH_TOKEN}@github.com/1bit-MONSTER/llama.cpp.git" + git remote add upstream https://github.com/ggml-org/llama.cpp.git + git fetch -q --filter=blob:none upstream "$UPSTREAM" + git fetch -q --filter=blob:none fork 1bit/vulkan-upstream + carried=$(git rev-parse FETCH_HEAD) + if [ "$carried" != "$OURS" ]; then + echo "::error::1bit/vulkan-upstream (${carried:0:12}) is not the engine's pin (${OURS:0:12}); reconcile them by hand" + exit 1 + fi + git fetch -q --filter=blob:none upstream "$OURS_BASE" + git checkout -q --detach "$carried" + if ! git -c user.name=llama-vulkan-bump -c user.email=llama-vulkan-bump@users.noreply.github.com \ + rebase -q --onto "$UPSTREAM" "$OURS_BASE"; then + git rebase --abort || true + echo "::error::our commits on 1bit/vulkan-upstream do not rebase onto ${TAG}; rebase them by hand" + exit 1 + fi + rebased=$(git rev-parse HEAD) + git tag "vulkan-upstream-${carried:0:12}" "$carried" + git push -q fork "refs/tags/vulkan-upstream-${carried:0:12}" + git push -q --force-with-lease="refs/heads/1bit/vulkan-upstream:$carried" fork "$rebased:refs/heads/1bit/vulkan-upstream" + { + echo "rebased=$rebased" + echo "patches<> "$GITHUB_OUTPUT" - name: Open the bump PR @@ -76,6 +123,8 @@ jobs: TAG: ${{ steps.pins.outputs.tag }} UPSTREAM: ${{ steps.pins.outputs.upstream }} OURS: ${{ steps.pins.outputs.ours }} + REBASED: ${{ steps.fork.outputs.rebased }} + PATCHES: ${{ steps.fork.outputs.patches }} run: | set -euo pipefail branch="bump-llama-vulkan/${TAG}" @@ -86,12 +135,15 @@ jobs: git config user.name "llama-vulkan-bump" git config user.email "llama-vulkan-bump@users.noreply.github.com" git switch -c "$branch" - git update-index --cacheinfo "160000,$UPSTREAM,third_party/llama.cpp-vulkan" + git update-index --cacheinfo "160000,$REBASED,third_party/llama.cpp-vulkan" git commit -q -m "Bump Vulkan llama.cpp: ggml-org/llama.cpp ${TAG} (${UPSTREAM:0:12})" git push -q origin "$branch" gh pr create --base main --head "$branch" \ --title "Bump Vulkan llama.cpp: ggml-org/llama.cpp ${TAG}" \ - --body "Moves third_party/llama.cpp-vulkan from \`${OURS:0:12}\` to upstream release [${TAG}](https://github.com/ggml-org/llama.cpp/releases/tag/${TAG}) (\`${UPSTREAM:0:12}\`, ${count} commits). + --body "Moves third_party/llama.cpp-vulkan (1bit-MONSTER/llama.cpp 1bit/vulkan-upstream) from \`${OURS:0:12}\` to \`${REBASED:0:12}\`: upstream release [${TAG}](https://github.com/ggml-org/llama.cpp/releases/tag/${TAG}) (\`${UPSTREAM:0:12}\`, ${count} upstream commits) plus the commits we carry. + + Our commits, rebased onto the release (none left means upstream has them all): + ${PATCHES:-none} CI here has no GPU. Before merging, on Strix Halo: - \`cmake -B build -G Ninja -DONEBIT_VULKAN=ON -DONEBIT_SERVE_TEST_GGUF=\$HOME/models/Qwen3-0.6B-Q4_K_M.gguf && cmake --build build && ctest --test-dir build -R serve_e2e_vulkan_upstream\`" + \`cmake -B build -G Ninja -DONEBIT_VULKAN=ON -DONEBIT_SERVE_TEST_GGUF=\$HOME/models/Qwen3-0.6B-Q4_K_M.gguf && cmake --build build && ctest --test-dir build -R serve_e2e_vulkan_upstream\`, and Qwen3.8-27B + \`--mtp\` (docs/vulkan.md)." diff --git a/.gitmodules b/.gitmodules index 1dad7983..53170bbc 100644 --- a/.gitmodules +++ b/.gitmodules @@ -57,7 +57,8 @@ shallow = true [submodule "third_party/llama.cpp-vulkan"] path = third_party/llama.cpp-vulkan - url = https://github.com/ggml-org/llama.cpp.git + url = https://github.com/1bit-MONSTER/llama.cpp.git + branch = 1bit/vulkan-upstream shallow = true [submodule "third_party/llama.cpp-rocmfpx"] path = third_party/llama.cpp-rocmfpx diff --git a/NOTICE b/NOTICE index 58f79e8c..b268113a 100644 --- a/NOTICE +++ b/NOTICE @@ -54,7 +54,8 @@ Peano, llvm-aie (third_party/llvm-aie) llama.cpp (third_party/llama.cpp: 1bit-MONSTER/llama.cpp, a fork of ggml-org/llama.cpp carrying AMD's HRX integration; third_party/llama.cpp-vulkan: -ggml-org/llama.cpp, an upstream release) +the same fork's 1bit/vulkan-upstream, an upstream release plus carried upstream +pull requests, among them ggml-org#28243 by Daniel Han and Ryan Monsurate) https://github.com/ggml-org/llama.cpp Copyright (c) 2023-2026 The ggml authors License: MIT diff --git a/docs/vulkan.md b/docs/vulkan.md index 25bfc1e2..b12dc48f 100644 --- a/docs/vulkan.md +++ b/docs/vulkan.md @@ -18,7 +18,8 @@ limitations under the License. `1bit serve --device vulkan` runs a `llama-server` built from upstream [llama.cpp](https://github.com/ggml-org/llama.cpp) (MIT), pinned to its latest -release in `third_party/llama.cpp-vulkan`. It is a separate tree from +release in `third_party/llama.cpp-vulkan`, plus the few upstream pull requests we +carry until upstream merges them ("Carried upstream PRs" below). It is a separate tree from `third_party/llama.cpp`, which stays on the llama.cpp + hrx-system pair AMD tests together for HRX ([hrx.md](hrx.md)). @@ -26,7 +27,7 @@ Two pins, because the two routes stay current in different ways: | Route | Source | Pinned to | Moved by | |---|---|---|---| -| `--device vulkan` | `third_party/llama.cpp-vulkan` | ggml-org's latest release | `bump-llama-vulkan.yml`, daily | +| `--device vulkan` | `third_party/llama.cpp-vulkan` (1bit-MONSTER/llama.cpp `1bit/vulkan-upstream`) | ggml-org's latest release + carried upstream PRs | `bump-llama-vulkan.yml`, daily | | `--device hrx` | `third_party/llama.cpp` + `third_party/hrx-system` | AMD's tested pair (ROCm/ggml-staging-automation) | `bump-hrx.yml`, daily | A new model architecture reaches the Vulkan route the day upstream releases it, @@ -34,6 +35,22 @@ without waiting for AMD's pair to move. Qwen3.8-Flash-Next (`qwen4exp`) is the case that prompted this: upstream added it on 2026-08-27, and AMD's pinned llama.cpp (`f1a0aca141de`) cannot load it. +## Carried upstream PRs + +`1bit/vulkan-upstream` is ggml-org's release plus these commits. On each new release +`bump-llama-vulkan.yml` rebases them onto it; one that upstream has merged becomes +empty and drops out, and one that no longer applies stops the bump for a hand rebase. +The commit the engine pinned before is kept as the tag `vulkan-upstream-`. + +| Upstream PR | What it adds | Carried as | +|---|---|---| +| [ggml-org#28243](https://github.com/ggml-org/llama.cpp/pull/28243) (Daniel Han, Ryan Monsurate) | Qwen3.8-Flash-Next's NextN/MTP draft head (`--spec-type draft-mtp`), and the fix for `-md` loading the target model instead of the draft file | `62484fba` on v0.5.0 | + +Reviewed line by line before it was pinned: it touches only the qwen4exp model, its +converter, and two lines of the shared speculative decoding (the `-md` path fix, and +KV sharing kept to gemma4-assistant drafts, as v0.5.0 already did). A draft head can +only change speed: the target model checks every drafted token. + ## Build Needs the Vulkan headers, loader and `glslc`. @@ -49,7 +66,9 @@ the upstream build and `--device hrx` the HRX build. Without `ONEBIT_VULKAN`, `--device vulkan` falls back to the HRX build's Vulkan backend, as before. `ONEBIT_LLAMA_SERVER` still overrides both. -## Verified (Strix Halo, v0.5.0 `7fe450e19305`) +## Verified (Strix Halo) + +v0.5.0 `7fe450e19305`: | Model | Result | |---|---| @@ -57,6 +76,13 @@ the upstream build and `--device hrx` the HRX build. Without `ONEBIT_VULKAN`, | Qwen3.8-27B UD-Q4_K_XL | "The capital of France is Paris.", 12.2 tok/s | | Qwen3.8-Flash-Next UD-Q4_K_XL (`--ctx-size 8192`) | "The capital of France is Paris.", 22.0 tok/s; the HRX pin fails to load it | +v0.5.0 + ggml-org#28243 (`62484fba`), llama-server built from the branch: + +| Model | Result | +|---|---| +| Qwen3-0.6B Q4_K_M | "The capital of France is Paris.", 315-331 tok/s (unchanged) | +| Qwen3.8-27B UD-Q4_K_XL + `--mtp` (Q4_0 head) | 35.0 / 28.1 / 32.2 tok/s code / prose / short; draft acceptance 188/200, 171/252, 6/6, the same as v0.5.0 | + Large models need `--ctx-size`: without it llama-server allocates the KV cache for the model's full trained context (262,144 tokens for Qwen3.8), which does not fit next to Flash-Next's 104 GiB of weights. diff --git a/third_party/llama.cpp-vulkan b/third_party/llama.cpp-vulkan index 7fe450e1..62484fba 160000 --- a/third_party/llama.cpp-vulkan +++ b/third_party/llama.cpp-vulkan @@ -1 +1 @@ -Subproject commit 7fe450e19305b828c199d602c23a8337aaa1f03b +Subproject commit 62484fba8e459b60cc886dba07b28787bfe33dc4