Vulkan: carry ggml-org#28243 (Qwen3.8-Flash-Next MTP) on our fork branch

.github/workflows/bump-llama-vulkan.yml

@@ -14,15 +14,20 @@
 # limitations under the License.
 #
 # Keep third_party/llama.cpp-vulkan on upstream llama.cpp's latest release
-# (docs/vulkan.md): when ggml-org publishes a new release, open a PR here moving
-# the submodule. Releases, not master: each is a commit upstream's CI built and
-# tested. GitHub-hosted CI has no GPU; run the checks in the PR body on Strix
-# Halo before merging. third_party/llama.cpp (HRX) follows AMD's tested pair
-# instead (bump-hrx.yml).
+# (docs/vulkan.md). The submodule is 1bit-MONSTER/llama.cpp 1bit/vulkan-upstream:
+# ggml-org's release plus the few upstream PRs we carry until upstream merges them
+# (Qwen3.8-Flash-Next's MTP head, ggml-org#28243). When ggml-org publishes a new
+# release, this workflow rebases those commits onto it (a commit upstream has since
+# merged becomes empty and drops out), tags the old tip so pinned commits stay
+# reachable, and opens a PR here moving the submodule. Releases, not master: each
+# is a commit upstream's CI built and tested. GitHub-hosted CI has no GPU; run the
+# checks in the PR body on Strix Halo before merging. third_party/llama.cpp (HRX)
+# follows AMD's tested pair instead (bump-hrx.yml).
 #
-# Uses the secret HRX_BUMP_TOKEN (Contents and Pull requests read/write on
-# 1bit-MONSTER/engine): a PR opened with the default GITHUB_TOKEN would not
-# start CI.
+# Uses the secret HRX_BUMP_TOKEN (Contents read/write on 1bit-MONSTER/llama.cpp
+# and 1bit-MONSTER/engine, Pull requests read/write on 1bit-MONSTER/engine): a PR
+# opened with the default GITHUB_TOKEN would not start CI, and that token cannot
+# push to the fork.
 name: bump-llama-vulkan
 
 on:
@@ -63,10 +68,52 @@ jobs:
           tag=$(gh api repos/ggml-org/llama.cpp/releases/latest --jq .tag_name)
           upstream=$(gh api "repos/ggml-org/llama.cpp/commits/${tag}" --jq .sha)
           ours=$(git ls-tree HEAD third_party/llama.cpp-vulkan | awk '{print $3}')
-          echo "upstream ${tag} ${upstream}, ours ${ours}"
+          # the release our pin sits on: where our commits branch off ggml-org's history
+          ours_base=$(gh api "repos/ggml-org/llama.cpp/compare/${upstream}...${ours}" --jq .merge_base_commit.sha)
+          echo "upstream ${tag} ${upstream}, ours ${ours} on ${ours_base}"
           {
-            echo "tag=$tag"; echo "upstream=$upstream"; echo "ours=$ours"
-            if [ "$upstream" = "$ours" ]; then echo "changed=false"; else echo "changed=true"; fi
+            echo "tag=$tag"; echo "upstream=$upstream"; echo "ours=$ours"; echo "ours_base=$ours_base"
+            if [ "$upstream" = "$ours_base" ]; then echo "changed=false"; else echo "changed=true"; fi
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Rebase our commits onto the release
+        id: fork
+        if: steps.pins.outputs.changed == 'true'
+        env:
+          GH_TOKEN: ${{ secrets.HRX_BUMP_TOKEN }}
+          TAG: ${{ steps.pins.outputs.tag }}
+          UPSTREAM: ${{ steps.pins.outputs.upstream }}
+          OURS: ${{ steps.pins.outputs.ours }}
+          OURS_BASE: ${{ steps.pins.outputs.ours_base }}
+        run: |
+          set -euo pipefail
+          # master mirrors ggml-org, so the fork holds the release's objects
+          gh api -X POST repos/1bit-MONSTER/llama.cpp/merge-upstream -f branch=master > /dev/null
+          git init -q fork && cd fork
+          git remote add fork "https://x-access-token:${GH_TOKEN}@github.com/1bit-MONSTER/llama.cpp.git"
+          git remote add upstream https://github.com/ggml-org/llama.cpp.git
+          git fetch -q --filter=blob:none upstream "$UPSTREAM"
+          git fetch -q --filter=blob:none fork 1bit/vulkan-upstream
+          carried=$(git rev-parse FETCH_HEAD)
+          if [ "$carried" != "$OURS" ]; then
+            echo "::error::1bit/vulkan-upstream (${carried:0:12}) is not the engine's pin (${OURS:0:12}); reconcile them by hand"
+            exit 1
+          fi
+          git fetch -q --filter=blob:none upstream "$OURS_BASE"
+          git checkout -q --detach "$carried"
+          if ! git -c user.name=llama-vulkan-bump -c user.email=llama-vulkan-bump@users.noreply.github.com \
+               rebase -q --onto "$UPSTREAM" "$OURS_BASE"; then
+            git rebase --abort || true
+            echo "::error::our commits on 1bit/vulkan-upstream do not rebase onto ${TAG}; rebase them by hand"
+            exit 1
+          fi
+          rebased=$(git rev-parse HEAD)
+          git tag "vulkan-upstream-${carried:0:12}" "$carried"
+          git push -q fork "refs/tags/vulkan-upstream-${carried:0:12}"
+          git push -q --force-with-lease="refs/heads/1bit/vulkan-upstream:$carried" fork "$rebased:refs/heads/1bit/vulkan-upstream"
+          {
+            echo "rebased=$rebased"
+            echo "patches<<EOF"; git log --format='- `%h` %s' "$UPSTREAM..$rebased"; echo "EOF"
           } >> "$GITHUB_OUTPUT"
 
       - name: Open the bump PR
@@ -76,6 +123,8 @@ jobs:
           TAG: ${{ steps.pins.outputs.tag }}
           UPSTREAM: ${{ steps.pins.outputs.upstream }}
           OURS: ${{ steps.pins.outputs.ours }}
+          REBASED: ${{ steps.fork.outputs.rebased }}
+          PATCHES: ${{ steps.fork.outputs.patches }}
         run: |
           set -euo pipefail
           branch="bump-llama-vulkan/${TAG}"
@@ -86,12 +135,15 @@ jobs:
           git config user.name "llama-vulkan-bump"
           git config user.email "llama-vulkan-bump@users.noreply.github.com"
           git switch -c "$branch"
-          git update-index --cacheinfo "160000,$UPSTREAM,third_party/llama.cpp-vulkan"
+          git update-index --cacheinfo "160000,$REBASED,third_party/llama.cpp-vulkan"
           git commit -q -m "Bump Vulkan llama.cpp: ggml-org/llama.cpp ${TAG} (${UPSTREAM:0:12})"
           git push -q origin "$branch"
           gh pr create --base main --head "$branch" \
             --title "Bump Vulkan llama.cpp: ggml-org/llama.cpp ${TAG}" \
-            --body "Moves third_party/llama.cpp-vulkan from \`${OURS:0:12}\` to upstream release [${TAG}](https://github.com/ggml-org/llama.cpp/releases/tag/${TAG}) (\`${UPSTREAM:0:12}\`, ${count} commits).
+            --body "Moves third_party/llama.cpp-vulkan (1bit-MONSTER/llama.cpp 1bit/vulkan-upstream) from \`${OURS:0:12}\` to \`${REBASED:0:12}\`: upstream release [${TAG}](https://github.com/ggml-org/llama.cpp/releases/tag/${TAG}) (\`${UPSTREAM:0:12}\`, ${count} upstream commits) plus the commits we carry.
+
+          Our commits, rebased onto the release (none left means upstream has them all):
+          ${PATCHES:-none}
 
           CI here has no GPU. Before merging, on Strix Halo:
-          \`cmake -B build -G Ninja -DONEBIT_VULKAN=ON -DONEBIT_SERVE_TEST_GGUF=\$HOME/models/Qwen3-0.6B-Q4_K_M.gguf && cmake --build build && ctest --test-dir build -R serve_e2e_vulkan_upstream\`"
+          \`cmake -B build -G Ninja -DONEBIT_VULKAN=ON -DONEBIT_SERVE_TEST_GGUF=\$HOME/models/Qwen3-0.6B-Q4_K_M.gguf && cmake --build build && ctest --test-dir build -R serve_e2e_vulkan_upstream\`, and Qwen3.8-27B + \`--mtp\` (docs/vulkan.md)."

.gitmodules

@@ -57,7 +57,8 @@
 	shallow = true
 [submodule "third_party/llama.cpp-vulkan"]
 	path = third_party/llama.cpp-vulkan
-	url = https://github.com/ggml-org/llama.cpp.git
+	url = https://github.com/1bit-MONSTER/llama.cpp.git
+	branch = 1bit/vulkan-upstream
 	shallow = true
 [submodule "third_party/llama.cpp-rocmfpx"]
 	path = third_party/llama.cpp-rocmfpx

NOTICE

@@ -54,7 +54,8 @@ Peano, llvm-aie (third_party/llvm-aie)
 
 llama.cpp (third_party/llama.cpp: 1bit-MONSTER/llama.cpp, a fork of
 ggml-org/llama.cpp carrying AMD's HRX integration; third_party/llama.cpp-vulkan:
-ggml-org/llama.cpp, an upstream release)
+the same fork's 1bit/vulkan-upstream, an upstream release plus carried upstream
+pull requests, among them ggml-org#28243 by Daniel Han and Ryan Monsurate)
   https://github.com/ggml-org/llama.cpp
   Copyright (c) 2023-2026 The ggml authors
   License: MIT

docs/vulkan.md

@@ -18,22 +18,39 @@ limitations under the License.
 
 `1bit serve --device vulkan` runs a `llama-server` built from upstream
 [llama.cpp](https://github.com/ggml-org/llama.cpp) (MIT), pinned to its latest
-release in `third_party/llama.cpp-vulkan`. It is a separate tree from
+release in `third_party/llama.cpp-vulkan`, plus the few upstream pull requests we
+carry until upstream merges them ("Carried upstream PRs" below). It is a separate tree from
 `third_party/llama.cpp`, which stays on the llama.cpp + hrx-system pair AMD
 tests together for HRX ([hrx.md](hrx.md)).
 
 Two pins, because the two routes stay current in different ways:
 
 | Route | Source | Pinned to | Moved by |
 |---|---|---|---|
-| `--device vulkan` | `third_party/llama.cpp-vulkan` | ggml-org's latest release | `bump-llama-vulkan.yml`, daily |
+| `--device vulkan` | `third_party/llama.cpp-vulkan` (1bit-MONSTER/llama.cpp `1bit/vulkan-upstream`) | ggml-org's latest release + carried upstream PRs | `bump-llama-vulkan.yml`, daily |
 | `--device hrx` | `third_party/llama.cpp` + `third_party/hrx-system` | AMD's tested pair (ROCm/ggml-staging-automation) | `bump-hrx.yml`, daily |
 
 A new model architecture reaches the Vulkan route the day upstream releases it,
 without waiting for AMD's pair to move. Qwen3.8-Flash-Next (`qwen4exp`) is the
 case that prompted this: upstream added it on 2026-08-27, and AMD's pinned
 llama.cpp (`f1a0aca141de`) cannot load it.
 
+## Carried upstream PRs
+
+`1bit/vulkan-upstream` is ggml-org's release plus these commits. On each new release
+`bump-llama-vulkan.yml` rebases them onto it; one that upstream has merged becomes
+empty and drops out, and one that no longer applies stops the bump for a hand rebase.
+The commit the engine pinned before is kept as the tag `vulkan-upstream-<sha12>`.
+
+| Upstream PR | What it adds | Carried as |
+|---|---|---|
+| [ggml-org#28243](https://github.com/ggml-org/llama.cpp/pull/28243) (Daniel Han, Ryan Monsurate) | Qwen3.8-Flash-Next's NextN/MTP draft head (`--spec-type draft-mtp`), and the fix for `-md` loading the target model instead of the draft file | `62484fba` on v0.5.0 |
+
+Reviewed line by line before it was pinned: it touches only the qwen4exp model, its
+converter, and two lines of the shared speculative decoding (the `-md` path fix, and
+KV sharing kept to gemma4-assistant drafts, as v0.5.0 already did). A draft head can
+only change speed: the target model checks every drafted token.
+
 ## Build
 
 Needs the Vulkan headers, loader and `glslc`.
@@ -49,14 +66,23 @@ the upstream build and `--device hrx` the HRX build. Without `ONEBIT_VULKAN`,
 `--device vulkan` falls back to the HRX build's Vulkan backend, as before.
 `ONEBIT_LLAMA_SERVER` still overrides both.
 
-## Verified (Strix Halo, v0.5.0 `7fe450e19305`)
+## Verified (Strix Halo)
+
+v0.5.0 `7fe450e19305`:
 
 | Model | Result |
 |---|---|
 | `tests/serve_e2e.sh` (Qwen3-0.6B Q4_K_M) | PASS (`serve_e2e_vulkan_upstream`) |
 | Qwen3.8-27B UD-Q4_K_XL | "The capital of France is Paris.", 12.2 tok/s |
 | Qwen3.8-Flash-Next UD-Q4_K_XL (`--ctx-size 8192`) | "The capital of France is Paris.", 22.0 tok/s; the HRX pin fails to load it |
 
+v0.5.0 + ggml-org#28243 (`62484fba`), llama-server built from the branch:
+
+| Model | Result |
+|---|---|
+| Qwen3-0.6B Q4_K_M | "The capital of France is Paris.", 315-331 tok/s (unchanged) |
+| Qwen3.8-27B UD-Q4_K_XL + `--mtp` (Q4_0 head) | 35.0 / 28.1 / 32.2 tok/s code / prose / short; draft acceptance 188/200, 171/252, 6/6, the same as v0.5.0 |
+
 Large models need `--ctx-size`: without it llama-server allocates the KV cache
 for the model's full trained context (262,144 tokens for Qwen3.8), which does not
 fit next to Flash-Next's 104 GiB of weights.