models: Qwen3.8-Flash-Next MTP by danielhanchen · Pull Request #28243 · ggml-org/llama.cpp

danielhanchen · 2026-09-02T12:30:25Z

Overview

Enables 1.3 to 2x faster MTP support for Qwen3.8-Flash-Next + shared MTP modules (re-uses embed_tokens to save disk space and VRAM / RAM). Builds on top of #27836

Additional information

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Used Claude + local models for verification and testing

ggerganov · 2026-09-02T13:28:36Z

+        // target for a draft head that declares nextn_shared_target_tensors; must outlive this model
+        const struct llama_model * model_shared;


The llama_context_params already support ctx_other. Reuse that logic instead of adding this.

ggerganov · 2026-09-02T13:32:17Z

+// a captured graph hard-codes its shapes, so with one key per split an alternating shape
+// (a speculative verify batch) resets warmup forever. O(1) on purpose: walking nodes undoes the
+// point of a cuda graph. A shape this fails to separate re-captures as before, so it cannot regress.
+static uint64_t ggml_cuda_graph_get_key(ggml_cgraph * cgraph) {
+    // unlike the previous key this dereferences nodes[0], so an empty graph is not safe here
+    if (cgraph->n_nodes <= 0) {
+        return 0;
+    }
+
+    uint64_t key = (uint64_t) (uintptr_t) cgraph->nodes[0];
+
+    auto mix = [&key](uint64_t v) {
+        key = (key ^ v) * 0x100000001b3ull;
+    };
+
+    mix(cgraph->n_nodes);
+
+    for (int d = 0; d < GGML_MAX_DIMS; d++) {
+        mix(cgraph->nodes[0]->ne[d]);
+        mix(cgraph->nodes[cgraph->n_nodes - 1]->ne[d]);
+    }
+
+    return key;


Better keep the CUDA changes for a follow-up PR.

danielhanchen · 2026-09-02T14:42:24Z

@ggerganov thanks will check when I wake :))

jesdga95 · 2026-09-02T21:43:26Z

Works great on dual consumer GPUs (5090 + 5060 Ti, 32 GB RAM): +25-40%, acceptance 0.77

9950X, 32 GB DDR5, RTX 5090 + RTX 5060 Ti (gen5 x8/x8), Ubuntu, CUDA 13.3, PR @ 2857e51.
Target: unsloth UD-IQ4_XS (auto-fit, host experts partly in page cache). Draft: shared Q4_K_M sidecar, dn-max 2, q8 KV, 65k ctx.

config	t/s
no draft	36.4
MTP, 28k mixed gen (thinking + code)	45.3 avg
MTP, code sections	48-54
MTP, prose	27-28

Acceptance 0.767 (16849/21960), mean len 2.53.

One minor issue I encountered: the fitter can't measure the shared head (borrow_shared_tensor refusal at launch), so the draft loads unbudgeted and OOMs on the fuller GPU. Workaround: --fit-target to reserve room for it.

Thanks for all of your hard work!

llbzow · 2026-09-03T01:09:13Z

Validation: 2× RTX A6000 48GB (PCIe, no NVLink), Windows Server 2019, CUDA 12.6 (sm_86) — both heads draft at ~0.37 acceptance, no config beats baseline

Built this PR @ 2857e5114 with @JayToltTech's LLAMA_STATE_SEQ_FLAGS_ON_DEVICE checkpoint patch cherry-picked on top (applies cleanly, one commit).

Target: unsloth/Qwen3.8-Flash-Next-UD-Q3_K_XL, -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -ot per_layer_token_embd.weight=CPU -c 524288 (YaRN ×2) -t 24 -b 2048 -ub 512. No-spec baseline on identical prompts: 41–42.5 tok/s decode.

Arm 1 — shared Q4_K_M sidecar, -ngld 99 --spec-type draft-mtp --spec-draft-n-max 3: loads and drafts, but acceptance is only 0.32–0.38 (mean len ≈ 2.1), even with temp 0 / top_k 1. Decode 23–24 t/s @ temp 0.6, best greedy run 39.6 t/s — never beats baseline.

Arm 2 — same but without -ot per_layer_token_embd.weight=CPU: draft load crashes: invalid vector subscript immediately after loading draft model. So on this setup the shared-embed borrow path only works when the target's embed tensor is CPU-resident; GPU-resident embed breaks the load entirely. (Possibly related: arm 1's low acceptance could also be a symptom of the borrow path misbehaving with the UD-Q3_K_XL embed.)

Arm 3 — standalone Q8_0 grafted head (the drluoto file that gave 82–95% acceptance on #27836 @ 1d8de7c1, same target, same box): first OOM'd at 512K (draft KV cache wants 1 GiB on device 1; the 90 GB target leaves ~6–9 GiB free). Re-tested at -c 65536: loads, drafts, but acceptance is again 0.36–0.39 even greedy — same collapse as the shared head. Greedy decode 40–42 t/s ≈ baseline, temp 0.6 drops to 25–28 t/s.

Same draft file, same target, same prompts: 0.82–0.95 acceptance on #27836 vs ~0.37 on this branch — so something between 1d8de7c1 and 2857e5114 (this PR's changes or the newer post-#27941 master base) appears to regress draft-mtp acceptance on this hardware/quant combo. Happy to rerun any fix quickly; the box rebuilds in ~6 min.

Side note: draft compute buffer (2253 MiB) initially OOMs on device 1 next to the 90 GB target, recovered by retrying without pipeline parallelism — harmless here, but worth knowing for tight-VRAM multi-GPU setups.

The CUDA graph cache keyed captured graphs by the raw memory address of their first node (cgraph->nodes[0]). A captured graph hard-codes its shapes, but speculative decoding constantly alternates between different batch shapes (draft steps, verify batches, catch-up decodes) on the same context - when a new shape happens to reuse the same first- node address as a stale cached graph for a different shape, capture either reuses the wrong graph or thrashes, permanently resetting warmup instead of ever converging to steady-state replay. Hash node count and both endpoint tensors' shapes into the key instead (O(1) - walking all nodes would defeat the point of a CUDA graph), add LRU eviction capped at 64 graphs so the map can't grow unbounded now that distinct shapes get distinct entries. A shape this still fails to separate re-captures exactly as before, so it can't regress anything. Cherry-picked from ggml-org#28243 (open, unmerged), which found this while working on qwen4exp MTP performance - the effect is generic to any speculative-decoding workload on this fork, not qwen4exp-specific, so pulling in just this piece rather than the rest of that PR (which also reworks qwen4exp trunk/draft tensor sharing and doesn't fix the mixer/export-timing bugs already fixed on this branch). Verified: qwen4exp MTP speculative decoding still produces correct, byte-identical (temp 0) output after this change. Speed effect is hard to isolate cleanly from the dominant MoE-cache warmup effect already documented on this branch, but the fix is justified on its own correctness merits regardless of measured delta. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

sdroege · 2026-09-03T06:12:57Z

+        throw std::runtime_error(format("%s: this model is a draft head without its own '%s'; "
+                    "load it as a draft of its target model, not on its own", __func__, name.c_str()));


This triggers when loading with --fit on as the draft model is loaded independently as part of that. Everything still works though: it's later loaded again together with the target model.

ovidiu-morar · 2026-09-03T06:32:48Z

Metal data point on this exact head (`2857e51`): dn=2 is greedy-identical, dn=5 is slower and diverges

All validations above are CUDA, so here is Apple Silicon. M5 Pro 64 GB, Metal, built at 2857e5114 (the same commit as the fork branch the Unsloth docs point to). Target: Qwen3.8-Flash-Next-UD-IQ3_XXS (original 3-shard file). Draft: the official mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf. llama-cli -ngl 48 -ot "^output=CPU" -c 2048 -n 200 --temp 0 --seed 7 --no-warmup --single-turn, same Romanian prompt for every run, 3 interleaved runs per arm, medians. Identity = sha256 of the generated token stream.

arm	gen t/s	vs no-spec	token stream at temp 0
no spec	26.8		reference `dea1d100`
`--spec-type draft-mtp --spec-draft-n-max 2`	26.3	-2%	identical
`--spec-draft-n-max 5` (the value the model page recommends)	22.1	-18%	diverges (`a19b4630`, 3/3 runs)

Three things worth separating:

The divergence is not new to this head. I measured the same greedy divergence at n-max >= 3 on qwen4exp : add NextN/MTP draft head (--spec-type draft-mtp) for Qwen3.8-Flash-Next #27836 (Aug 30, report) and on the fork at 586b15e (Sept 1), and the divergent stream here is byte-identical to the one from Sept 1. The reference stream is also byte-identical across the two heads, so the target itself has not moved. The diff between 586b15e and 2857e5114 does not touch the accept/reject path (common_sampler_sample_and_accept_n is unchanged; the changes are the shared-embedding borrow, the DFlash fusion and the Metal tuning tables), which is consistent with the result.
On this hardware the page's recommended setting is both slower and not greedy-identical, while n-max 2 is identical and roughly neutral. That matches the page's own caveat about smaller gains on lower-bandwidth Macs, and it may be worth stating n-max 2 as the safe default there until the n-max >= 3 divergence is understood.
Every report so far, here and on the model page, measures speed and acceptance. None checks identity. If someone on CUDA or Vulkan can run the same prompt at temp 0 with and without the draft at n-max 5 and compare the token streams, that would tell whether the divergence is Metal-specific or just unmeasured elsewhere.

Happy to rerun on any later head; the harness is three commands.

jeverson-bmi · 2026-09-03T12:07:23Z

I have been trying this and the baseline from master actually has faster token gen than with MTP on, both shared and standalone. Using 5090+A6000(Ampere) + 128GB DDR5.

Adds the MTP head's own hyper-connection mixer tensor names and lists the NextN tensors under the qwen4exp architecture.

Adds --spec-type draft-mtp support for Qwen3.8-Flash-Next. The MTP head folds the next token's embedding into the trunk's wide hyper-connection residual, runs one trunk-style block (dense attention + MoE) over it, and collapses the result with its own mixer before reusing the trunk's LM head. - read nextn_predict_layers so n_layer() excludes the MTP block - load the trailing block through the existing trunk path: is_recr() and is_ple() are already false past the trunk, so it needs no special casing - eh_proj fuses the checkpoint's fc_embedding and fc_hidden side by side, so one matmul computes fc_embedding@e + fc_hidden@h - the head carries its own hyper-connection mixer, mirroring the trunk's hc_head_*, which stands in for the output norm qwen4exp does not have - export the wide pre-collapse residual as t_h_nextn from both graphs, so the driver can feed it back for the next draft step - route MTP contexts to a plain KV cache filtered to the trailing layer The draft block attends densely for now: the trunk's QSA only prunes context past a 2048-token budget, so dense is a numerical superset and drafts are verified either way. Indexer tensors are still loaded.

The MTP block is one trunk-shaped block (dense attention + MoE wrapped in hyper-connections) plus a head-level combiner, so once _QwenMtpMixin renames mtp.layers.0.* to the trailing block index its tensors ride the existing qwen4exp mappings unchanged. Two head-level pieces need handling: - fc_embedding and fc_hidden fuse into the eh_proj the shared NextN code expects, since W_e@e + W_h@h == [W_e|W_h] @ concat(e, h) - mtp.hyper_connection_mixer.* is the head's own copy of the trunk's hc_head_* output mixer, unindexed in the checkpoint and per-block in the GGUF compress_ratios is read with length block_count, so it gains a trailing 0 for the MTP block, which attends densely. --no-nextn drops the head; --mtp exports it on its own.

A NextN/MTP draft exported with --mtp carries the token embeddings, output norm and lm head so it can be loaded as a standalone model. For every current sidecar those three tensors are most of the file: ggml-org/Qwen3.8-27B-GGUF mtp-Qwen3.8-27B-Q4_0.gguf is 1.565 GiB, of which 1.332 GiB (85%) is the copy, against 0.223 GiB for the MTP block itself. Add an opt-in --mtp-shared-embd that leaves them out and marks the file with nextn_shared_target_tensors. The loader then resolves those names against the already loaded target model. The graph side needs no change: the nextn blocks of twelve archs already fall back to model.tok_embd and model.output. The borrow is gated on the new key, so a sidecar published before this change cannot reach it and keeps its current behaviour. Shapes are checked against the target and a mismatch is refused, as is loading such a file on its own.

_QwenMtpMixin is not a ModelBase subclass, so it re-declares the attributes it reads off cls for the type checker. filter_tensors reads cls.mtp_shared_embd without a matching declaration, which ty reports as unresolved-attribute. The declaration is a bare annotation, matching no_mtp and mtp_only above it. That creates no class attribute, so it cannot shadow ModelBase.mtp_shared_embd even though the mixin precedes the model class in the MRO; a default value here would have. Assisted-by: Claude

A draft-only export declares the full block count but ships the MTP block alone, so the trunk tensors load as null and only the MTP graph is buildable. Context reservation builds the trunk graph, which walked those nulls and segfaulted. A shared-embedding draft is caught earlier by the borrow check, since it has no token_embd of its own. A self-contained draft keeps one, so it passed that check and reached here. Assisted-by: Claude

Reuses the existing llama_context_params::ctx_other plumbing instead of adding a model_shared load parameter and a nextn_shared_target_tensors metadata key. A draft-only export now simply omits token_embd/output, and the qwen4exp MTP graph resolves them against the target context at graph build, following dflash and gemma4-assistant. Loading such a file on its own reports that it needs -md. Assisted-by: Claude

Assisted-by: Claude

gabrielfreire · 2026-09-04T14:32:35Z

Pulled the latest commits on this PR and started getting this error in the logs and it seems MTP wasn't activated

45.08.275.530 E decode: failed to initialize batch
45.08.275.531 E llama_decode: failed to decode, ret = -1
45.08.275.532 E spec        draft: llama_decode[1] returned -1
45.08.371.747 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84307
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84307
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.371.752 E decode: failed to initialize batch
45.08.371.753 E llama_decode: failed to decode, ret = -1
45.08.371.754 E spec        draft: llama_decode[1] returned -1
45.08.469.748 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84309
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84309
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.469.752 E decode: failed to initialize batch
45.08.469.752 E llama_decode: failed to decode, ret = -1
45.08.469.754 E spec        draft: llama_decode[1] returned -1
45.08.566.593 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84311
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84311
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.566.597 E decode: failed to initialize batch
45.08.566.598 E llama_decode: failed to decode, ret = -1
45.08.566.599 E spec        draft: llama_decode[1] returned -1
45.08.669.020 I slot print_timing: id  0 | task 17766 | n_gen =    100, tg =  15.11 t/s, tg_3s =  15.26 t/s
45.08.673.694 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84312
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84312
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.673.697 E decode: failed to initialize batch
45.08.673.698 E llama_decode: failed to decode, ret = -1
45.08.673.699 E spec        draft: llama_decode[1] returned -1
45.08.768.871 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 84314
 - the tokens for sequence 0 in the input batch have a starting position of Y = 84314
 for M-RoPE, it is required that the position satisfies: X < Y
45.08.768.875 E decode: failed to initialize batch
45.08.768.876 E llama_decode: failed to decode, ret = -1
45.08.768.877 E spec        draft: llama_decode[1] returned -1
45.08.859.902 I slot print_timing: id  0 | task 17766 | prompt eval time =    6649.72 ms /   178 tokens (   37.36 ms per token,    26.77 tokens per second)
45.08.859.907 I slot print_timing: id  0 | task 17766 |        eval time =    6741.95 ms /   103 tokens (   66.10 ms per token,    15.13 tokens per second)
45.08.859.908 I slot print_timing: id  0 | task 17766 |       total time =   13391.67 ms /   281 tokens
45.08.859.909 I slot print_timing: id  0 | task 17766 |    graphs reused =      11160
45.08.859.913 I slot print_timing: id  0 | task 17766 | draft acceptance = 0.81395 (   35 accepted /    43 generated), mean len =  1.81
45.08.861.454 I slot      release: id  0 | task 17766 | stop processing: n_tokens = 84316, truncated = 0

Volunteer-1 · 2026-09-18T13:41:23Z

Thank you danielhanchen for the rebasing!

Please verify whether the fourth commit 3fb9b98

qwen4exp : support draft-head-only GGUFs (unsloth layout)
Draft-head files carry no trunk tensors and no hc_head mixer; they use
nextn.shared_head_norm plus the block's own ffn projections instead.
Detect such files (n_layer_nextn > 0 and no blk.0 hc_attn_norm), make the
trunk hc_head tensors optional for them, and fall back in the MTP graph.
Also fix the draft load using the target path instead of the draft path.

from #28097 is still required after this PR. It fixes two bugs: erroneous search for the draft tensors, erroneous search path for the draft file.

danielhanchen · 2026-09-18T22:02:32Z

Will address issues and review :)

Upstream PR ggml-org#28243 adds shared-MTP support for qwen4exp and keeps unsloth's original naming: eh_proj stays whole, and the mixer tensors stay hc_head_norm/down/up. This fork splits eh_proj and renames the mixer, which is what scripts/mtp/mtp_sidecar.py converts to. If that PR lands, unsloth's sidecar loads on upstream unconverted and the head published alongside this fork does not. Nothing breaks today, but the names should not be treated as standard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Two conflicts. src/llama-model.cpp is operand order only, identical semantics. src/models/models.h: the fork declared a protected graph(..., bool mtp) constructor that nothing defined or called any more (the fork lost its qwen4exp graph_mtp in the 0190529 upstream merge and kept only the declaration), so the PR's no_build_t tag constructor replaces it rather than coexisting with a dead overload.

… names A draft-only export from the ROCmFPx fork (agentionai Qwen3.8-Flash-Next-MTP- ROCmFP4-FAST) has no blk.N.nextn.hc_head_* tensors: that converter wrote the MTP block's hyper-connection mixer as output_hc_norm/down/up, since the file has no trunk of its own, and its graph_mtp reads model.hc_head_*. Upstream's head (ggml-org#28243) loads nextn.hc_head_* and failed with "tensor not found". When the file is draft-only and the nextn copies are absent, load them as not required and point the MTP layer at the trunk-named tensors instead. The unsloth shared-Q8_0 head, which carries nextn.hc_head_*, is unaffected. With this head on gfx1151 (Vulkan, cm1 fixed), fixed n_max 4: 33.2 / 61.6 / 52.2 t/s prose / json / code at acceptance 0.38 / 0.94 / 0.75; the fpx binary with the same pairing does 27.9 / 51.8 / 43.3.

…gml-org#28243)

Assisted-by: Claude

…card Add artifacts/unavailable.json and render support so a reference cell the host cannot produce for hardware reasons (llama.cpp OOM-killed at -c 133760 with the 104 GiB model) reads n/a instead of TODO; missing software features such as the MTP sidecar (ggml-org/llama.cpp#28243) stay TODO. Drop prose that restates the tables and the loading commands.

CISC · 2026-09-22T09:07:02Z

+    _MTP_MIXER_PREFIX = "mtp.hyper_connection_mixer."
+
+    @classmethod
+    def filter_tensors(cls, item):
+        name, gen = item
+        _, _, mixer_suffix = name.partition(cls._MTP_MIXER_PREFIX)


Suggested change

_MTP_MIXER_PREFIX = "mtp.hyper_connection_mixer."

@classmethod

def filter_tensors(cls, item):

name, gen = item

_, _, mixer_suffix = name.partition(cls._MTP_MIXER_PREFIX)

@classmethod

def filter_tensors(cls, item):

name, gen = item

_, _, mixer_suffix = name.partition("mtp.hyper_connection_mixer.")

CISC · 2026-09-22T09:17:58Z

+    // the trunk must keep at least one block: n_layer() == n_layer_all - n_layer_nextn
+    if (hparams.n_layer_nextn >= hparams.n_layer_all) {
+        throw std::runtime_error(format("%s must be less than %s, got %u",
+                                        ml.llm_kv(LLM_KV_NEXTN_PREDICT_LAYERS).c_str(),
+                                        ml.llm_kv(LLM_KV_BLOCK_COUNT).c_str(), hparams.n_layer_nextn));
+    }
+
    ml.get_key_or_arr(LLM_KV_EXPERT_FEED_FORWARD_LENGTH, hparams.n_ff_exp_arr, hparams.n_layer_all, false);


Suggested change

// the trunk must keep at least one block: n_layer() == n_layer_all - n_layer_nextn

if (hparams.n_layer_nextn >= hparams.n_layer_all) {

throw std::runtime_error(format("%s must be less than %s, got %u",

ml.llm_kv(LLM_KV_NEXTN_PREDICT_LAYERS).c_str(),

ml.llm_kv(LLM_KV_BLOCK_COUNT).c_str(), hparams.n_layer_nextn));

}

ml.get_key_or_arr(LLM_KV_EXPERT_FEED_FORWARD_LENGTH, hparams.n_ff_exp_arr, hparams.n_layer_all, false);

ml.get_key_or_arr(LLM_KV_EXPERT_FEED_FORWARD_LENGTH, hparams.n_ff_exp_arr, hparams.n_layer_all, false);

It can only be <= at this point, and since any incorrect value would lead to a non-working model I don't see any reason to check for equality (which would just not process any layers at all).

Build llama.cpp from the open Qwen3.8-Flash-Next MTP pull request (ggml-org/llama.cpp#28243, commit 6fcaa16f) as a parallel devshell package exposed as llama-server-mtp; bench.json's speculative.reference.server selects it for the speculative cells only, and the card says so. Gufo MTP matches llama.cpp MTP within 2–11% on prose at d0/4K and leads by 5–36% on the repetitive workload; llama.cpp's MTP server is OOM-killed from the 12K prefix and at C6–C8, which the driver now records as unavailable so render prints n/a for those cells automatically. Add the image-encoder runner: prefill time of a request carrying a gradient PNG on both servers with the same BF16 projector; Gufo is +151% at 256x256 and +55% at 1024x1024. Download the Flash-Next projector into the model directory. Restore the aggregate delivered tok/s metric in render (a subagent had switched it to per-request decode-rate sums) and re-render the Qwen3.8 27B concurrency tables on it.

ggerganov · 2026-09-22T18:11:13Z

+    parser.add_argument(
+        "--mtp-shared-embd", action="store_true",
+        help="With --mtp, leave the token embeddings, output norm and LM head out of the draft and take them from the target model at load time. Much smaller draft, but it needs a llama.cpp new enough to read it.",
+    )


How much space do we save? My estimate is about 1GB of Q8_0?

If this is correct, my recommendation is to remove this sharing functionality in order to reduce the complexity. It is not worth it and additionally, am efficient MTP setup actually benefits from not sharing the target tensors and instead using fast low-bit quantizations such as Q4_0 for the token embeddings and the lm head.

I also don’t think we need to set this explicitly for MTP. I feel it should be less than 1 GB.

During MTP conversion, we can export the MTP head either together with the target model or separately. In the former case, the MTP head does not need its own token_embd or lm_head, since it can access them later through ctx_other. In the latter case, the exported one will be a self-contained draft model by design.

Squash of upstream PR ggml-org#28243 (net diff vs bb3c853), including the 09-21 review fixes: draft load path, nextn_layer_offset. Assisted-by: opencode

khumarahn · 2026-09-23T11:49:48Z

With this PR, --fit on doesn't seem to work. Is this a known issue?

I don't really understand why, but it seems that the size of the mtp weights is not taken into account. I can work around it by increasing --fit-target, but it gets ugly with on a two-gpu machine: I have to leave extra space on all gpus because I don't know where the weights are going. And --spec-draft-device doesn't work: it segfaults if I didn't guess the right device.

Otherwise, I'm getting a nice performance boost.

Wladastic · 2026-09-23T11:52:15Z

With this PR, --fit on doesn't seem to work. Is this a known issue?

I don't really understand why, but it seems that the size of the mtp weights is not taken into account. I can work around it by increasing --fit-target, but it gets ugly with on a two-gpu machine: I have to leave extra space on all gpus because I don't know where the weights are going. And --spec-draft-device doesn't work: it segfaults if I didn't guess the right device.

Otherwise, I'm getting a nice performance boost.

I think that might be a different issue, for me almost all models need adjustments when --fit on is set. (In context of using mtp only, not in general)

ruixiang63

It would be great if we could minimize unnecessary changes and stay consistent with the current MTP design. #26725 might be a good reference.

ruixiang63 · 2026-09-23T10:07:33Z

        LOG_INF("%s: loading draft model '%s'\n", __func__, model_path.c_str());

-        llama_model * model_dft = llama_model_load_from_file(params.model.path.c_str(), mparams);
+        llama_model * model_dft = llama_model_load_from_file(model_path.c_str(), mparams);


This isn't a fix, both expressions are the same string. I'd suggest just dropping this hunk from the PR.

@ruixiang63 Are you sure about that? See 3fb9b98

Have you encountered any issues with MTP on the current master branch? If so, please open a separate issue.
This change does not seem relevant to the current Qwen3.8-Flash-Next MTP implementation.

ruixiang63 · 2026-09-23T12:40:18Z

+    parser.add_argument(
+        "--mtp-shared-embd", action="store_true",
+        help="With --mtp, leave the token embeddings, output norm and LM head out of the draft and take them from the target model at load time. Much smaller draft, but it needs a llama.cpp new enough to read it.",
+    )


I also don’t think we need to set this explicitly for MTP. I feel it should be less than 1 GB.

During MTP conversion, we can export the MTP head either together with the target model or separately. In the former case, the MTP head does not need its own token_embd or lm_head, since it can access them later through ctx_other. In the latter case, the exported one will be a self-contained draft model by design.

ruixiang63 · 2026-09-23T12:40:35Z

            if args.mtp:
                model_class.mtp_only = True

+        if args.mtp_shared_embd:


ruixiang63 · 2026-09-23T12:46:02Z

        if is_mtp and cls.no_mtp:
            return None
-        if cls.mtp_only and not is_mtp and name not in (
+        if cls.mtp_only and not is_mtp and (cls.mtp_shared_embd or name not in (


Based on the suggestion below, I don’t think we need mtp_shared_embd anymore.

ruixiang63 · 2026-09-23T12:51:29Z

        llama_set_embeddings_nextn(ctx_dft, true, /*masked*/ true);

-        is_mem_shared = llama_get_ctx_other(ctx_dft) == ctx_tgt;
+        char arch[64] = {0};


see https://github.com/ggml-org/llama.cpp/pull/28243/changes#r4082502758. If we follow how other models support MTP, this will not be needed. e.g. #26725

ruixiang63 · 2026-09-23T12:53:06Z

            if is_mtp and cls.no_mtp:
                return None
-            if cls.mtp_only and not is_mtp and name not in (
+            if cls.mtp_only and not is_mtp and (cls.mtp_shared_embd or name not in (


this will not be needed as well.

ruixiang63 · 2026-09-23T12:54:09Z

    }

-    if (model.arch == LLM_ARCH_EAGLE3 || model.arch == LLM_ARCH_DFLASH) {
+    if (model.arch == LLM_ARCH_EAGLE3 || model.arch == LLM_ARCH_DFLASH || model.arch == LLM_ARCH_QWEN4EXP) {


Here is not necessary as well. Let's follow the current MTP support design.

celendis · 2026-09-24T03:55:41Z

SYCL validation: qwen4exp MTP on 2× Arc Pro B70

Setup: Qwen3.8-Flash-Next (UD-IQ3_XXS), two B70s, --split-mode layer with ffn layers 0-11,36-47 on CPU/DDR5. Unsloth's standalone Q8_0 draft model.

--spec-type draft-mtp --spec-draft-n-min 2 --spec-draft-n-max 7 --spec-draft-p-min 0.75
--spec-draft-model mtp-Qwen3.8-Flash-Next-Q8_0.gguf --spec-draft-device SYCL1

My build carries one SYCL patch (PR 28796), and I did not use --fit so the --fit issue above was never exercised here.

Except for the --spec-* flags, this is the same binary, greedy + fixed seed, 8 categories × 3 measured passes using BetterBench 0.6.0. Decode tok/s:

category	MTP off	MTP on	delta
file_edit	29.0	41.5	+43.3%
json	29.0	41.2	+42.1%
math	29.2	38.9	+33.0%
summarization	29.3	36.8	+25.6%
code	28.8	36.1	+25.4%
chat	28.7	32.0	+11.7%
prose	28.4	29.5	+3.8%
reasoning	28.8	27.9	−3.2%
median	28.7	36.3	+25.5%

… on master)

Brings in the qwen4exp NextN/MTP draft head and cross-model tensor borrowing, so Unsloth's shared-* MTP heads can draft for Qwen3.8-Flash-Next via --spec-type draft-mtp.

…, final numbers

github-actions Bot added model Model specific ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend conversion labels Sep 2, 2026

ggerganov reviewed Sep 2, 2026

View reviewed changes

giveen mentioned this pull request Sep 3, 2026

qwen4exp: upstream catch-up, sparse-fa, and NextN/MTP support TheTom/llama-cpp-turboquant#340

Merged

evandhoffman mentioned this pull request Sep 3, 2026

Mainline llama.cpp is getting a Qwen3.8-Flash-Next MTP graph (#28243, draft) — the thing #77 is blocked on evanwtf/local-llm#110

Open

sdroege reviewed Sep 3, 2026

View reviewed changes

TheTom mentioned this pull request Sep 3, 2026

CUDA: full test-backend-ops sweep segfaults in ggml_backend_cuda_graph_compute at ROLL on GB10 (passes with graphs disabled) TheTom/llama-cpp-turboquant#347

Open

ryanmonsurate and others added 10 commits September 3, 2026 20:04

gguf-py : register the qwen4exp NextN tensors

c8c3a5b

Adds the MTP head's own hyper-connection mixer tensor names and lists the NextN tensors under the qwen4exp architecture.

qwen4exp: allow loading a draft-only MTP export

30b6537

qwen4exp mtp: trim comments

44c7960

qwen4exp mtp: cut comments that restate the code

2c96729

Assisted-by: Claude

danielhanchen force-pushed the qwen4exp/mtp branch from 2857e51 to 2c96729 Compare September 4, 2026 03:08

This comment was marked as spam.

Sign in to view

ChangXiang-SCU mentioned this pull request Sep 4, 2026

llama: GPU-resident LRU cache for host-offloaded MoE expert weights #27861

Draft

shikanime mentioned this pull request Sep 18, 2026

Eval bug: qwen4exp / deepseek-v4 abort at first decode on Vulkan (RADV, gfx1151) #29028

Closed

shikanime mentioned this pull request Sep 19, 2026

Track: Flash-Next MTP speculative decoding (blocked on upstream llama.cpp PR #28243) shikanime-labs/manifests#2429

Open

liquidspikes added a commit to liquidspikes/llama.cpp that referenced this pull request Sep 19, 2026

qwen4exp MTP: borrow shared embeddings and LM head via ctx_other (PR g…

9baf80c

…gml-org#28243)

podolchakagency mentioned this pull request Sep 20, 2026

Eval bug: Qwen3.8-Flash-Next MTP draft GGUFs fail to load on b11058 (tensor not found) — no MTP path on Pascal #29174

Open

danielhanchen and others added 2 commits September 21, 2026 11:52

Merge branch 'master' into pr-28243-rebase

6fd37b7

Address review comments

6fcaa16

Assisted-by: Claude

CISC reviewed Sep 22, 2026

View reviewed changes

ggerganov reviewed Sep 22, 2026

View reviewed changes

sourenaraya pushed a commit to sourenaraya/llama.cpp that referenced this pull request Sep 23, 2026

models: Qwen3.8-Flash-Next MTP

4296753

Squash of upstream PR ggml-org#28243 (net diff vs bb3c853), including the 09-21 review fixes: draft load path, nextn_layer_offset. Assisted-by: opencode

ruixiang63 reviewed Sep 23, 2026

View reviewed changes

SelfRef added a commit to SelfRef/llama.cpp-rdna3 that referenced this pull request Sep 24, 2026

merge: upstream PR ggml-org#28243 (new head: review comments, rebased…

5b74cd8

… on master)

rcfa mentioned this pull request Sep 25, 2026

vulkan : on gfx1150 (Strix Point) the q8_0/q6_K mat-vec chunking makes 3- and 5-column passes slower on Qwen3.8-Flash-Next, so draft-MTP n-max 2 gains nothing halo-box/strix-llama.cpp#88

Open

This was referenced Sep 25, 2026

MTP draft head for qwen4exp GGUFs on the llama.cpp engine ddalcu/mlx-serve#548

Open

fix(gguf): serve qwen4exp GGUFs ds4 cannot load on the llama engine ddalcu/mlx-serve#546

Merged

This was referenced Sep 25, 2026

Studio: reserve VRAM for shared MTP heads unslothai/unsloth#10149

Open

fit: measure a draft head that borrows the target's embeddings unslothai/llama.cpp#227

Open

bong-water-water-bong mentioned this pull request Sep 26, 2026

Vulkan: carry ggml-org#28243 (Qwen3.8-Flash-Next MTP) on our fork branch 1bit-MONSTER/engine#110

Merged

datanerdie mentioned this pull request Sep 26, 2026

Eval bug: Metal: per-file mmap mapping spans CPU-resident tensors, so a 27 GiB lazy-read table between GPU tensors counts against the GPU working set (Qwen3.8-Flash-Next OOM) #29465

Open

neurall added a commit to neurall/llama.cpp that referenced this pull request Sep 26, 2026

README: MTP added in release b11327 (PR ggml-org#28243, ggml-org#27917)…

79e9090

…, final numbers

		// target for a draft head that declares nextn_shared_target_tensors; must outlive this model
		const struct llama_model * model_shared;

		throw std::runtime_error(format("%s: this model is a draft head without its own '%s'; "
		"load it as a draft of its target model, not on its own", __func__, name.c_str()));

Conversation

danielhanchen commented Sep 2, 2026

Overview

Additional information

Requirements

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danielhanchen commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jesdga95 commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llbzow commented Sep 3, 2026

Validation: 2× RTX A6000 48GB (PCIe, no NVLink), Windows Server 2019, CUDA 12.6 (sm_86) — both heads draft at ~0.37 acceptance, no config beats baseline

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ovidiu-morar commented Sep 3, 2026

Metal data point on this exact head (2857e51): dn=2 is greedy-identical, dn=5 is slower and diverges

Uh oh!

jeverson-bmi commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as spam.

gabrielfreire commented Sep 4, 2026

Uh oh!

Volunteer-1 commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielhanchen commented Sep 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

khumarahn commented Sep 23, 2026

Uh oh!

Wladastic commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ruixiang63 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruixiang63 Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

celendis commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SYCL validation: qwen4exp MTP on 2× Arc Pro B70

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

danielhanchen commented Sep 2, 2026 •

edited

Loading

jesdga95 commented Sep 2, 2026 •

edited

Loading

Metal data point on this exact head (`2857e51`): dn=2 is greedy-identical, dn=5 is slower and diverges

jeverson-bmi commented Sep 3, 2026 •

edited

Loading

Volunteer-1 commented Sep 18, 2026 •

edited

Loading

Wladastic commented Sep 23, 2026 •

edited

Loading

ruixiang63 Sep 23, 2026 •

edited

Loading

celendis commented Sep 24, 2026 •

edited

Loading