model : re-enable -sm tensor for qwen4exp

[kh0pper]

Overview

Re-enables -sm tensor for qwen4exp. #27941 disabled it because test-llama-archs -a qwen4exp asserted on the Meta device once the fixture carried a PLE layer, and removed the test's earlier Meta skip for the arch.

The abort is a scheduler placement, not QSA. test-llama-archs builds the model with the embeddings host-resident, so the PLE embedding gather (ggml_get_rows on per_layer_token_embd) is a CPU node. In the qwen4exp graph hc_init (the ggml_repeat_4d that fans the embedding out to the hc streams) is first materialised inside layer 0's PLE path, after that gather. ggml_backend_sched_split_graph pass 2 expands a device assignment upwards only until it meets a CPU-assigned node, so the REPEAT is never reached from the first Meta node and "expand rest" leaves it on the CPU. The later ggml_reshape_3d(hc_init) inside the PLE query norm is then a view of a host-resident node inside the meta split, which ggml_backend_meta_graph_compute only tolerates for view_src->op == GGML_OP_NONE: GGML_ASSERT(ggml_backend_buffer_is_meta(tensor->buffer)) at ggml-backend-meta.cpp:476. deepseek4 builds the same hc init but its REPEAT is followed directly by a weight matmul, so the expansion reaches it.

Fix: ggml_build_forward_expand(gf, res_hc) right after hc_init is built. The REPEAT then directly precedes the first device node and pass 2 assigns it there; the embedding reshape stays in the CPU split and is copied in as a split input, the same shape deepseek4's graph has.

test-llama-archs on this branch (master e6ab7c1 + the change; seed 4097205655; RelWithDebInfo, RADV gfx1151, Mesa 25.2.8):

arch	Vulkan	Meta (-sm tensor)
qwen4exp	OK (1.17e-07)	OK (1.17e-07), was GGML_ASSERT
deepseek4	OK (8.44e-07)	OK (8.44e-07)
qwen35	OK (8.33e-08)	OK (8.33e-08)

Real model: Qwen3.8-Flash-Next UD-Q4_K_XL, -sm tensor over two RPC devices (#26610 tree with this change, one Vulkan ggml-rpc-server per Strix Halo box, ctx 65536) vs a single device, greedy: byte-identical on a short prompt and on a 1.8k-token prompt (1752 generated tokens), and identical to master's single-device output.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. AI-assisted analysis of the scheduler passes and test runs; the change and the tests were reviewed and run by me.

[Zerschranzer]

Heads-up: with --split-mode tensor, --fit on is ignored:

W common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort

In router mode the instance then OOMs on load and the only trace is:

E alloc_tensor_range: failed to allocate ROCm0 buffer of size 28313139456

Tested on an ROCm build, 2x RX 9060 XT.

Edit:
Without fit = on, and after manually testing to find the optimal values for my dual GPU setup, it is running with:

split-mode = tensor
tensor-split = 1,1
n-cpu-moe = 29

Additionally, tokens/s and PP speed remained nearly identical, within a fluctuation margin of ±10%. VRAM allocation is roughly 97% to 96%. However, tensor split mode seems to maintain higher stability over long context lengths.

[cb88]

RX 9060 XT

Similar experience to @Zerschranzer

2xMI50 32GB and I can load and run it with the following preset , doesn't quite fit in 2 cards with the MMPROJ , getting about 19t/s, 14.5t/s @ 75k context , 7.7t/s at 212k , about 6.5t/s at full context so its a bit better than -sm layer still which is I think under 5t/s at that length.
[Qwen 3.8 Flash Next TS]
model = /home/cb88/models/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
mmproj = /home/cb88/models/mmproj/mmproj-F16.gguf
ctx-size = 262144
parallel = 1
load-on-startup = off
ctk = q8_0
ctv = q8_0
load-mode = mmap
lzm = on
sm = tensor
tensor-split = 1,1
n-cpu-moe = 10

[kh0pper]

Rebased onto master. The 12 red jobs at the old base were a -Werror=maybe-uninitialized on norms in tests/test-backend-ops.cpp, a file this PR does not touch.

@Zerschranzer @cb88 — --fit on under -sm tensor is unrelated to this change: common/fit.cpp throws llama_params_fit is not implemented for SPLIT_MODE_TENSOR for every arch, and #28777 hits the same warning on CUDA with a different model. Worth its own issue; the part that actually bites is router mode turning that warning into a bare alloc_tensor_range OOM instead of refusing to start.

Thanks for the numbers. cb88's 6.5 t/s at full context, against an estimated under 5 with -sm layer, is the case this change exists for.

AI usage disclosure: YES, AI-assisted analysis; I ran the builds and tests.

[angliubc]

Independent real-hardware validation with 8× V100-SXM2-32GB (NVLink, NCCL backend) — the change works, and here is the multi-GPU performance data, which the Strix Halo RPC setup above does not cover.

Setup: Qwen3.8-Flash-Next UD-Q4_K_XL (111 GB), master lineage tree (e6ab7c1-equivalent arch rules, qwen4exp split rules in llama-model.cpp), -sm tensor -ngl 999, llama-bench, greedy correctness probes over the OpenAI endpoint.

Correctness (temp 0, greedy): 9.11 vs 9.9 → 9.9; count 1-30 exact; rectangle 12×5 → perimeter 34 / area 60; short-form Chinese prose coherent. All match single-device output.

Throughput (tg64, 3 repeats):

config	t/s
TP4	44.41 ± 6.26
TP4 tg512	47.93
TP8	30.37 ± 5.86
layer-split baseline (4 GPU)	35–37

TP4 now beats the layer-split baseline by ~25% on this hardware — the meta backend's NCCL path holds up well over NVLink (128 small allreduces/token at ~6 ms total).

One caveat for NVIDIA users hitting this after the merge: on the unsloth MTP fork lineage the per-device CUDA graph cache cap (max_cuda_graphs in ggml-cuda/common.cuh) is 64, and the meta backend dispatches ~128 subgraphs (one per AllReduce split point) to every device — that cap made the LRU evict every token, warmup never completed, and decode ran fully eager at ~6 t/s with zero cudaGraphLaunch. Raising it fixed TP4 6.8 → 44 t/s. Upstream master's cache has no cap, so master is unaffected; flagging it in case anyone tests TP on a derived tree and sees eager decode with no captures.

🤖 Generated with Claude Code