` | Slot count too big for that GPU — reduce slots, or move layers off it with `-ts`. |
| `no CPU-resident MoE layers` | Experts are already on GPU (no `--n-cpu-moe`) — nothing to cache. |
| No init line, no warning | Architecture not wired for the cache — model runs unchanged. |
| Model loads, then context creation OOMs | Pack fits but KV/compute don't — drop a few slots or shrink/compress KV. |
+| Crash on the first long prompt | Not enough free VRAM for the prompt's workspace — drop slots until ~1 GB stays free. |
+
+## Prefill speedups for offloaded experts
+
+Two environment variables speed up prompt processing when experts live in RAM, independent of the
+cache. Both are off by default and token-identical.
+
+| Env var | What it does |
+| --- | --- |
+| `GGML_CUDA_REGISTER_HOST=1` | Page-locks (pins) the mmap'd CPU expert weights so host→device copies go straight over DMA instead of through the driver's hidden bounce buffer (~6–7 → ~20 GB/s). |
+| `GGML_SCHED_PREFETCH_EXPERTS=1` | Prefetches each layer's experts on a second CUDA stream, so the weight uploads overlap compute instead of stalling the GPU. |
+
+On an RTX 3060 12 GB with Qwen3.6-35B-A3B (`--n-cpu-moe 26`), prefill at 2048 went from
+**~1143 → ~1880 tok/s (+64%)**:
+
+```bash
+GGML_CUDA_REGISTER_HOST=1 GGML_SCHED_PREFETCH_EXPERTS=1 \
+./build/bin/llama-bench -m model.gguf -ngl 99 -ncmoe 26 -p 2048 -n 0 -r 5 -b 2048 -ub 2048
+```
+
+---
+
+# Upstream llama.cpp
+
+Everything below is the upstream README.
+
+
+
+
+
+LLM inference in C/C++
+
+[](https://opensource.org/licenses/MIT)
+[](https://github.com/ggml-org/llama.cpp/releases?q=tag:v0)
+[](https://github.com/ggml-org/llama.cpp/releases?q=b)
+[](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml)
+[](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
+[](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
+
+[ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3AServeurpersoCom%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3Aallozaur%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Alhez%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming)%20sort%3Aupdated-desc) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
+
+
+
## Recent API changes
diff --git a/src/llama-model.cpp b/src/llama-model.cpp
index 90afd38a7eaf..3e09e0b2406a 100644
--- a/src/llama-model.cpp
+++ b/src/llama-model.cpp
@@ -1887,117 +1887,126 @@ void llama_model_base::init_moe_expert_cache() {
return;
}
- ggml_backend_dev_t dev = nullptr;
+ ggml_backend_dev_t dev_first = nullptr;
for (const auto & d : devices) {
- if (!d.is_meta) { dev = d.dev; break; }
+ if (!d.is_meta) { dev_first = d.dev; break; }
}
- if (dev == nullptr || ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_GPU) {
+ if (dev_first == nullptr || ggml_backend_dev_type(dev_first) != GGML_BACKEND_DEVICE_TYPE_GPU) {
LLAMA_LOG_WARN("%s: no GPU device - expert cache disabled\n", __func__);
return;
}
- ggml_backend_buffer_type_t buft = ggml_backend_dev_buffer_type(dev);
- // candidate layers: routed experts resident in host memory
- std::vector pack_layers;
+ // candidate layers: routed experts resident in host memory, grouped by the GPU that runs the layer so the
+ // hot experts sit on the same device as the rest of the layer (layers left on the CPU use the first GPU)
+ std::map> pack_layers_by_dev;
for (int il = 0; il < (int) layers.size(); il++) {
const auto & l = layers[il];
if (l.ffn_gate_exps && l.ffn_up_exps && l.ffn_down_exps && freq.count(il) &&
l.ffn_gate_exps->buffer && ggml_backend_buft_is_host(ggml_backend_buffer_get_type(l.ffn_gate_exps->buffer))) {
- pack_layers.push_back(il);
+ ggml_backend_dev_t dev = dev_layer(il);
+ if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_GPU) {
+ dev = dev_first;
+ }
+ pack_layers_by_dev[dev].push_back(il);
}
}
- if (pack_layers.empty()) {
+ if (pack_layers_by_dev.empty()) {
LLAMA_LOG_INFO("%s: no CPU-resident MoE layers - expert cache not built\n", __func__);
return;
}
- ggml_init_params ctx_params = {
- /*.mem_size =*/ (5*pack_layers.size() + 1)*ggml_tensor_overhead(),
- /*.mem_buffer =*/ nullptr,
- /*.no_alloc =*/ true,
- };
- ggml_context * ctx = ggml_init(ctx_params);
-
- for (int il : pack_layers) {
- auto & l = layers[il];
- const ggml_tensor * g = l.ffn_gate_exps;
- const ggml_tensor * u = l.ffn_up_exps;
- const ggml_tensor * d = l.ffn_down_exps;
- const int64_t n_expert = g->ne[2];
- const int64_t S = std::min(n_slots, n_expert);
- l.ffn_gate_exps_hot = ggml_new_tensor_3d(ctx, g->type, g->ne[0], g->ne[1], S);
- l.ffn_up_exps_hot = ggml_new_tensor_3d(ctx, u->type, u->ne[0], u->ne[1], S);
- l.ffn_down_exps_hot = ggml_new_tensor_3d(ctx, d->type, d->ne[0], d->ne[1], S);
- l.moe_map_hot = ggml_new_tensor_2d(ctx, GGML_TYPE_I32, 1, n_expert);
- l.moe_map_cold = ggml_new_tensor_2d(ctx, GGML_TYPE_I32, 1, n_expert);
- ggml_format_name(l.ffn_gate_exps_hot, "blk.%d.ffn_gate_exps_hot", il);
- ggml_format_name(l.ffn_up_exps_hot, "blk.%d.ffn_up_exps_hot", il);
- ggml_format_name(l.ffn_down_exps_hot, "blk.%d.ffn_down_exps_hot", il);
- ggml_format_name(l.moe_map_hot, "blk.%d.moe_map_hot", il);
- ggml_format_name(l.moe_map_cold, "blk.%d.moe_map_cold", il);
- }
-
- ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, buft);
- if (buf == nullptr) {
- LLAMA_LOG_WARN("%s: pack allocation failed - expert cache disabled\n", __func__);
- ggml_free(ctx);
- for (int il : pack_layers) {
- auto & l = layers[il];
- l.ffn_gate_exps_hot = l.ffn_up_exps_hot = l.ffn_down_exps_hot = nullptr;
- l.moe_map_hot = l.moe_map_cold = nullptr;
- }
- return;
- }
- // weights usage pins the pack tensors to their backend during graph
- // assignment - without it a CPU-assigned consumer can drag the hot
- // matmuls (and a per-layer weight copy) onto the CPU
- ggml_backend_buffer_set_usage(buf, GGML_BACKEND_BUFFER_USAGE_WEIGHTS);
-
- // fill packs (expert dim is outermost: one contiguous slab per expert)
std::vector slab;
std::vector map_hot, map_cold;
- size_t total_bytes = 0;
- for (int il : pack_layers) {
- auto & l = layers[il];
- const int64_t n_expert = l.ffn_gate_exps->ne[2];
- const int64_t S = l.ffn_gate_exps_hot->ne[2];
+ for (const auto & [dev, pack_layers] : pack_layers_by_dev) {
+ ggml_backend_buffer_type_t buft = ggml_backend_dev_buffer_type(dev);
- std::vector> ranked; // (-count, expert)
- for (const auto & [e, c] : freq[il]) {
- if (e < n_expert) {
- ranked.push_back({-c, e});
+ ggml_init_params ctx_params = {
+ /*.mem_size =*/ (5*pack_layers.size() + 1)*ggml_tensor_overhead(),
+ /*.mem_buffer =*/ nullptr,
+ /*.no_alloc =*/ true,
+ };
+ ggml_context * ctx = ggml_init(ctx_params);
+
+ for (int il : pack_layers) {
+ auto & l = layers[il];
+ const ggml_tensor * g = l.ffn_gate_exps;
+ const ggml_tensor * u = l.ffn_up_exps;
+ const ggml_tensor * d = l.ffn_down_exps;
+ const int64_t n_expert = g->ne[2];
+ const int64_t S = std::min(n_slots, n_expert);
+ l.ffn_gate_exps_hot = ggml_new_tensor_3d(ctx, g->type, g->ne[0], g->ne[1], S);
+ l.ffn_up_exps_hot = ggml_new_tensor_3d(ctx, u->type, u->ne[0], u->ne[1], S);
+ l.ffn_down_exps_hot = ggml_new_tensor_3d(ctx, d->type, d->ne[0], d->ne[1], S);
+ l.moe_map_hot = ggml_new_tensor_2d(ctx, GGML_TYPE_I32, 1, n_expert);
+ l.moe_map_cold = ggml_new_tensor_2d(ctx, GGML_TYPE_I32, 1, n_expert);
+ ggml_format_name(l.ffn_gate_exps_hot, "blk.%d.ffn_gate_exps_hot", il);
+ ggml_format_name(l.ffn_up_exps_hot, "blk.%d.ffn_up_exps_hot", il);
+ ggml_format_name(l.ffn_down_exps_hot, "blk.%d.ffn_down_exps_hot", il);
+ ggml_format_name(l.moe_map_hot, "blk.%d.moe_map_hot", il);
+ ggml_format_name(l.moe_map_cold, "blk.%d.moe_map_cold", il);
+ }
+
+ ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, buft);
+ if (buf == nullptr) {
+ LLAMA_LOG_WARN("%s: pack allocation failed on %s - expert cache disabled for its %zu layers\n",
+ __func__, ggml_backend_buft_name(buft), pack_layers.size());
+ ggml_free(ctx);
+ for (int il : pack_layers) {
+ auto & l = layers[il];
+ l.ffn_gate_exps_hot = l.ffn_up_exps_hot = l.ffn_down_exps_hot = nullptr;
+ l.moe_map_hot = l.moe_map_cold = nullptr;
}
+ continue;
}
- std::sort(ranked.begin(), ranked.end());
+ // weights usage pins the pack tensors to their backend during graph
+ // assignment - without it a CPU-assigned consumer can drag the hot
+ // matmuls (and a per-layer weight copy) onto the CPU
+ ggml_backend_buffer_set_usage(buf, GGML_BACKEND_BUFFER_USAGE_WEIGHTS);
- map_hot.assign(n_expert, -1);
- map_cold.resize(n_expert);
- for (int64_t e = 0; e < n_expert; e++) {
- map_cold[e] = (int32_t) e;
- }
- for (int64_t s = 0; s < S && s < (int64_t) ranked.size(); s++) {
- const int32_t e = ranked[s].second;
- map_hot[e] = (int32_t) s;
- map_cold[e] = -1;
- const ggml_tensor * srcs[3] = { l.ffn_gate_exps, l.ffn_up_exps, l.ffn_down_exps };
- ggml_tensor * dsts[3] = { l.ffn_gate_exps_hot, l.ffn_up_exps_hot, l.ffn_down_exps_hot };
- for (int t = 0; t < 3; t++) {
- const size_t nb = srcs[t]->nb[2];
- slab.resize(nb);
- ggml_backend_tensor_get(srcs[t], slab.data(), e*nb, nb);
- ggml_backend_tensor_set(dsts[t], slab.data(), s*nb, nb);
- total_bytes += nb;
+ // fill packs (expert dim is outermost: one contiguous slab per expert)
+ size_t total_bytes = 0;
+ for (int il : pack_layers) {
+ auto & l = layers[il];
+ const int64_t n_expert = l.ffn_gate_exps->ne[2];
+ const int64_t S = l.ffn_gate_exps_hot->ne[2];
+
+ std::vector> ranked; // (-count, expert)
+ for (const auto & [e, c] : freq[il]) {
+ if (e < n_expert) {
+ ranked.push_back({-c, e});
+ }
+ }
+ std::sort(ranked.begin(), ranked.end());
+
+ map_hot.assign(n_expert, -1);
+ map_cold.resize(n_expert);
+ for (int64_t e = 0; e < n_expert; e++) {
+ map_cold[e] = (int32_t) e;
+ }
+ for (int64_t s = 0; s < S && s < (int64_t) ranked.size(); s++) {
+ const int32_t e = ranked[s].second;
+ map_hot[e] = (int32_t) s;
+ map_cold[e] = -1;
+ const ggml_tensor * srcs[3] = { l.ffn_gate_exps, l.ffn_up_exps, l.ffn_down_exps };
+ ggml_tensor * dsts[3] = { l.ffn_gate_exps_hot, l.ffn_up_exps_hot, l.ffn_down_exps_hot };
+ for (int t = 0; t < 3; t++) {
+ const size_t nb = srcs[t]->nb[2];
+ slab.resize(nb);
+ ggml_backend_tensor_get(srcs[t], slab.data(), e*nb, nb);
+ ggml_backend_tensor_set(dsts[t], slab.data(), s*nb, nb);
+ total_bytes += nb;
+ }
}
+ ggml_backend_tensor_set(l.moe_map_hot, map_hot.data(), 0, n_expert*sizeof(int32_t));
+ ggml_backend_tensor_set(l.moe_map_cold, map_cold.data(), 0, n_expert*sizeof(int32_t));
}
- ggml_backend_tensor_set(l.moe_map_hot, map_hot.data(), 0, n_expert*sizeof(int32_t));
- ggml_backend_tensor_set(l.moe_map_cold, map_cold.data(), 0, n_expert*sizeof(int32_t));
- }
- pimpl->ctxs_bufs.emplace_back(ggml_context_ptr{ctx}, std::vector{});
- pimpl->ctxs_bufs.back().second.emplace_back(buf);
+ pimpl->ctxs_bufs.emplace_back(ggml_context_ptr{ctx}, std::vector{});
+ pimpl->ctxs_bufs.back().second.emplace_back(buf);
- LLAMA_LOG_INFO("%s: expert cache: %zu layers x %d slots, %.2f MiB uploaded to %s\n",
- __func__, pack_layers.size(), n_slots, total_bytes/1024.0/1024.0, ggml_backend_buft_name(buft));
+ LLAMA_LOG_INFO("%s: expert cache: %zu layers x %d slots, %.2f MiB uploaded to %s\n",
+ __func__, pack_layers.size(), n_slots, total_bytes/1024.0/1024.0, ggml_backend_buft_name(buft));
+ }
}
ggml_tensor * llama_model_base::create_tensor(llama_model_loader & ml, const LLM_TN_IMPL & tn, const std::initializer_list & ne, int flags) {
diff --git a/tools/moe-trace/moe-trace.cpp b/tools/moe-trace/moe-trace.cpp
index 032d49d977f9..29636ff9a9bb 100644
--- a/tools/moe-trace/moe-trace.cpp
+++ b/tools/moe-trace/moe-trace.cpp
@@ -9,12 +9,13 @@
// separate prefill routing from decode routing.
//
// Usage:
-// MOE_TRACE_OUT=trace.csv llama-moe-trace -m model.gguf -ngl 99 -ncmoe 26 -fa on \
+// MOE_TRACE_OUT=trace.csv llama-moe-trace -m model.gguf -ngl 99 -ncmoe 26 -fa on --temp 0 \
// -p "prompt text" -n 512
#include "arg.h"
#include "common.h"
#include "log.h"
+#include "sampling.h"
#include "llama.h"
#include
@@ -123,12 +124,14 @@ int main(int argc, char ** argv) {
tc.pos += n_eval;
}
- // greedy decode
+ // decode with the sampling flags (--temp, --top-k, --top-p, --min-p, --seed, ...) so the trace
+ // follows the same expert routing a server with those defaults would see
tc.in_prompt = false;
- llama_sampler * smpl = llama_sampler_init_greedy();
+ common_sampler * smpl = common_sampler_init(model, params.sampling);
llama_token tok = 0;
for (int i = 0; i < params.n_predict; i++) {
- tok = llama_sampler_sample(smpl, lctx, -1);
+ tok = common_sampler_sample(smpl, lctx, -1);
+ common_sampler_accept(smpl, tok, true);
if (llama_vocab_is_eog(vocab, tok)) {
break;
}
@@ -141,7 +144,7 @@ int main(int argc, char ** argv) {
LOG_INF("decoded %d/%d\n", i, params.n_predict);
}
}
- llama_sampler_free(smpl);
+ common_sampler_free(smpl);
fclose(tc.out);
LOG_INF("trace written to %s\n", out_path);