A CUDA expert tier for GGUF mixture-of-experts models.
Keep frequently used experts on your GPU while other expert weights stay in host memory.
Tip
New here? Start with the MoE Cache wiki guide. It covers the build, setup, and limits. From there, choose a hardware guide, follow the Windows WDDM instructions, or use the Docker Compose guide.
This maintained llama.cpp fork adds an opt-in expert cache for MoE models whose expert weights exceed VRAM. It keeps the familiar llama.cpp server and API. Without a cache flag, normal llama.cpp model placement remains available.
NVIDIA CUDA only today. The MoE cache requires an NVIDIA GPU. Have an AMD GPU or another system with separate RAM and VRAM? Join the OptLlama Discord to discuss support.
- Budgeted expert caching: choose VRAM per device or a slot count; cold expert weights stay host-backed.
- Automatic fast paths: compatible workloads can use grouped CUDA decode, cached prefill, and expert prefetch.
- Memory and drafting controls: bound host pinning and configure a separately loaded speculative draft model independently.
The feature guide lists eligibility, defaults, and fallback behavior.
Selected single-request decode results from the full benchmark comparison:
| Model | Stock llama.cpp | MoE cache fork | Peak VRAM difference |
|---|---|---|---|
| Qwen3.6 35B | 42.8 tok/s | 111.6 tok/s | -3.2% |
| Gemma 4 | 34.2 tok/s | 102.2 tok/s | +3.4% |
| Nemotron 3.5 Lightning | 57.0 tok/s | 114.9 tok/s | +0.1% |
| Ornith 1.5 | 35.2 tok/s | 97.4 tok/s | -2.3% |
Measured on an RTX 5070 Ti 16 GB with about 62 GiB RAM. Each pair was within 5% peak VRAM. These are selected historical results, not measurements of the current branch head or predictions for another machine. The wiki benchmark suite includes regressions, tested revisions, quantizations, exact commands, output notes, and other models.
This is the shortest source-build path. For a measured setup with model-specific placement, start in the wiki.
Build the moe-cache branch with an NVIDIA CUDA toolkit and CMake:
git clone --branch moe-cache https://github.com/GenerelSchwerz/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -jStart the server with a MoE GGUF you already have:
./build/bin/llama-server -m /path/to/model.gguf --moe-expert-cache-mib 4096Open the local address printed by the server for chat, or connect an OpenAI-compatible client. The 4096 MiB budget is only an example. Leave VRAM for the rest of the model, context, and runtime buffers.
The model, quantization, context, and placement determine the memory required. The cache changes where routed expert weights live; it does not change which experts the model selects.
| Flag | Purpose |
|---|---|
--moe-expert-cache-mib MiB |
Set the expert cache budget per CUDA device. |
--moe-expert-cache-size N |
Choose a slot count per cached tensor instead of a MiB budget. |
--moe-expert-cache-layers N[,N-M,...] |
Limit caching to selected MoE layers. |
--moe-expert-cache-host-pinned-mb N |
Bound host memory registration and staging. |
Use a MiB budget or a nonzero slot count. Both are off by default. Eligible decode workloads can use the fork's grouped CUDA path automatically; the feature guide explains when it applies.
Check the result: after a request, look for moe-cache hit and miss statistics in the server log. The command line alone does not prove the cache ran. Performance depends on the model, available memory, storage, and request pattern.
- Fork features and limits - defaults, host memory, grouped decode, and speculative drafts.
- Build with CUDA and use the server.
- Multi-GPU behavior - layer-split cache placement and validation. Tensor split with the cache enabled is unsupported.
- Benchmark suite - stock versus fork measurements and per-model reproduction records.
I'm a recent computer science graduate interested in AI development (especially local AI), low-level programming, and embedded systems. If your team is hiring in these areas, email me.
If this fork has been useful, you can support my work on Ko-fi.
- Built on llama.cpp and ggml.
- Code is under the repository's MIT license; model files have their own licenses.
- Ko-fi button artwork is from Ko-fi's brand kit.