MoE streaming: Qwen3.8-Flash-Next streamed and measured; serve maps per-layer embeddings

[bong-water-water-bong]

• Qwen3.8-Flash-Next UD-Q4_K_XL (111 GB: 71.7 GiB of experts, 26.8 GiB of per-layer embeddings) decodes streamed from 4.6 GiB of GPU memory plus the slots. Exact routing (perplexity 11.32) gives 3.5-3.7 tok/s at 4608 slots (19%) and 5.4-6.2 at 9216 (38%).
• Resident-aware routing costs more here than on Coder-30B. r 0.5 gives KLD 0.07; r 0.9 gives 0.015-0.028 for +5-15%.
• serve --moe-slots also maps per_layer_token_embd on the CPU. That's Flash-Next's table, and the tensor is row-gathered only. Checked with Coder-30B: the chat answers, and serve plus llama-server hold 128 MiB of anonymous memory.
• docs/moe-streaming.md gets the new section "Flash-Next, streamed", and Next item 2 now asks for a better stand-in. serve.md gets a note.

🤖 Generated with Claude Code

[context7]

Docs7 for 1bit-monster/engine

Result	Status	Action
Deployment	➖ Not used	—
Content review	➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan.	—

Commit 575ec1b

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing Error Handling for File Paths The code inserts environment variables and command-line arguments related to MOE file handling without validating if the file paths are accessible or valid. If o.model does not resolve to a valid file or if the file cannot be accessed due to permissions, this could lead to runtime errors or undefined behavior when the subprocess attempts to load the model. This is particularly critical in a production environment where file access may be restricted or paths may be misconfigured. env.push_back("ONEBIT_MOE_FILE=" + std::filesystem::absolute(o.model).string()); env.push_back("ONEBIT_MOE_SLOTS=" + std::to_string(o.moe_slots)); if (!o.moe_subst.empty()) env.push_back("ONEBIT_MOE_SUBST=" + o.moe_subst); // the experts stay file-backed: without these, llama.cpp copies every one into RAM // (a pinned Vulkan host buffer, or a CPU repack), which is what streaming avoids // (per_layer_token_embd: Qwen3.8-Flash-Next's 26.8 GiB table, only ever row-gathered) argv.insert(argv.end(), {"-ot", "exps=CPU,per_layer_token_embd=CPU", "--no-host", "--no-repack", "--load-mode", "mmap"}); Unverified Performance Claims The documentation claims that Qwen3.8-Flash-Next decodes at 3.5-6 tok/s exactly from 18-32 GiB of GPU memory, but these numbers are not verified with measurements on the specified hardware (AMD Strix Halo's XDNA 2 NPU). Without actual hardware measurements, these performance claims are unverified and could mislead users about the real-world performance of the system. - **A model larger than the box's free memory decodes at 3.5-6 tok/s exactly**, from 18-32 GiB of GPU memory in all, instead of the 77 GiB it needs resident (40-49 tok/s with MTP).