Home

1bit engine

A model-agnostic C++ inference engine for AMD Ryzen AI (Strix Halo). It runs inside Lemonade: Lemonade's onebit recipe runs 1bit serve, which serves each model behind an OpenAI-compatible API on the NPU, on the iGPU (HRX and Vulkan), through ZINC (including NVIDIA) or MLX. Repository docs live in docs/. This wiki records measured results, hardware facts and working practice.

Status (2026-09-26)

Step	What	State
1	The engine runs inside Lemonade: 1bit serve (#21), vendored Lemonade removed (#22); the onebit recipe passes Lemonade's LLM tests on Vulkan and HRX, being prepared for upstream	merged (#6, #21, #22)
2	HRX + Vulkan in one llama.cpp build	merged (#7); our patched pin of AMD's ggml-hrx, rebased on every bump (#29): IQ3_XXS, honest op claims, decode past 256 tokens fixed (#46), and MoE models above 128 experts run (Qwen3.6-35B-A3B: prefill 909, decode 35 tok/s, KLD 0.005 vs Vulkan; before, batches faulted and decode was wrong) (#84). --prefill-device hrx: HRX prefills, Vulkan decodes, one zero-copy KV cache (#30). Vulkan also has its own upstream pin (#28)
3	NPU fast lane on full ELFs, served through Lemonade's onebit recipe	merged (#8, #9): Qwen3-0.6B at 91 tok/s; XDNA driver + XRT pinned upstream and built privately (#12). Layer kernel still captured: our own dx layer kernel is in development (private), and the NPU reads HRX memory in place (dma-buf, zero copy). Q4NX files: all three chunk kinds read (#33)
3x	Qwen3.6-35B-A3B MoE on the NPU (closed source, private add-on)	served by 1bit serve --device npu in builds with -DONEBIT_NPU_PRIVATE (#85): parity vs fp64 on 3 positions (argmax 846 / 198 / 3710), 16.3-16.5 tok/s decode. See NPU#qwen36-35b-a3b-moe-on-the-npu-closed-source
4	Laya router	merged, opt-in (#18, #90, #91, #93): C++ scorer matches the Python reference (max logit diff 8.6e-6, same argmax on root and typed-decisions/); 1bit serve --device auto --laya-model <dir> routes each request. Load 2.7 s once, then 0.38 s per decision on Strix Halo (was 8.75 s; threaded encoder, byte-identical output, #93)
5	Every HF model, kept current (registry + daily census)	merged (#94): registry generated from the pinned llama.cpp, HRX fork and ZINC (265 HF architectures); first sweep 415,414 text-generation models, mapped 93.28%, checked 64.18% of those with an architecture. serve_e2e 19/20 across 7 architectures: all seven pass on HRX since the llama.cpp pin 96f6b89 (#104). GLM-4.7-Flash had been wrong on batched prompts because HRX's MUL_MAT_ID read route ids packed; Qwen3-Coder-30B-A3B now keeps flash attention at full context. See docs/registry.md
+	ZAYA1-8B (Zyphra) on Vulkan, HRX and ROCm, from our llama.cpp	merged (llama.cpp #2, #5, #6; engine #107): matches transformers FP32 at 95/96 teacher-forced positions; Q4_K_M decode 93 tok/s on Vulkan, 61.5 ROCm, 25.5 HRX. See GPU HRX and Vulkan#zaya1-8b-zyphra-on-all-three-gpu-devices-2026-09-25
+	ONNX models on the Radeon (Linux): ONNX Runtime's WebGPU provider (Dawn over Vulkan), ONNX_WEBGPU=1 scripts/build-onnx.sh	merged (#109): Qwen3-4B int4 decodes 52.9 tok/s on the GPU vs 26.8 on the CPU; serve_e2e --device onnx passes. See docs/onnx.md
+	Apple: Lemonade mlx recipe (lemon-mlx-engine)	merged (#10), verified on an M4
+	Hugging Face tokenizers v0.23.2 behind our C ABI: any tokenizer.json, byte-exact on 18 models (#19)
+	Linux kernel pinned to upstream v7.3-rc4 with amdxdna in-tree; packages built, not installed (#17)
+	ZINC pinned upstream; zinc Lemonade recipe; NVIDIA through ZINC's CUDA backend	merged (#14, with #15): Vulkan on Strix Halo 295 tok/s; RTX 5090 167-173 tok/s (Qwen3.5-9B, " Paris.")
+	Qwen3.8-Flash-Next MTP on Vulkan: the Vulkan pin carries upstream PR ggml-org#28243 (reviewed line by line) on our fork branch 1bit/vulkan-upstream, rebased onto every upstream release until upstream merges it; it also fixes -md loading the target model instead of the draft	merged (#110): Qwen3.8-27B --mtp unchanged (same draft acceptance); Flash-Next + MTP measured 40-49 tok/s on the same patch code, re-run on the merged build pending
+	MoE streaming from NVMe: expert traces (3 families × code/chat/30k long, plus Flash-Next), cache policies replayed, the expert cache (moe/: pinned slots, O_DIRECT reads in flight, gate-ahead prefetch; 1bit moe-cache), the three-way split measured both ways, the sweet spot per model	merged (#113): shared-budget LRU best; gate-ahead top-k covers 74-85% of misses; at 75% of experts in RAM 32-48 tok/s (Coder-30B), 32-39 (Qwen3.6-35B), 33-40 (GLM-4.7-Flash); every split loses to all-Vulkan0 (experts: Vulkan 135-171 GB/s, HRX 42-73, CPU 63-83, NPU 39); MTP leaves the drive's reads per token unchanged; demand reads are latency-bound, so reads must be queued across layers. Size-class slots (#137), packed slot arrays (#141); streamed in decode by 1bit serve --moe-slots N on Vulkan (#150, fork #19): perplexity identical to all-Vulkan, Coder-30B 28-38 tok/s at 75% of experts, 8-10 at 25% (resident 79-91). Prefetch off the decode path (#154, fork #20): 43-45 tok/s at 75% (serve 39); the per-layer host round trip caps streaming near 57, and at 25% the drive is the limit. ARC, LRU-2 and W-TinyLFU don't beat the shared LRU (#156). --moe-subst 0.5 (#161, fork #21): a resident expert stands in for a missing one: reads -20-27%, KLD 0.008-0.012, decode +10-35%. See docs/moe-streaming.md
+	DwarfStar (antirez/ds4) as a backend: 1bit serve --device ds4 (source pinned in third_party/ds4, built against TheRock by scripts/build-ds4.sh; --ssd-streaming passes through)	merged (#111, #112): DeepSeek V4 Flash Q2 answers through the engine at 14.8 tok/s decode. See docs/dwarfstar.md
+	Zyphra family on Vulkan: Zamba2 (carried ggml-org#21412), Zamba v1 and BlackMamba (our model code), a Vulkan Mamba-2 scan for d_state 64 and a Vulkan Mamba-1 scan (every Mamba-1 model scanned on the CPU before), and a converter fix for SentencePiece-style tokenizer.json vocabs	merged (#114, #116, #117, #118): Zamba v1 and BlackMamba match the reference at 96/96 teacher-forced positions; decode Zamba2 1.2B 89.8 / 2.7B 45.2 / 7B 17.1 tok/s, Zamba v1 7B 14.9, BlackMamba-1.5B Q8_0 369, Mamba-370M 25.8 -> 173.5. ZAYA1-74B (sliding-window layers) in progress. See docs/vulkan.md
+	Zyphra's whole LLM family: ZAYA1-74B-preview (sliding-window layers), ZAYA1-base and ZAYA1-reasoning-base (Zyphra's legacy checkpoint layout, converted by our converter), and the vision models ZAYA1-VL-8B (vision-only LoRA on image tokens, bidirectional image attention) and Zamba2-VL 1.2B/2.7B/7B through 1bit serve --mmproj	merged (#129, #131, #139, #149; llama.cpp #11, #14, #15, #16, #17, #18): ZAYA1-74B Q4_K_M 35.4 tok/s decode; ZAYA1-8B-legacy converts to tensors byte-identical to ZAYA1-8B; ZAYA1-VL-8B matches Zyphra's own FP32 code at 100/101 teacher-forced positions; Zamba2-VL vision embeddings match transformers at mean cosine 0.9999 (CPU); a ZAYA bug with several sequences per ubatch (concurrent requests) found and fixed (#15). See docs/vulkan.md
+	OPT, GPT-Neo, CodeGen and GPT-J from our llama.cpp (upstream has no model code for them), routed by 1bit serve --device vulkan like ZAYA	merged (#138; llama.cpp #8): against transformers FP32, teacher-forced, opt-125m 96/96, gpt-neo-125m 96/96, codegen-350M-mono 96/96 (was 0/96: its fused projection is query, value, key), tiny GPT-J 89/89; the registry maps 323 HF architectures (vulkan 323, hrx 300, zinc 64). See docs/vulkan.md
+	HRX decode made deterministic: 1bit serve --device hrx decodes without the split flash-attention kernel by default (ONEBIT_HRX_DECODE_SPLIT=1 turns it back on); the kernel's multi-pass rework is pinned but stays off	merged (#148, #151): with the kernel, identical requests differed by up to 3.66 nats and Qwen3-Coder-30B faulted the GPU in 2 of 3 runs (#140, #123); without it Qwen3-0.6B 322 tok/s (was 169±34), ZAYA1-8B 25.5 (was 23.5), Coder-30B 5/5 clean at 66-71. See docs/hrx.md
+	1bit serve on the GPU: --mtp (2.4-2.9x on Qwen3.8-27B, 3.4x on code), --parallel (continuous batching, 5.8x the best single stream), --adaptive (Vulkan first, ROCm overflow), RAG (--embed, --rerank), --device rocm	merged (#41, #42)
+	Lean option: ROCmFP4 (Vulkan) and ROCmI4 (ROCm, W4A4) behind --lean	merged (#37); see Quantization
+	Unsloth Dynamic sweet spots for the top models: UD-Q4_K_XL lean, UD-Q5_K_XL accurate	measured 2026-09-24, Quantization
+	Documentation site at 1bit.gg with the blog	merged (#31, #36, #44)

Pages

• NPU: what XDNA 2 multiplies natively (int8 x int4, no ternary), the fast lane, the 35B MoE on the NPU, and how to read an ERT timeout.
• Quantization: why 4-bit on the NPU, ternary to 4-bit, Unsloth Dynamic GGUFs on the GPU, and the lean option (ROCmFP4, ROCmI4).
• GPU HRX and Vulkan: the two iGPU devices, measured speeds, and building.
• NVIDIA and ZINC: ZINC's CUDA backend on a rented RTX 5090, its model coverage per backend, and how to rent and reach the box.
• Apple Metal and MLX: the Metal backend and the MLX engine on an Apple M4.
• Working on the shared NPU: rules for agents sharing the one NPU.