Quantization

Quantization

On the NPU: 4-bit

The NPU multiplies int8 x int4 natively and has no sub-4-bit mode (see NPU). Its container format is Q4NX: a grid of 32 x 256-weight tiles, one chunk each. There are three chunk kinds, fully decoded (1bit-MONSTER npu-infer, the Qwen3.5-4B work; in the engine since PR #33, npu/q4nx.h):

Chunk	Kind	Weight	Seen in
5120 B	q4_1: bf16 scale and zero per 32 columns	code * scale + zero	Qwen3, Llama, Gemma, Phi
4736 B	Q4_K: u8 scale and min per 32 columns, bf16 S, M per row	S * scale * code + M * min	Qwen3.5-4B projections
8704 B	Q8: bf16 scale per 32 columns, int8 codes	d * code	Qwen3.5-4B lm_head, embedding

A tensor's chunk kind is its shape's last dimension. Q4_K repacks into q4_1 exactly apart from rounding two bf16 products (max error 0.00032 on a real tile, within the rounding bound), so the fast lane and dx run it unchanged; Q8 needs an 8-bit path.

• Ternary and 1-bit models convert to Q4NX exactly ({-1,0,+1} x s = nibbles 0/1/2 with min -s, step s). Bonsai-8B in Q4NX matches its CPU reference on the NPU (corr 0.9984).
• Per-model sweet spots are still being measured: pi goal mudeyyef-y97olm, paused. Candidates are Q4NX, q8, a q8 head, q8 attention, and a mix per tensor role taken from the model's Unsloth Dynamic file. Each is judged by KL <= 0.01 nats and perplexity within 3% of fp32.

On the GPU (llama.cpp: Vulkan, HRX): Unsloth Dynamic

An Unsloth Dynamic (UD) GGUF is an ordinary GGUF. "UD" names a recipe that picks a standard ggml type per tensor. For example, UD-Q4_K_XL keeps attn_v, ffn_down and the embedding at Q6_K, uses Q5_K/Q4_K elsewhere, and IQ4_XS on about 5 layers. Every UD file contains some IQ types, which llama.cpp decodes.

Qwen3-0.6B, KL against BF16 over 20 x 512 tokens (measured on the previous build, main b11258e), and llama-bench tg128 on the current build (llama.cpp f1a0aca, hrx-system 51b1739, 2026-09-23):

File	Size	PPL (BF16 10.47)	Mean KLD	Same top token	Vulkan0	HRX0
Q4_K_M	397 MB	10.82	0.063	89.1%	349 tok/s	314
UD-Q4_K_XL	405 MB	10.68	0.052	89.8%	324	293
UD-Q2_K_XL	302 MB	16.23	0.662	66.3%	388	17
UD-IQ2_M	269 MB	20.44	0.857	64.0%	396	13
UD-IQ1_S	215 MB	10633	7.41	6.0%	436	40

The sub-4-bit UD files used to fail on HRX0: ggml-hrx had no IQ3_XXS matmul. The engine's pin now has one (engine #29), and the HRX0 numbers above are from that build (llama.cpp 267d864, 2026-09-24). They now compute correctly: over wikitext-2 20 x 512, HRX0 and Vulkan0 give the same perplexity within 0.3% (UD-Q4_K_XL 24.43 / 24.45, UD-Q2_K_XL 39.54 / 39.45, UD-IQ2_M 59.14 / 59.06). They are slow on HRX0, though: parts of these files still run on the CPU there. Use Vulkan for sub-4-bit files.

• At 4 bits, UD-Q4_K_XL is better than Q4_K_M: its KL is 17% lower for 2% more bytes, at 7% lower Vulkan decode speed.
• The UD 1- and 2-bit recipes are for very large models. At 0.6B they lose too much.

Source: 1bit-MONSTER docs/wiki/unsloth-dynamic-ggufs.md (PR #2681).

Sweet spots across the top Unsloth models (2026-09-24)

The most-downloaded Unsloth Dynamic models, each UD quant against a reference (BF16 when it fits in 25 GiB, else Q8_0): KL divergence and same-top-token rate over wikitext-2 40 x 512, and llama-bench pp512 / tg128 in tok/s. Vulkan is the engine's upstream pin (llama.cpp 7fe450e); ROCm is ROCmFPX fb08d7c built MMQ-only (hipBLAS is wrong on gfx1151). Strix Halo, one run each.

Model	Quant	Size	Mean KLD	Same top	Vulkan pp / tg	ROCm pp / tg
Qwen3.5-4B (ref BF16)	UD-Q2_K_XL	1.8 GiB	0.226	79.0%	1993 / 81.3	1826 / 67.7
	UD-Q3_K_XL	2.3 GiB	0.057	88.7%	1895 / 67.1	1842 / 61.2
	UD-Q4_K_XL	2.7 GiB	0.018	92.7%	1992 / 60.0	1836 / 55.2
	UD-Q5_K_XL	3.0 GiB	0.0099	95.3%	1962 / 55.4	1799 / 51.1
Qwen3.5-9B (ref BF16)	UD-Q2_K_XL	3.8 GiB	0.159	83.1%	1023 / 50.8	966 / 44.8
	UD-Q3_K_XL	4.7 GiB	0.046	90.5%	1027 / 41.1	1134 / 38.3
	UD-Q4_K_XL	5.6 GiB	0.016	93.3%	1085 / 36.0	1114 / 33.9
	UD-Q5_K_XL	6.3 GiB	0.0092	95.1%	957 / 32.7	966 / 31.0
Qwen3-Coder-30B-A3B (ref Q8_0)	UD-Q2_K_XL	11.0 GiB	0.113	87.1%	1479 / 100.5	483 / 81.1
	UD-Q3_K_XL	12.9 GiB	0.058	90.4%	1158 / 92.4	1477 / 66.4
	UD-Q4_K_XL	16.5 GiB	0.027	93.4%	1297 / 84.7	1542 / 66.8
	UD-Q5_K_XL	20.2 GiB	0.012	95.6%	1180 / 76.4	1674 / 61.8
Qwen3.6-27B (ref Q8_0)	UD-Q2_K_XL	11.0 GiB	0.119	85.9%	348 / 16.9	269 / 15.6
	UD-Q3_K_XL	13.5 GiB	0.041	91.0%	346 / 14.2	340 / 13.4
	UD-Q4_K_XL	16.4 GiB	0.018	93.9%	358 / 12.0	339 / 11.4
	UD-Q5_K_XL	18.7 GiB	0.0081	95.1%	338 / 10.0	315 / 9.7
Qwen3.6-35B-A3B (ref Q8_0)	UD-Q2_K_XL	11.4 GiB	0.113	85.8%	1395 / 83.0	1200 / 58.2
	UD-Q3_K_XL	15.7 GiB	0.045	90.8%	1325 / 65.5	1325 / 51.1
	UD-Q4_K_XL	20.8 GiB	0.014	94.6%	1323 / 61.0	1313 / 47.8
	UD-Q5_K_XL	24.8 GiB	0.0091	95.8%	1127 / 53.7	1345 / 44.0

• UD-Q4_K_XL is the lean pick on every model: the smallest file with mean KLD at or under 0.03 (0.014-0.027).
• UD-Q5_K_XL is the sweet spot where accuracy matters: KLD under 0.01 on four of the five models. Qwen3-Coder-30B-A3B does not get there with any UD quant (0.012 at Q5), and its references is Q8_0, not BF16.
• UD-Q2_K_XL is not worth it at these sizes: KLD 0.11-0.23 and 79-87% same top token.
• Vulkan decodes faster than ROCm on every file (3-43%). Prefill is split: ROCm is faster on Qwen3.5-9B Q3-Q5, three of the four Coder files and the 35B-A3B at Q5.
• Not measured: gemma-4-12b-it (this llama.cpp build gives its BF16 reference a perplexity of about 480, so it does not run gemma-4 correctly). MiniMax-H3 was on the download list by mistake: it is a video-generation model (image-text-to-video on Hugging Face), not a text model.

The lean option: ROCmFP4 and ROCmI4 (2026-09-24)

ROCmFPX (MIT) is a llama.cpp fork with AMD-focused formats that upstream llama.cpp cannot read. The engine carries it as the lean option (1bit serve --lean, docs/lean.md, engine #37): ROCmFP4 on Vulkan, ROCmI4 on ROCm with the gfx1151 W4A4 path.

Qwen3.8-27B, every file quantized from the same BF16. KL against BF16 over wikitext-2 40 x 512 tokens (PPL(BF16) 5.92); llama-bench pp512 / tg128 tok/s; MTP chat is decode with the MTP head on three prompts (code / prose / short), Vulkan. ROCmFPX fb08d7c.

File	Size	PPL	Mean KLD	99% KLD	Same top token	Vulkan	ROCm	ROCm W4A4	MTP chat
UD-Q4_K_XL (default)	16.4 GiB	5.95	0.008	0.062	95.3%	353 / 11.9	307 / 10.8		22.6 / 20.5 / 19.4
ROCmFP4 STRIX_LEAN (lean)	13.8 GiB	6.08	0.055	0.482	88.7%	348 / 14.2	411 / 13.7		27.8 / 23.3 / 24.1
ROCmI4 (lean, ROCm)	13.9 GiB	6.12	0.051	0.466	89.7%	6 / 4.1	397 / 13.6	465 / 13.6

• ROCmFP4 decodes 19% faster on Vulkan (14-24% with MTP), from a 16% smaller file, at about 7x the KL divergence of UD-Q4_K_XL.
• ROCmI4 is ROCm-only (no Vulkan kernel). W4A4 lifts prompt processing 17% and leaves decode unchanged: ROCmFPX's MTP gains come from MTP verification, which is batched like prefill.
• Vulkan beats ROCm on standard GGUF: UD-Q4_K_XL is 353 / 11.9 on Vulkan against 307 / 10.8 on ROCm.
• The ROCmFPX files had no importance matrix; UD-Q4_K_XL was made with one.

Both lean routes pass serve_e2e on Strix Halo (ROCmFP4 on Vulkan0; ROCmI4 on ROCm0, ROCmI4 W4A4: enabled).

Reference: transformers runs GGUF quants natively (2026-09-22)

Hugging Face's transformers + llama.cpp quants post: AutoModelForCausalLM.from_pretrained(repo, gguf_file=...) now runs GGUF quants (Q4_K_M, Q5_K_M, Q6_K, ...) with ggml's own Metal kernels, packed rather than dequantized on load where a kernel exists. So far it is Apple Silicon only, starts with Qwen3.5 (dense and MoE), and needs transformers main plus a matching kernels.

For us this is an independent reference at the quantized level: on the rented M4 it can score a GGUF's logits (for example a ZINC or HRX run of the same file) against transformers using the same weights, not only against a BF16 original.