CUDA: mixed K/V cache types silently run flash attention on CPU (2x slower generation) unless built with GGML_CUDA_FA_ALL_QUANTS

[GaspardSapin]

Summary

With a default CUDA build, ggml_cuda_get_best_fattn_kernel() returns BEST_FATTN_KERNEL_NONE when K->type != V->type, and ggml_cuda_fattn_kv_type_supported() rejects q4_1/q5_0/q5_1 (ggml/src/ggml-cuda/fattn.cu, #ifndef GGML_CUDA_FA_ALL_QUANTS blocks). A quantized V cache requires flash attention, so FLASH_ATTN_EXT falls back to the CPU on every attention layer. Nothing in the logs says so, and the cost grows with context length.

Building with -DGGML_CUDA_FA_ALL_QUANTS=ON fixes it.

Setup

• Windows 11, RTX 5070 Ti (sm_120), Ryzen 7 9800X3D, CUDA 13.4, VS 2022
• Branch prism @ 01ae597
• Model: Ternary-Bonsai-2-27B-PQ2_0.gguf (qwen35 27B)
• llama-bench -m 'Ternary-Bonsai-2-27B-PQ2_0.gguf' -ngl 99 -fa 1 -p 0 -n 3000 -r 2 -ctk <K> -ctv <V>

Results: tg3000 (t/s)

K / V	default build	FA_ALL_QUANTS=ON
f16 / f16	86.76	85.93
bf16 / bf16	86.00	85.94
q8_0 / q8_0	85.11	85.06
q4_0 / q4_0	84.86	84.85
f16 / q8_0	40.43	85.60
q8_0 / q4_0	38.75	84.85

Other combinations that only work with the option, all FA_ALL_QUANTS=ON: f16/q4_0 85.46, q8_0/q5_1 84.87, q8_0/q5_0 84.30, q5_1/q5_1 84.49, q5_0/q5_0 83.70, q4_1/q4_1 85.09.

With the default build, all 45 combinations outside {f16, bf16, q8_0, q4_0} x same type fall back to CPU (67-72 t/s on tg128 vs ~81 on GPU). With a pre-filled context it gets much worse: q8_0/q4_0 tg128 @ d8192 is 11.44 t/s (default) vs 74.79 t/s (FA_ALL_QUANTS=ON).

Cost of the option: ggml-cuda.dll goes from 35 MB to 73 MB and the build takes noticeably longer. No speed regression on the combinations that were already supported.

Suggestions

1. Log a warning when CUDA rejects FLASH_ATTN_EXT because of the K/V types, pointing to GGML_CUDA_FA_ALL_QUANTS.
2. Consider enabling GGML_CUDA_FA_ALL_QUANTS in the fork's release binaries: a mixed KV cache (e.g. q8_0/q4_0) is a natural way to fit long contexts on 16 GB cards with these models.
3. Mention it in the README / build docs.