Vulkan PTQ1_0: prefill ceiling ~16 tok/s on fp16-less NVIDIA (GTX 1070/Pascal, driver 535.98) — 6x behind RDNA1 on the same build

[sundog315]

Summary

On a GTX 1070 (Pascal, sm_61) running a self-built prism-branch Vulkan binary, the integer-dot decode kernel landed on the branch fixes PTQ1_0 decode dramatically (0.81 -> 15.3 tok/s, i.e. the b10709 release -> prism HEAD, -fa on). But prompt processing is stuck at ~16 tok/s steady-state regardless of every config lever tried. The same codebase on an AMD BC-250 (RDNA1, no coopmat, no int dot) reports pp512 = 95 tok/s, so the gap on this NVIDIA device (~6x) looks to be fp16-compute + proprietary-driver-related rather than config.

Hardware / software

• GPU: NVIDIA GeForce GTX 1070 8 GB (Pascal, sm_61), driver 535.98 (Win10 22H2)
• Device line (self-built prism HEAD):
  text

  NVIDIA GeForce GTX 1070 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none

• Build: prism branch HEAD as of 2026-09-23 (contains mul_mat_vecq_ptq1_0.comp / dequant_pq2_0.comp), MinGW ucrt64 gcc 16.2, Ninja, -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release, glslc 2026.3
• Model: Ternary-Bonsai-2-27B-PTQ1_0.gguf (5.53 GiB, qwen35 arch)
• Host: Xeon E5-2680 v4 (14C/28T), 128 GB RAM

Results

llama-bench -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -p 512 -n 4 -r 2:

build / config | pp512 | tg -- | -- | -- release b10709, -fa 1 | 14.13 @pp128 | 0.81 @tg32 prism HEAD, -fa on, FP16 KV | 16.2 | 14.7 prism HEAD, -fa on, q4_0 KV | 16.22 ± 0.06 | 13.85 ± 0.39 prism HEAD, -fa off, q4_0 KV | 16.40 ± 0.03 | 14.47 ± 0.11 prism HEAD, -fa on, q4_0 KV, GGML_VK_FORCE_MMVQ=1 | 16.27 ± 0.41 | 14.42 prism HEAD, -fa on, q4_0 KV, -ub 128 | 16.28 | 14.30 prism HEAD, -fa on, q4_0 KV, -p 64 | 27.3 | 14.71 prism HEAD, -fa on, q4_0 KV, -p 2048 | 16.05 | –

Server-side (llama-server, -fa on -ctk/-ctv q4_0 --kv-mean-center -c 16384): a 2,839-token prompt prefilled at 15.25 tok/s; short 18-token prompts ~4.8 tok/s (≈3 s per-request fixed overhead: graph build + template).

Observations

• The pp64 = 27 t/s figure is the attention-light, small-batch case; steady-state (pp512 and pp2048 both) is a flat ~16 tok/s.
• Decode on this device is now good (15.3 tok/s bench, ~11 t/s server) thanks to the int-dot mat-vec — this report is purely about the batched path.
• With fp16: 0, the batched mul_mat dequant path runs f32 throughout (matmul_ptq1_0_f32); NVIDIA proprietary Vulkan on Pascal is legacy. The BC-250 numbers (pp512 95 t/s, same prism code, fp16 supported + RADV/Mesa) suggest fp16 compute + driver compiler explain most of the 6x.
• Correctness verified on this setup (chat + vision; kv-mean-center calibration round-trips cleanly) — performance report only.

Ask

1. Is there a roadmap item for a batched PTQ1_0 path that helps devices without fp16 compute (e.g. an MMQ-style q8_1 matmul pipeline analogous to the int-dot mat-vec that landed for decode, or an explicit f32 batched kernel tuning pass)?
2. Worth checking whether something specific in the current batched path regresses NVIDIA old-gen: with attention amortized at pp2048 the slope is flat at 16, which looks like pure matmul throughput rather than attention.
3. Happy to test candidate builds on this device (bench + test-backend-ops runs, PPL checks) if useful.

Environment for reproducibility: Win10 22H2, driver 535.98, -ngl 99, FA on, q4_0 KV with fork's --kv-mean-center bias file.