Skip to content

sycl: dequantize PTQ1_0 and PQ2_0 prompt batches to FP16 (2-3x prompt processing) - #278

Open
dwymark wants to merge 1 commit into
PrismML-Eng:prismfrom
dwymark:sycl-fp16-prompt-gemm
Open

dwymark wants to merge 1 commit into
PrismML-Eng:prismfrom
dwymark:sycl-fp16-prompt-gemm

Conversation

@dwymark

@dwymark dwymark commented Sep 26, 2026

Copy link
Copy Markdown

Overview

SYCL has MMQ disabled for all types, so quantized prompt batches take the dequantize-then-GEMM path in ggml_sycl_op_mul_mat_sycl. In the default build (GGML_SYCL_F16=OFF) that path expands the weights to FP32 and runs an FP32 GEMM, which does not use the XMX matrix engines on Intel GPUs. This PR sends PTQ1_0 and PQ2_0 down the existing FP16 branch instead: half the bytes per dequantized weight, and oneDNN (or MKL with GGML_SYCL_ENABLE_DNN=0) runs an FP16 GEMM with FP32 output.

PTQ1_0 and PQ2_0 weights are exactly representable after FP16 expansion; FP16 conversion of activations and the GEMM arithmetic can still change results. This PR enables the path by default only for the two measured formats. Other quantized types retain the FP32 default and can still use the FP16 path with GGML_SYCL_F16=ON. Decode is unchanged, since single tokens and small batches go through MMVQ.

Additional information

Intel Arc 140V (Lunar Lake iGPU), Windows, oneAPI 2025.3, driver 32.0.101.9030, Release. Base prism @ adfffbe, Ternary-Bonsai-2-27B.

llama-bench -ngl 99 -fa 1 -r 1, four rounds in separate processes 30 s apart with the arm order flipped each round; median (range) in t/s:

base this PR
PQ2_0 pp512 53.8 (52.0-55.0) 119.0 (104.3-132.4) 2.2x
PQ2_0 pp2048 38.0 (32.2-43.2) 116.4 (110.7-129.8) 3.1x
PQ2_0 tg128 6.02 6.02
PTQ1_0 pp512 54.0 (52.3-55.8) 142.4 (131.0-146.3) 2.6x
PTQ1_0 pp2048 45.3 (32.2-46.7) 111.4 (106.1-137.1) 2.5x
PTQ1_0 tg128 4.72 4.94

This laptop lowers its memory clock under sustained load, which accounts for the spread.

Correctness:

  • test-backend-ops -o MUL_MAT: PQ2_0 142/142 and PTQ1_0 165/165, with oneDNN and with GGML_SYCL_ENABLE_DNN=0. 15 cases per type have n = 9, 16 or 64 and take the changed path.
  • llama-perplexity --kl-divergence, WikiText-2, -c 2048 -b 512, 8 chunks, against a base file from prism 0781925 (identical SYCL code). Unmodified base build: PPL ratio 1.000035, mean KLD 0.000000, same top 100%. This PR: PPL ratio 1.000035 +- 0.000036, mean KLD 0.000001, max KLD 0.000898, same top 99.963%. PTQ1_0 and PQ2_0 give identical numbers, as they pack the same trits.

Only tested on Xe2. Builds on #235.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES. Claude Code (Anthropic) found the change while profiling prompt processing, wrote the patch, and ran the measurements above on my machine, under my direction. I reviewed the change and can explain it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DSrpF2XYi9dav1e3hSAkaS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant