Conversation
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DSrpF2XYi9dav1e3hSAkaS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
SYCL has MMQ disabled for all types, so quantized prompt batches take the dequantize-then-GEMM path in
ggml_sycl_op_mul_mat_sycl. In the default build (GGML_SYCL_F16=OFF) that path expands the weights to FP32 and runs an FP32 GEMM, which does not use the XMX matrix engines on Intel GPUs. This PR sends PTQ1_0 and PQ2_0 down the existing FP16 branch instead: half the bytes per dequantized weight, and oneDNN (or MKL withGGML_SYCL_ENABLE_DNN=0) runs an FP16 GEMM with FP32 output.PTQ1_0 and PQ2_0 weights are exactly representable after FP16 expansion; FP16 conversion of activations and the GEMM arithmetic can still change results. This PR enables the path by default only for the two measured formats. Other quantized types retain the FP32 default and can still use the FP16 path with
GGML_SYCL_F16=ON. Decode is unchanged, since single tokens and small batches go through MMVQ.Additional information
Intel Arc 140V (Lunar Lake iGPU), Windows, oneAPI 2025.3, driver 32.0.101.9030, Release. Base
prism@ adfffbe, Ternary-Bonsai-2-27B.llama-bench -ngl 99 -fa 1 -r 1, four rounds in separate processes 30 s apart with the arm order flipped each round; median (range) in t/s:This laptop lowers its memory clock under sustained load, which accounts for the spread.
Correctness:
test-backend-ops -o MUL_MAT: PQ2_0 142/142 and PTQ1_0 165/165, with oneDNN and withGGML_SYCL_ENABLE_DNN=0. 15 cases per type have n = 9, 16 or 64 and take the changed path.llama-perplexity --kl-divergence, WikiText-2,-c 2048 -b 512, 8 chunks, against a base file from prism 0781925 (identical SYCL code). Unmodified base build: PPL ratio 1.000035, mean KLD 0.000000, same top 100%. This PR: PPL ratio 1.000035 +- 0.000036, mean KLD 0.000001, max KLD 0.000898, same top 99.963%. PTQ1_0 and PQ2_0 give identical numbers, as they pack the same trits.Only tested on Xe2. Builds on #235.
Requirements