Conversation
|
Tested on an Intel Arc B390 (Panther Lake Xe3 iGPU, UMA; the Vulkan backend treats it as Xe2-class, so #238's integer-dot path is its default). Windows 11, MSYS2 GCC 16.2, Heads-up for anyone reproducing: Correctness: Kernel,
That's ~9x on the fallback shader, which lands it level with the integer-dot path on this GPU. End-to-end,
* The first tip run's 27B pp512 was high in round 1 only (~215 vs ~160 in every other run); I read it as a first-run outlier. Prefill goes through This matches the scope in the description: no change on the default path for an integer-dot device, and a large win on the fallback, which becomes roughly as fast as MMVQ here. That should matter on Intel/other GPUs without integer dot product, and for the Tested with Claude Code. |
|
Thanks for testing this — Arc B390 is exactly the hardware I couldn't cover, and the 2B model is a case I never ran. Good catch on For a second data point, same methodology (paired runs alternated against an unmodified build of the parent commit, same toolchain, to cancel thermal drift on a laptop) — RX 6700M, RDNA2/gfx1031, Windows 11, Vulkan SDK 1.4.357, MSVC, Bonsai 2 27B PTQ1_0,
Perplexity over a fixed 512-token sample was identical on both builds (8.6924 +/- 1.43177), and One difference worth noting between our GPUs: on RDNA2 the integer-dot path is still about 2.4x faster than this float shader (24.6 vs 10.4 t/s), whereas on your Xe3 they come out level. So on AMD this remains strictly a fallback-path improvement, as the description says, rather than a candidate for the default path. Known limitation, in case it matters for review: the multi-column variants still re-decode the weights per column, so batch 2 costs ~1.8x batch 1 here. I tried hoisting the decode above the column loop and it regressed single-token throughput (register pressure), so the simpler form is what's in the PR. Also, only Written with Claude Code. |
76798f9 to
e22fa21
Compare
|
Tested on a card without integer dot, which is the case this PR is for: AMD RX 570 (Polaris10 / gfx803), Linux, RADV Mesa 26.1.2,
So 4.5x on generation on GCN, prompt unchanged as expected. For comparison, the table-decode patch I linked in #185 gets 6.99 tg128 on the same card, so this is slightly ahead and makes the mat-vec half of mine unnecessary. Mine also changes the Builds and measurements were AI-assisted. |
PTQ1_0 has no float mat-vec shader of its own, so it falls back to the generic mul_mat_vec.comp, which reaches it through dequantize()/dequantize4() and calls ptq1_0_trit() once per weight: an 8-bit load plus a loop of up to four multiplies to skip the earlier trits in the same byte. Every packed byte is re-read and re-decoded five times. Since PrismML-Eng#238 an integer-dot mat-vec (mul_mat_vecq_ptq1_0.comp) covers devices with VK_KHR_shader_integer_dot_product, and ggml_vk_should_use_mmvq() picks it for k >= 2048 on AMD and NVIDIA, so on those GPUs this shader is not on the hot path. It still matters for devices without integer dot support, for k < 2048, and whenever MMVQ is declined or disabled. mul_mat_vec_ptq1_0.comp reads each 28-byte block as seven 32-bit words and peels the five trits off a byte with the base-3 recurrence two bytes at a time in 16-bit lanes (255*3 = 765 stays inside a lane), which is 10 multiplies per 20 weights instead of about 60. Consecutive bytes of a word map to consecutive elements for a fixed trit, so each trit step consumes one vec4 of activations. (trit - 1) is split into sum(trit*y) - sum(y) so the row-independent term is gathered once per work item. A work item is half a block so rows of 5120 still spread across the workgroup, and the loop order (word, row, column) keeps the register footprint independent of NUM_COLS. Measured on an RX 6700M (RDNA2, Windows, Vulkan 1.4.357) with Ternary-Bonsai-2-27B-PTQ1_0, -ngl 99 -fa 1 -ctk q4_0 -ctv q4_0, against an unmodified build of the parent commit with the same toolchain, runs alternated to cancel thermal drift: GGML_VK_DISABLE_MMVQ=1 (this shader on the hot path) tg64 3.50 -> 10.44 t/s (2.98x) default (MMVQ active, this shader off the hot path) tg128 24.6 -> 24.8 t/s (unchanged, within noise) pp512 71.0 -> 70.4 t/s (mul_mm untouched, within noise) test-backend-ops -b Vulkan0: MUL_MAT 28/28 and MUL_MAT_ID 75/75 pass for type_a=ptq1_0; the full suite is 17187/17190 and the 3 GATED_DELTA_NET failures also fail on an unmodified build (36/39 both). Perplexity over a fixed 512-token sample is identical to the unmodified build: 8.6924 +/- 1.43177.
e22fa21 to
3958846
Compare
3 participants
What
PTQ1_0 has no float mat-vec shader of its own, so it falls back to the generic
mul_mat_vec.comp, which reaches it throughdequantize()/dequantize4()and callsptq1_0_trit()once per weight: an 8-bit load plus a loop of up to four multiplies toskip the earlier trits packed in the same byte. Every packed byte is re-read and
re-decoded five times. A per-op profile (
GGML_VK_PERF_LOGGER=1) on RDNA2 showedMUL_MAT_VEC ptq1_0at 93% of the time per token and ~130 GFLOPS, while plain f32mat-vec in the same graph reached ~1700 GFLOPS.
Scope, after #238: the integer-dot shader (
mul_mat_vecq_ptq1_0.comp) alreadycovers devices with
VK_KHR_shader_integer_dot_product, andggml_vk_should_use_mmvq()selects it fork >= 2048on AMD and NVIDIA, so on thoseGPUs this shader is not on the hot path and the numbers there are unchanged. It
still matters for:
k < 2048,mmvq_mode == -1/GGML_VK_DISABLE_MMVQ=1.So this is an improvement to the fallback path, not to the default path on recent
discrete GPUs.
How
16-bit lanes —
255*3 = 765stays inside a lane, so lanes never interfere. 10multiplies per 20 weights instead of about 60.
trit step consumes exactly one
vec4of activations.weight = (trit - 1) * dis evaluated asd * (sum(trit*y) - sum(y)), so therow-independent
sum(y)is gathered once per work item.qhword, or words 3..5) so rows of5120 still spread across the workgroup.
vector is loaded once per column, and the register footprint stays independent of
NUM_COLS(which goes up to 8).Routing matches how
tq2_0and the k-quants are handled: one condition invulkan-shaders-gen.cpp.mul_mm,mul_mat_vecq,get_rows,dequantand theCPU/CUDA/Metal paths are untouched, and no other quant type is affected.
Benchmarks
AMD Radeon RX 6700M (RDNA2, gfx1031), Windows 11, Vulkan SDK 1.4.357, MSVC.
Ternary-Bonsai-2-27B-PTQ1_0,-ngl 99 -fa 1 -ctk q4_0 -ctv q4_0.This branch vs an unmodified build of the parent commit, same toolchain, runs
alternated to cancel thermal drift.
With the dequant path forced (
GGML_VK_DISABLE_MMVQ=1) — what this PR changes:Default settings, where MMVQ takes over on this GPU — unchanged, as expected:
Both default-path differences are inside the run-to-run spread on this laptop.
Aside for Windows + AMD users reproducing this: a model that does not fit the
CPU-visible VRAM heap gets partly backed by system memory and loses 10-15x
regardless of this PR.
GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1was set for everymeasurement above, on both sides.
Correctness
test-backend-ops -b Vulkan0 -o MUL_MAT -p type_a=ptq1_0: 28/28 pass.test-backend-ops -b Vulkan0 -o MUL_MAT_ID -p type_a=ptq1_0: 75/75 pass.test-backend-ops -b Vulkan0: 17187/17190. The 3 failures areGATED_DELTA_NET(type=f32, ..., raw_gates=1)and are pre-existing — an unmodifiedbuild fails the identical 3 (
-o GATED_DELTA_NETis 36/39 on both).-c 128 --chunks 4 -ub 1): 8.6924+/- 1.43177 on both the unmodified build and this branch. At
-ub 4on thisbranch: 8.6935.
of a 300-word prompt before diverging at a near-tie, as expected from a different
summation order.
Not covered
GCN hardware to verify on, though the decode work removed is vendor-independent.
have that model locally to measure.
~1.8x batch 1 here). An earlier variant that hoisted the decode differently was
slower on single tokens, so the faster of the two measured forms is kept.