Skip to content

metal : opt-in PQ2_0 mat-vec for two columns (GGML_METAL_PQ2_MULTICOL=1) - #279

Open
jasontitus wants to merge 1 commit into
PrismML-Eng:prismfrom
jasontitus:downstream/metal-pq2-multicol
Open

jasontitus wants to merge 1 commit into
PrismML-Eng:prismfrom
jasontitus:downstream/metal-pq2-multicol

Conversation

@jasontitus

Copy link
Copy Markdown

Overview

Two-column PQ2_0 products (two requests, or one request with an MTP draft of 1) take mul_mv_ext. This adds an opt-in two-column mat-vec, GGML_METAL_PQ2_MULTICOL=1, in the style of #262: each byte's three floors are computed once for both columns, and each column's collapse coefficients once for all rows. The collapse arithmetic is kernel_mul_mv_pq2_0_f32's. From three columns mul_mv_ext stays faster, so only two columns use it. Off by default; batched and broadcast src1 are handled like the other mat-vec kernels.

Additional information

M5 Max, Ternary Bonsai 2 27B PQ2_0, this branch, switch off vs on, three A-B-B-A quartets with 8 s cooldowns (tok/s; llama-server rates are aggregate, 128 greedy tokens):

Case Off On Speedup [quartet range]
tg128 (control) 45.35 45.19 0.997 [0.987-1.006]
pp2 57.94 68.11 1.175 [1.172-1.181]
pp3 (control) 71.51 71.74 1.003 [0.995-1.011]
llama-server, 2 requests 48.12 55.51 1.154 [1.135-1.186]
llama-server, 1 request, MTP draft 1 45.00 52.42 1.165 [1.153-1.177]

Generated tokens identical in every pair.

Correctness:

  • test-backend-ops -o MUL_MAT: all pass with the switch off and on; the new kernel runs for plain, batched (ne12 = 2) and broadcast (r2 = 2) cases. New cases: two columns with a row tail and broadcast, and with a strided B.
  • Full model, batches of 1-8 tokens (108 positions): switch off, logits bitwise equal to upstream; switch on first differs at the first 2-token batch, max NMSE 3.9e-10, same top token at every position.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES. Claude (Anthropic) helped with the implementation, testing, measurements and this description; I reviewed the change and own it.

@jasontitus
jasontitus force-pushed the downstream/metal-pq2-multicol branch from 9ae2fda to 89bde0c Compare September 26, 2026 04:51
jasontitus added a commit to jasontitus/llama.cpp that referenced this pull request Sep 26, 2026
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YHqq1nTe46u7euASncLsee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant