Skip to content

metal : PTQ1_0 multi-column mat-vec for 5-8 columns - #277

Open
jasontitus wants to merge 1 commit into
PrismML-Eng:prismfrom
jasontitus:downstream/metal-ptq1-multicol-8
Open

jasontitus wants to merge 1 commit into
PrismML-Eng:prismfrom
jasontitus:downstream/metal-ptq1-multicol-8

Conversation

@jasontitus

Copy link
Copy Markdown

Overview

#262 added GGML_METAL_PTQ1_MULTICOL=1, a PTQ1_0 mat-vec for 2-4 columns; 5-8 columns still take mul_mv_ext. Batched decode, MTP verification and multi-request steps often produce 5-8 columns (4 requests with MTP draft 1 are 8).

This PR splits 5-8 columns into two tiles of at most four (5 -> 3+2, 6 -> 3+3, 7 -> 4+3, 8 -> 4+4) on the existing c2/c3/c4 kernels:

Additional information

M5 Max, Ternary Bonsai 2 27B PTQ1_0, this branch, GGML_METAL_PTQ1_MULTICOL=1 in both arms, max 4 (#262) vs max 8, three A-B-B-A quartets with 8 s cooldowns (tok/s; MTP rows are llama-server with draft-mtp, draft 1, aggregate, 128 greedy tokens):

Case Columns #262 This PR Speedup [quartet range]
pp4 (control) 4 69.86 70.34 1.007 [1.000-1.013]
pp5 5 45.14 61.82 1.370 [1.368-1.373]
pp6 6 30.76 74.78 2.433 [2.373-2.552]
pp7 7 34.91 73.20 2.097 [2.059-2.164]
pp8 8 40.45 86.82 2.146 [2.138-2.161]
tg128 (control) 1 43.67 43.39 0.994 [0.987-1.003]
MTP, 3 requests 6 24.48 52.23 2.134 [2.123-2.148]
MTP, 4 requests 8 28.53 55.07 1.932 [1.888-2.023]

Generated tokens identical in every pair.

Correctness:

  • test-backend-ops -o MUL_MAT: 1493/1493 with the switch off, on with max 4, on with max 8 (the partial-tile pipelines compile and run), and with an out-of-range max. New cases: 5-7 columns with row tails and broadcast, and 5 and 7 with a strided B.
  • Full model, batches of 1-8 tokens (108 positions): switch off, logits bitwise equal to upstream; max 4, bitwise equal to metal : add PTQ1_0 mat-vec for two to four columns #262; max 8 first differs at the first 5-token batch (different summation order than mul_mv_ext), max relative difference 3.7e-5, same top token at every position.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES. Claude (Anthropic) helped with the implementation, testing, measurements and this description; I reviewed the change and own it.

GGML_METAL_PTQ1_MULTICOL=1 (PrismML-Eng#262) runs the PTQ1_0 multi-column kernel for 2-4 columns; 5-8 still take the
generic mul_mv_ext path. This splits 5-8 columns into two tiles of at most four (5 -> 3+2, 6 -> 3+3, 7 -> 4+3,
8 -> 4+4) on the existing c2/c3/c4 kernels, so batched decode, MTP verification and multi-request steps up to
eight tokens stay on it.

- a partial last tile clamps its column reads to the last valid column and skips the writes past it; a bool
  function constant (FC_MUL_MV + 5) removes those checks when every tile is complete, so 2-4 columns compile to
  exactly the PrismML-Eng#262 kernel (logits bitwise identical)
- GGML_METAL_PTQ1_MULTICOL_MAX (clamped to 4..8, default 8) caps the widths the kernel takes; 4 restores PrismML-Eng#262
- test-backend-ops: 5-7 columns with row tails and broadcast, and 5/7 with a strided B

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YHqq1nTe46u7euASncLsee
jasontitus added a commit to jasontitus/llama.cpp that referenced this pull request Sep 26, 2026
…(PQ2_0 two columns) measured and ready

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YHqq1nTe46u7euASncLsee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant