metal: route skinny f16 matmuls through mlx gemv_wide (~2.9x on M4)

[czoli1976]

f16 products of a few rows against a large transposed weight — x[M,K] @ w[N,K]ᵀ
with M in 2..15, which is what a small batch or a speculative decode step
produces — currently go to a GEMM tile that pads M to 64 and leaves most of the
bandwidth unused. MlxGemm only takes the gemv path at M == 1.

This ports mlx's gemv_wide kernel (ml-explore/mlx#3888) into mlx_gemv.metal,
same vendoring idiom as the rest of that file, and routes those shapes to it,
keeping mlx's own gate: f16, K a multiple of 4, N > 1, and architecture
generation ≥ 15.

Numbers

bench_skinny_gemm (added here), K=2048, ms/dispatch, f16, x @ w.T.

M4, 8-core GPU (applegpu_g16g, macOS 26.6) — before vs after, through the
real MlxGemm dispatch:

M	before	after
2	0.218	0.073	2.99×
4	0.220	0.078	2.83×
8	0.218	0.077	2.83×

M == 1 and M ≥ 16 are untouched. At N=32000 the same band gains 1.34×.

The kernel also beats routing these shapes to GgmlGemm, which was the other
option: after this change Ggml is 1.10× (M=2) to 1.80× (M=8) slower than the MLX
path at N=2048.

On the architecture gate

mlx keeps gemv_wide off below generation 15 on the grounds that pre-M3 parts
are limited by load issue rate rather than bandwidth. That gate is kept here, but
for what it is worth the kernel is a clear win on an M1 Pro too (applegpu_g13s,
generation 13): 0.123 → 0.043 ms at M=2, 0.064 at M=4, 0.083 at M=8, i.e.
1.5–2.9×. Happy to widen the gate if you would rather take that; I have no M2 to
check the rest of the pre-M3 range, which is why I left mlx's rule alone.

f32 is deliberately excluded, matching mlx, which instantiates f16/bf16 only:
measured on the M4 it is 0.50–0.67× at M ≥ 4, i.e. a regression.

Validation

cargo test -p tract-metal --release on the M4 (where the gate is live): 81
passed, 1 failed — that one failure is test_mfa_attention_causal_const_is_noop,
pre-existing on main and unrelated (#2546). Same on an M1 Pro, where the gate
leaves the path inactive. New tests: gemv_wide_* cover f16 M=2..15, f32, tail
rows and the gate itself against a host reference, and
mlx_gemm_skinny_f16_matches_reference drives M=2..15 through the real dispatch.
fmt and clippy clean.

mlx (ml-explore/mlx) is MIT, Copyright (c) 2023-2025 Apple Inc.; attributed in
the source header.

🍍

[czoli1976]

On the architecture gate: I've posted the M1 Pro measurements upstream as ml-explore/mlx#3967 in case they want to revisit the generation-15 cutoff. This PR keeps mlx's gate either way.

[github-actions]

🔴 Bench vs main — 1 speed regression(s) · ⚠️ 5 secondary

Reference: 2026-09-01 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +5.4%	dummy_conmer_12M evaltime · pulse_120ms	apple-m1-max	1.79 ms/pulse 0.0149 RTF → 1.89 ms/pulse 0.0158 RTF

Improvements

Δ	metric	device	main → PR
🟢 -16.3%	speaker_id evaltime · pulse8	i9-11900kb_rtx-4060	0.0639 ms/pulse 0.000799 RTF → 0.0535 ms/pulse 0.000668 RTF
🟢 -11.5%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s evaltime · cpu	i9-11900kb_rtx-4060	0.765 ms → 0.677 ms
🟢 -8.3%	speaker_id evaltime · pulse8	cortex-a55	0.831 ms/pulse 0.0104 RTF → 0.762 ms/pulse 0.00952 RTF
🟢 -7.4%	speaker_id evaltime · pulse8	cortex-a53	1.13 ms/pulse 0.0142 RTF → 1.05 ms/pulse 0.0131 RTF
🟢 -6.9%	speaker_id evaltime · pulse8	apple-m1-max	0.0593 ms/pulse 0.000741 RTF → 0.0552 ms/pulse 0.00069 RTF

+2 more improvement(s)

Δ	metric	device	main → PR
🟢 -6.6%	speaker_id evaltime · pulse8	cortex-a7	3.43 ms/pulse 0.0429 RTF → 3.21 ms/pulse 0.0401 RTF
🟢 -5.4%	speaker_id evaltime · pulse8	orangepi-rv2	2.36 ms/pulse 0.0295 RTF → 2.23 ms/pulse 0.0279 RTF

⚠️ 5 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +42.2%	arm_ml_kws_cnn_m load · pass	cortex-a9	83 ms → 118 ms
⚠️ +40.0%	arm_ml_kws_cnn_m load · pass	beaglev-ahead	75 ms → 105 ms
⚠️ +37.2%	arm_ml_kws_cnn_m load+optimize · pass	cortex-a9	129 ms → 177 ms
⚠️ +31.2%	arm_ml_kws_cnn_m load+optimize · pass	beaglev-ahead	112 ms → 147 ms
⚠️ +10.3%	arm_ml_kws_cnn_m load+optimize · pass	cortex-a7	116 ms → 128 ms

[kali]

/ci benches

[github-actions]

🔴 Bench vs main — 1 speed regression(s) · ⚠️ 7 secondary

Reference: 2026-09-01 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

Δ	metric	device	main → PR
🔴 +5.7%	dummy_conmer_12M evaltime · pulse_120ms	apple-m1-max	1.79 ms/pulse 0.0149 RTF → 1.9 ms/pulse 0.0158 RTF

Improvements

Δ	metric	device	main → PR
🟢 -15.9%	speaker_id evaltime · pulse8	i9-11900kb_rtx-4060	0.0639 ms/pulse 0.000799 RTF → 0.0537 ms/pulse 0.000671 RTF
🟢 -11.5%	parakeet_tdt_600m_v3_f32f32_preprocessor_1s evaltime · cpu	i9-11900kb_rtx-4060	0.765 ms → 0.677 ms
🟢 -7.3%	speaker_id evaltime · pulse8	cortex-a53	1.13 ms/pulse 0.0142 RTF → 1.05 ms/pulse 0.0131 RTF
🟢 -6.8%	speaker_id evaltime · pulse8	orangepi-rv2	2.36 ms/pulse 0.0295 RTF → 2.2 ms/pulse 0.0275 RTF
🟢 -6.6%	speaker_id evaltime · pulse8	apple-m1-max	0.0593 ms/pulse 0.000741 RTF → 0.0554 ms/pulse 0.000692 RTF

+1 more improvement(s)

Δ	metric	device	main → PR
🟢 -5.7%	speaker_id evaltime · pulse8	cortex-a9	2.65 ms/pulse 0.0332 RTF → 2.5 ms/pulse 0.0313 RTF

⚠️ 7 secondary regression(s)

Δ	metric	device	main → PR
⚠️ +37.3%	arm_ml_kws_cnn_m load · pass	beaglev-ahead	75 ms → 103 ms
⚠️ +27.7%	arm_ml_kws_cnn_m load+optimize · pass	beaglev-ahead	112 ms → 143 ms
⚠️ +19.9%	hey_snips_v4_model17 load · 2sec	cortex-a7	4.27 s → 5.12 s
⚠️ +19.9%	hey_snips_v4_model17 load+optimize · 2sec	cortex-a7	5.06 s → 6.06 s
⚠️ +14.5%	arm_ml_kws_cnn_m load · pass	cortex-a9	83 ms → 95 ms
⚠️ +10.9%	arm_ml_kws_cnn_m load+optimize · pass	cortex-a9	129 ms → 143 ms
⚠️ +6.5%	en_tdnn_15M RSS @ ready · pulse_120ms	cortex-a55	112 MB → 120 MB

[kali]

today's dummy_conmer on m1-max is a freak bench.