Vulkan BC-250 results: packed FP16 PQ2_0 prefill and GDN state-copy fusion

[renovys]

BC-250 (gfx1013/RDNA1, RADV, no cooperative matrices or accelerated integer dot) results for two additional PQ2_0 optimizations, separate from the dedicated mat-vec change.

Packed FP16 prefill. Pairing rows for v_pk_fma_f16 preserves the tested scalar accumulation order. On the 27B PQ2_0 model, llama-bench pp512 improved from 142.94 to 187.78 t/s and pp2048 from 140.97 to 184.70 t/s. Live-server 7k prefill improved from about 129 to 165 t/s. An initial packing layout changed rounding too much against our deployed reference and was rejected. The retained row-pair layout passed the reference-comparison gates. These results apply to this non-coopmat RDNA1 path, not a general recommendation for modern GPUs.

GATED_DELTA_NET + state-copy fusion. Writing the state directly into the strided state-cache destination removes 48 copy dispatches per token. The matcher checks the state-tail source view, destination format/alignment/strides, snapshot count, and aliasing. Only exact elementwise aliasing of the copy destination is exempted from the overlap guard. Added tests cover K=1, snapshots, decode with retained snapshots, partial slots and strided caches.

Live-server matched A/B, using the packed-FP16 build as A and adding fusion for B (same 27B PQ2_0+MTP model, ctx102400 shared across two slots, K q8_0 / V q4_1, batch2048/ubatch512, checkpoint1, nmax5/pmin0.5):

metric	A1	B1	A2	B2
fixed-length generation, tokens/s	27.90	28.96	28.03	29.13
thinking enabled, tokens/s	31.57	33.19	31.82	33.09
7k prefill, tokens/s	165.2	166.0	165.4	166.0
two-stream aggregate, tokens/s	28.19	28.28	26.99	28.55
two real requests, total seconds	110.74	111.75	110.54	108.20

Generation gains reproduced at about 4%; real-request wall time is effectively flat across the two pairs. MTP can change output length even with fixed seeds, so real-request elapsed times should not be read as a controlled kernel-speed measure.

Correctness for the deployed integrated build: backend-ops 17496/17499, with the same three pre-existing raw_gates=1 GDN failures; six new state-copy tests pass. Korean and English/code KL top-1 agreement with the prior deployed reference was 100%. This is reference agreement, not proof of model quality. English long-context and semantic correctness of the real-request responses were not evaluated.

An earlier packed-FP16 B1 result (13.42 t/s median) was contaminated by an automated request arriving during the measurement. B1 began at 08:44:47; its first sample was 29.67 t/s. The scheduled request ran from about 08:45:02 to 08:46:04, overlapping the remaining four samples (15.72, 12.99, 13.42 and 12.63 t/s). That result is excluded from isolated-throughput comparisons. The benchmark checks for idle slots only at startup, so a later request can still contaminate a run.

These changes are deployed locally. Is there interest in reviewing the packed-FP16 path or state-copy fusion separately? AI assistance was used for implementation, review, testing and this report (Devin SWE-2 max and Codex; earlier operational work also used GLM Flash; independent review by Muse max and GLM-5.3 max).

[renovys]

Follow-up deployment data (BC-250 only): after the kernel changes, alternating two rounds of MTP settings favored n_max=3, p_min=0.5 for our workload. Versus n_max=5, median-based averages were +5.8% for non-thinking generation, -1.9% for thinking generation, and about 793 MiB less GTT usage.

We then spent the recovered memory on a larger shared context pool: 102400 → 131072 tokens, keeping two slots, K q8_0 / V q4_1, one context checkpoint and no RAM prompt cache. This context increase added about 952 MiB GTT in our recorded peak comparison.

A single simultaneous test with 62000 input + 128 output tokens per slot completed with HTTP 200, no truncation and the same server PID. Minimum MemAvailable was 335 MiB, only 85 MiB above our provisional 250 MiB guard. This tests 124000 input + 256 output tokens, not full 131072-token saturation, repeated worst-case loads, or semantic quality. Swap/PSI were not recorded during this run. The guard has a 600-second cooldown and three-restarts-per-hour cap, so it is not continuous protection.

Short-prompt generation medians were 30.75 t/s after increasing context and 30.16 t/s after reusing both slots following the long test; these small samples are not a matched equivalence study. The long test took about 23 minutes per request, with scheduling interference; the second request's generation rate was 5.20 t/s. Short-prompt throughput should not be extrapolated to long contexts.

This is a local operating point, not a cross-device recommendation. Our operating response to the first unexpected memory-guard restart or an observed MemAvailable below 250 MiB is to try 122880 tokens first, per the operator’s preference. This intermediate setting is not yet tested; 102400 remains the second fallback if problems persist. This is an operating procedure, not a new automatic context-rollback feature. AI-assisted review and reporting: Codex, Muse max and GLM-5.3 max.

[renovys]

Update: both optimizations were re-validated on the current prism base (adfffbe41). The GDN + state-copy fusion and the packed-FP16 PQ2_0 prefill are now carried as commits on top of adfffbe41 (GDN fusion 84c49b97c, PQ2_0 stack incl. packed-FP16 6748d5612). test-backend-ops on Vulkan0: 17673/17673 pass. Prefill packed-FP16 and decode remain functionally intact on BC-250; PPL matches the previous build to 4 decimal places (9.3437 on wikitext-2 test, 8 chunks). Detailed A/B deltas are confounded with the base bump, so I am reporting functional equivalence rather than isolated per-optimization deltas here.