ROCm: restore DeepSeek V4 Flash decode performance and Q4 SSD streaming - #623
Conversation
|
Works for me, thanks! Gen speed increased from ~9.16 t/s to ~15.6 t/s with DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf. Before: After: |
|
Looks good so far here, one thing to note, not sure this is even meant to be supported, but using the IQ2 quant + the ssd-streaming flag causes a hang after prefill. (Q4K is fine). Trying distributed in a moment. |
|
With these settings: It constantly very soon gets into loops in thinking. |
Did you have that also before? Just trying to establish if this is a regression, I didn't do much testing on SSD streaming as I don't use it. |
Unfortunately I haven't tried That looks very suspicious because it starts looping very early on both temp 0 and temp 1.0 on the same prompt that |
|
With these settings I definitely see a regression: Current main: 54b36ed ... and it goes into loops. Current PR 623: |
With these settings on current main it doesn't even run: |
|
So, summarizing my observations... that's probably not a regression but just behavior change in already broken implementation of SSD streaming on ROCm with DeepSeek. With DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf
With DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf
No issues with GLM. |
If you do SSD streaming with the smaller model, does it have the same behaviour? And I imagine with the small model totally resident in memory you do not observe the looping behaviour, right? |
I'll try to verify, but from my current observation |
|
Now I've tested with Here's with PR 623 build. With srteaming: It ended up looping over the same few phrases pretty soon and exhausted the whole context. Then I removed SSD streaming: It finished well and gen speed was higher. |
|
For people following this, I have completed SWE bench for the Q4 distributed one and the quality seems to have gone down a bit, so I'm trying to see if I can recover the original quality from a month ago. |
|
Independent cross-validation (from #640, which I'm closing in favor of this PR): I root-caused the same regression independently via git bisect over the 51 commits between
Restoring the four preq decode call sites (single matmul / paired / HC-expand / attention-output) recovers 15.03 t/s (real chat 16.63 t/s, first-token 120 ms → 68 ms) — matching this PR's numbers (9.0 → 15.5 t/s at ctx 2048). Notes from my side:
|
|
Tested this PR on Strix Halo (gfx1151, Radeon 8060S, ROCm 7.2.3, 128 GB unified), DeepSeek-V4-Flash IQ2XXS
Adding some per-stage data that may be useful, since it pins the cause to the dispatch rather than the kernels. I wrapped every non-GLM
Every Q8_0 path regressed 1.9-2.7x while F16 matmuls, the IQ2/Q2_K routed MoE and attention decode were untouched — consistent with single-token decode falling through to the f32 shared-x kernels, which run a 256-thread block per output row instead of a warp per row. Two side notes:
|
@kyuz0 any news on this? |
|
@darkbasic what kind of news? :D It is a merged PR. |
After A LOT of digging, I think the issue was with the way I score SWE bench. It turns our Gemini 3.1 Pro as an LLM judge can be too generous, so I am moving away from that and going to simple unit tests, which is going to give the LLM less room for creativity. So, to test the performance of this model instead, I made a version of Terminal bench that has longer horizon tasks that stress operations over a long context (which SWE bench doesn't do), I selected 20 tasks and run it, and the Q4 model got 19/20. My control, llama.cpp with the unsloth Q8 one... it's still going but I think they have inference issues as it's failing or timing out on a lot of tasks. Unfortunately the strix halo situation with llama.cpp at the moment is hit and miss, there are custom quants like rocmfpx, quite a few Vulkan and ROCm forks and patches around to spot fix individual gfx1151 quirks on kernels. I realistically would need to take 1-2 weeks to be able to go through every of them and properly test it, most people only test pp/tg, but when you get the model on a real long horizon task benchmark, a lot of implementations seems to fail. Right now, I am also getting the smaller IQ2_XSS on my mini terminal bench to see how well it does compared to the full version. |
7 participants
Summary
This restores the ROCm DeepSeek V4 Flash paths that regressed when the backend
moved Q8 decode activations to FP32:
low-rank attention-output projections;
the pointer-backed routed-MoE launch;
the SSD fix explicitly excludes
DS4_MODEL_FAMILY_GLM_DSA;--qualityon the existing FP32 activation path;DS4_ROCM_DSV4_PREQUANT_DECODE=0as a diagnostic rollback.Tested models
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.ggufDeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.ggufPerformance
AMD Radeon 8060S / gfx1151, ROCm 7.2.4.
Resident IQ2, 64-token decode:
Prefill stayed effectively unchanged. Decode improved by 70-76%.
Two-node Q4 decode, layers
0:21 / 22:output, compared with the currentupstream base (
antirez/mainat54b36ed):Profiling localized the regression to Q8 attention and shared-expert
projections. Routed Q4 MoE performance was unchanged.
Distributed prefill also benefits from the existing pipeline when DeepSeek
uses smaller chunks. For a 4096-token prompt:
--dist-prefill-chunkThe recommended two-node DeepSeek setting is:
In the normal frontier benchmark this improved prefill by 24.5% at context
2048 and 41.9% at context 4096. Decode performance was unchanged.
Correctness
The DeepSeek Flash 100-case fixture scored 2,289 continuation tokens.
Resident IQ2, restored Q8 path versus FP32 activation path:
Distributed Q4, exact PR candidate against the same hosted-reference fixture:
The resident A/B found no measurable quality regression. The distributed
candidate completed all 100 cases, but no clean distributed A/B against the
current upstream base was collected.
Q4 SSD streaming previously failed on its first decode with:
With this PR it decoded coherently through the 512, 1024, and 2048 frontiers
under repeated expert-cache replacement. No missing-expert failure, OOM,
non-finite output, or incorrect decode was observed.
Two-node example
The same Q4 model and context size must be available on both nodes.
Worker:
Coordinator: