Skip to content

rocm: implement the two DSpark kernels so --dspark can draft on gfx1151 (fixes #597) - #670

Open
zom-2018 wants to merge 1 commit into
antirez:mainfrom
zom-2018:rocm-dspark-kernels
Open

rocm: implement the two DSpark kernels so --dspark can draft on gfx1151 (fixes #597)#670
zom-2018 wants to merge 1 commit into
antirez:mainfrom
zom-2018:rocm-dspark-kernels

Conversation

@zom-2018

@zom-2018 zom-2018 commented Aug 3, 2026

Copy link
Copy Markdown

Summary

Fixes #597 — "DSpark never produces a draft on ROCm / gfx1151".

Root cause: the DSpark engine path in ds4.c is backend-shared, but two GPU
kernels it needs existed only in ds4_cuda.cu. ROCm builds resolved them
through ROCM_UNAVAILABLE_INT stubs in ds4_rocm_unavailable.cu, so the
session probe always failed and every propose cycle ended with proposed=0.

This PR implements both kernels for ROCm (HIP) and removes the two stubs:

  • dspark_markov_argmax_kernel + ds4_gpu_dspark_markov_argmax_tensor
    (rocm/ds4_rocm_indexer.cuh): on-device markov-chain argmax over the vocab
    with q8_0 w1/w2 rows — direct port of the CUDA kernel, same monotonic-key
    atomicMax tie-break.
  • attention_noncausal_raw_batch_heads_kernel +
    ds4_gpu_attention_noncausal_raw_batch_heads_tensor
    (rocm/ds4_rocm_attention_launch.cuh): non-causal batch attention over the
    raw KV ring for draft blocks — direct port, same one-block
    max/denominator/value accumulation order; keeps the
    DS4_DSPARK_VERIFY_NONCAUSAL host-side oracle for the first 3 launches.

Signatures match the existing declarations in ds4_gpu.h; no other files are
touched.

Scope / safety

  • ROCm-only change. No effect on CUDA, Metal, or CPU builds.
  • No behavior change unless --dspark is used (the kernels are only reached
    through the DSpark propose/verify path).

Measured on AMD Strix Halo (Ryzen AI Max+ 395, Radeon 8060S gfx1151), ROCm 7.2

Before (upstream): proposed=0 every cycle (exactly the #597 symptom).

After, --dspark-confidence 0.3, 300 generated tokens, DeepSeek-V4-Flash-0731
abliterated GGUF (80.76 GiB) + its DSpark support drafter:

cycles=89 proposed=216 accepted_draft=206 accept_rate=95.37% avg_accept=2.315
full=39 partial=5 draft_len_hist=1:8,2:5,3:1 accepted_len_hist=0:118,1:10,2:3,3:1

At the default --dspark-confidence 0.9 the adaptive scheduler stays
conservative: proposed=21, accept_rate=90.5%.

Wall-clock is currently break-even on this bandwidth-bound APU (the 6-row
target verify costs ~4–5x a single decode pass, dominated by routed-expert
weight reads), so this is not a speedup claim — it makes DSpark functional
on ROCm, which is the precondition for any follow-up verify-path optimization.

Testing

  • make strix-halo clean (gfx1151).
  • CLI generation with --dspark + DS4_DSPARK_STATS=1: drafts proposed,
    verified and accepted; coherent output; stats counters advance.
  • Without --dspark: unchanged behavior.

…ero-draft)

Port the two CUDA-only DSpark kernels to ROCm so the speculative path
actually drafts on Strix Halo:

- dspark_markov_argmax_kernel + ds4_gpu_dspark_markov_argmax_tensor
  (rocm/ds4_rocm_indexer.cuh): on-device markov-chain argmax over the
  vocab with q8_0 w1/w2 rows, same monotonic-key atomicMax tie-break.
- attention_noncausal_raw_batch_heads_kernel +
  ds4_gpu_attention_noncausal_raw_batch_heads_tensor
  (rocm/ds4_rocm_attention_launch.cuh): non-causal batch attention over
  the raw KV ring for draft blocks, incl. DS4_DSPARK_VERIFY_NONCAUSAL
  host oracle (first 3 launches).
- drop both ROCM_UNAVAILABLE stubs.

Previously the engine probe failed on ROCm (unavailable stubs), so
--dspark entered the propose path but emitted proposed=0 every cycle.

(cherry picked from commit d3c8e665f62d41d384145e4b5f807b93048bd058)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DSpark never produces a draft on ROCm / gfx1151: 1197 cycles, proposed=0, prop_hidden=0.000

2 participants