Skip to content

MTP for qwen4_exp (Qwen3.8-Flash-Next): is the draft/verify path planned beyond DSV4? #421

Description

@Rhonstin

The roadmap (#79) lists speculative decoding including MTP, and #69 lays out an exact DSpark draft/verify implementation for DeepSeek-V4. Both are DSV4-scoped, so I could not tell from either whether qwen4_exp is in scope.

Qwen3.8-Flash-Next ships its own MTP head — mtp.layers.0.mlp.experts.*, 128x128 block-FP8, ~3.1k tensors — and models/qwen4_exp/weight.py currently drops it outright:

Dropped: mtp.* (speculative head, including its stacked mtp.layers.0.mlp.experts.*) and model.visual.* (served text-only)

Two questions:

  1. Is MTP for qwen4_exp planned, and roughly where does it sit relative to the DSV4 work in [2/3] feat(dsv4): implement exact DSpark speculative decoding #69?
  2. Will the draft/verify machinery from [2/3] feat(dsv4): implement exact DSpark speculative decoding #69 be generalised into a shared abstraction, or is DSpark expected to stay DSV4-specific with a separate implementation per architecture?

Asking because the payoff looks larger on the offload path than on a resident one. On an RTX 3090 (PCIe 3.0 x16, 104 GB host RAM) serving RadixArk/Qwen3.8-Flash-Next-NVFP4, decode runs at ~19 tok/s while prefill reaches ~910 tok/s on a 39k prompt — the gap is expert transfer per decode step, which is exactly what verifying gamma tokens in one pass would amortise.

Happy to test a branch on that setup if it helps.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions