You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The roadmap (#79) lists speculative decoding including MTP, and #69 lays out an exact DSpark draft/verify implementation for DeepSeek-V4. Both are DSV4-scoped, so I could not tell from either whether qwen4_exp is in scope.
Qwen3.8-Flash-Next ships its own MTP head — mtp.layers.0.mlp.experts.*, 128x128 block-FP8, ~3.1k tensors — and models/qwen4_exp/weight.py currently drops it outright:
Dropped: mtp.* (speculative head, including its stacked mtp.layers.0.mlp.experts.*) and model.visual.* (served text-only)
Asking because the payoff looks larger on the offload path than on a resident one. On an RTX 3090 (PCIe 3.0 x16, 104 GB host RAM) serving RadixArk/Qwen3.8-Flash-Next-NVFP4, decode runs at ~19 tok/s while prefill reaches ~910 tok/s on a 39k prompt — the gap is expert transfer per decode step, which is exactly what verifying gamma tokens in one pass would amortise.
The roadmap (#79) lists speculative decoding including MTP, and #69 lays out an exact DSpark draft/verify implementation for DeepSeek-V4. Both are DSV4-scoped, so I could not tell from either whether
qwen4_expis in scope.Qwen3.8-Flash-Nextships its own MTP head —mtp.layers.0.mlp.experts.*, 128x128 block-FP8, ~3.1k tensors — andmodels/qwen4_exp/weight.pycurrently drops it outright:Two questions:
qwen4_expplanned, and roughly where does it sit relative to the DSV4 work in [2/3] feat(dsv4): implement exact DSpark speculative decoding #69?Asking because the payoff looks larger on the offload path than on a resident one. On an RTX 3090 (PCIe 3.0 x16, 104 GB host RAM) serving
RadixArk/Qwen3.8-Flash-Next-NVFP4, decode runs at ~19 tok/s while prefill reaches ~910 tok/s on a 39k prompt — the gap is expert transfer per decode step, which is exactly what verifying gamma tokens in one pass would amortise.Happy to test a branch on that setup if it helps.