Repository navigation
[AgentX] MiniMax-M3.1-Flash GB300 SGLang: store DSpark draft KV in FP8 / [AgentX] MiniMax-M3.1-Flash GB300 SGLang：DSpark draft KV 改用 FP8 - #4600
Conversation
Set speculative-draft-kv-cache-dtype fp8_e4m3, matching the vLLM DSpark drafter, so the bf16 draft KV no longer takes HBM and HiCache host pool space from target KV. The whole curve reruns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zijiexia
added
the
full-sweep-fail-fast
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # FP8 draft KV, as vLLM's DSpark drafter uses; bf16 costs ~10 GB of HBM per rank. | ||
| speculative-draft-kv-cache-dtype: fp8_e4m3 |
There was a problem hiding this comment.
🔴 After merging, the GB300 MiniMax-M3.1-Flash SGLang AgentX curve runs its DSpark draft below shipped precision, which CONTRIBUTING.md#draft-model-precision forbids. speculative-draft-kv-cache-dtype: fp8_e4m3 targets only the draft. No target kv-cache-dtype is set, so target KV stays at the bf16 default. Fix: drop the draft-only override in both recipes and their changelog line, or use one upstream-supported KV dtype for target and draft alike. Same pattern at 2 sites (gb300-fp4-mtp/agentic.yaml:37, gb300-fp4-mtp/agentic-tp2.yaml:37).
Why this was flagged
The base agg role args of agentic.yaml:37 and agentic-tp2.yaml:37 now set speculative-draft-kv-cache-dtype: fp8_e4m3. Neither recipe sets a target kv-cache-dtype. The PR's own changelog entry (perf-changelog.yaml, new block) says this replaces "the bf16 default", so the effective draft KV precision drops below the pinned image's default for this checkpoint. On the base branch the draft KV runs at the shipped bf16 default and is compliant. CONTRIBUTING.md lists "Dtype or KV-cache dtype overrides aimed at the draft that lower its precision below the framework's default for that checkpoint" as forbidden. It says matching an upstream recipe (here, vLLM's DSpark drafter) and passing evals or synthetic AL do not exempt this. The allowed case is FP8 draft KV inherited from an FP8 target KV, which does not apply here. Every point except agentic-low-latency.yaml in config key minimaxm31flash-fp4-gb300-sglang-agentic-dspark would publish non-compliant throughput gained by cheapening the draft.
Verification: agentic.yaml:36-37 and agentic-tp2.yaml:36-37 add speculative-draft-kv-cache-dtype: fp8_e4m3 to the base agg args, and no SGLang minimaxm31flash recipe sets kv-cache-dtype. So the effective draft KV precision drops from the pinned image's default (bf16) to FP8, which CONTRIBUTING.md lines 86-87 forbid. On the base branch the draft runs at the default bf16 KV and is compliant.
zijiexia
removed
the
full-sweep-fail-fast
zijiexia
added
the
full-sweep-fail-fast
There was a problem hiding this comment.
Code review found no new issues
No new issues were found in this update; 1 finding from earlier reviews is still open above.
Still open from earlier reviews (1):
- 🔴
inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/minimaxm31flash/sglang/gb300-fp4-mtp/agentic.yaml:37—After merging, the GB300 MiniMax-M3.1-Flash SGLang AgentX curve runs its DSpark draft below shipped precision, which CO…
If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#47027 has been merged. This PR's AgentX config ( This PR has no authorized |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40634232927 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40634232927 |
The SGLang DSpark drafter stores its KV cache in bf16 by default. On B300 TP2 that took 21 GB of HBM and 71 GB of the 370 GB HiCache host pool per rank, leaving about 10% less GPU KV than vLLM, whose DSpark drafter uses FP8 KV. At high concurrency the smaller GPU cache lowered the GPU prefix hit rate and cost throughput against vLLM.
This PR sets
speculative-draft-kv-cache-dtype: fp8_e4m3in the base args of the GB300 MiniMax-M3.1-Flash SGLang AgentX recipe, so the freed HBM and host memory go to target KV. The batch-size-1 MonoKernel Low-Latency point (agentic-low-latency.yaml) keeps bf16 draft KV, since it runs a separate kernel path and has no KV capacity pressure. The changelog entry is not append-only, so the whole curve reruns on the new recipe. The same change for B300 is in #4524.Risk: FP8 draft KV can lower real DSpark acceptance. Throughput runs use the golden synthetic AL, so the sweep's eval is what checks this.
中文
SGLang 的 DSpark drafter 默认用 bf16 存 KV cache。在 B300 TP2 上，它每个 rank 占用 21 GB HBM 和 370 GB HiCache host 池中的 71 GB，GPU KV 容量比 vLLM 少约 10%（vLLM 的 DSpark drafter 用 FP8 KV）。高并发时 GPU cache 更小，GPU prefix 命中率下降，吞吐落后于 vLLM。
本 PR 在 GB300 MiniMax-M3.1-Flash SGLang AgentX recipe 的 base args 中设置
speculative-draft-kv-cache-dtype: fp8_e4m3，把省下的 HBM 和 host 内存留给 target KV。batch size 1 的 MonoKernel 低延迟点（agentic-low-latency.yaml）保持 bf16 draft KV：它走独立的 kernel 路径，也没有 KV 容量压力。changelog 条目不是 append-only，整条曲线会用新 recipe 重跑。B300 的同样改动在 #4524。风险：FP8 draft KV 可能降低真实的 DSpark 接受率。吞吐测试使用 golden synthetic AL，因此需要由 sweep 中的 eval 来验证。
AI model disclosure
Prepared with Claude Code using
claude-opus-5-5. Reviewed by the human author before submission.🤖 Generated with Claude Code