Repository navigation
[NVIDIA][AgentX] Move GB300 DSV4.1-Flash Dynamo + SGLang to the bonus-fixed nightlies with prefill-HiCache disagg points and TP4 c1 megakernel / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang 升级到含 bonus 修复的 nightly，分离式改用 prefill HiCache，TP4 c1 启用 megakernel - #4595
weireweire wants to merge 12 commits into
Conversation
…HiCache Root cause: at c7480 the Mooncake external linker reads L3 hits layer by layer on the forward path; after the AgentX turn burst, prefill stalls on those reads (about 40 % of wall time), device prefixes age out, and the deployment does not recover. Fix: a new config key on nightly-dev-cu13-20261027-5884e4fe (Dynamo 1.7.0.dev20261026) with 9 and 10 DEP2 prefill workers and one DEP16 decode. Prefill keeps evicted prefixes in a local host-DRAM HiCache tier (ratio 1.8, write-back, direct I/O); decode keeps SGLang's default polling interval of 1. Dynamo 1.7 backend admission limits are raised so turns queue in the scheduler, and NVIDIA/srt-slurm#629 is backported so the readiness wait asks /health?instances=true when /health is status-only. The agg and dspark-disagg keys are listed so they rerun in the same sweep. Validation: on GB300 with the same recipe and AgentX client, 10 x DEP2 batch 128 gave 924,652 tok/s/GPU at p90 117.2 tok/s/user over 3600 s.
…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - spec-decoding: mtp | ||
| conc-list: [7480] | ||
| srt-recipe: disagg-dep16-hicache-variants.yaml:override_10p1d_dep2_dep16_b160_c7480 | ||
| kv-offloading: dram | ||
| kv-offload-backend: { name: hicache } | ||
| prefill: | ||
| num-worker: 10 | ||
| tp: 2 | ||
| ep: 2 | ||
| dp-attn: true | ||
| decode: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| ep: 16 | ||
| dp-attn: true |
There was a problem hiding this comment.
🔴 If either 10-prefill c7480 job of a sweep is rerun, the other 10-prefill job's result is dropped. Klaud's coverage check then fails the sweep. Rows override_10p1d_dep2_dep16_c7480 and override_10p1d_dep2_dep16_b160_c7480 differ only by recipe. They share every field of agentic_key (infx/results/artifacts.py:123), so superseded_attempt_files treats one as a retry of the other. Fix: give each row its own result identity, e.g. key superseded_attempt_files on recipe_fingerprint plus conc, the identity check_coverage already uses.
Why this was flagged
The two rows at nvidia-master.yaml:9752-9781 have the same prefill (10 x tp2 ep2 dpa), decode (1 x tp16 ep16 dpa), conc 7480, spec mtp and kv-offload-backend {name: hicache}. agentic_key (infx/results/artifacts.py:123-168) is built only from those fields, so both point results get the same key. superseded_attempt_files (artifacts.py:197-215) drops any file whose rows all have an older run_attempt than the latest attempt for that key. Klaud recovery POSTs rerun-failed-jobs (infx/klaud/lifecycle.py:705) after a failure or preemption, and only the failed job reruns. Say the b160 job reruns as attempt 2 while the b128 point finished in attempt 1. agentic_point_files (artifacts.py:188-194) then discards the b128 file. check_coverage (infx/klaud/validation.py:307-320) expects (recipe_fingerprint, 7480, image) for both rows and raises "Full-sweep result coverage or consistency failed". RESULT_FILENAME adds the recipe fingerprint, which keeps artifact names distinct, but the superseding logic never reads that fingerprint.
Verification: The rows override_10p1d_dep2_dep16_c7480 and override_10p1d_dep2_dep16_b160_c7480 in configs/nvidia-master.yaml differ only in srt-recipe. agentic_key (infx/results/artifacts.py:123-168) leaves out recipe_fingerprint. On the base branch these rows do not exist, so the failure is new.
|
Thanks for the contribution!
中文感谢你的贡献！
|
…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml
…ightly and drop TP2/EP2 c16 nightly-dev-cu13-20261028-586355a8 includes sgl-project/sglang#46404. Raise the Dynamo 1.7 backend admission limits as on the disagg rows. TP2/EP2 c16 sits far below the line from TP4 c16 to the disaggregated points, so it is removed with its override.
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#47027 has been merged. This PR's AgentX configs ( This PR has no authorized |
… 20261028 nightly and retire the Mooncake-linker disagg key The hicache key now runs on nightly-dev-cu13-20261028-586355a8, which includes sgl-project/sglang#46404. On that image the KV pool sizes at 890 B/token, so hicache-ratio drops to 1.2 (1.8 exceeds host memory) and decode mem-fraction-static to 0.75 (0.85 runs out of memory at graph capture). Points: 5P c2884 (bs40), 8P c4096 (bs64) and 10P c7480 (bs160). The 9P and 10P bs128 c7480 points are dropped; bs160 is higher at the same interactivity. The dspark-disagg key on 65633129 and its recipe are retired.
At decode batch 160, 9 x DEP2 reaches the same p90 interactivity as 10 x DEP2 with two fewer GPUs, so its throughput per GPU is higher.
…recipe in the changelog
weireweire
changed the title
…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml
…ightly and use the megakernel runtime-optimization switch at TP4 c1 nightly-dev-cu13-20261029-c10e0e17 bundles sgl-nvidia-mega 54b428921749 (megakernel API 2, sgl-project/sglang#47145), so TP4 c1 starts with --enable-nvidia-megakernel again. SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION (sgl-project/sglang#47105) replaces the 16 runtime-optimization variables the recipe set one by one.
weireweire
added
the
full-sweep-fail-fast
weireweire
changed the title
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#47214 has been merged. This PR's AgentX configs ( This PR has no authorized |
Move the GB300 DeepSeek-V4.1-Flash Dynamo + SGLang AgentX curve to nightlies that include sgl-project/sglang#46404: the disaggregated points to digest-pinned
lmsysorg/sglang:nightly-dev-cu13-20261028-586355a8and the aggregated points tonightly-dev-cu13-20261029-c10e0e17(below). The 586355a8 image bundles Dynamo1.7.0.dev20261027and SGLang0.0.0.dev1+g586355a83. #46404 draws the DSpark simulated-acceptance bonus at the simulated row instead of the real mismatch row. No SGLang engine patch is added.The disaggregated points move to a new key,
dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache, whose prefill uses a local host-DRAM HiCache tier instead of the Mooncake external linker. The existingdsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disaggkey (nightly-dev-cu13-20261025-65633129, which predates #46404) and itsdisagg-variants.yamlrecipe are retired.Removed points. The retired key's published points were measured on
65633129, before #46404. Each one is covered by this PR's points on the same Dynamo + SGLang curve (tput_per_gpu@mean_intvty; head values are our 1200 s runs, which understate the sweep):c1228 is not carried over. It also sat under the published curve's own line from agg TP2/EP2 c16 (58,177 @ 401.6) to c2884 (about 442,000 at 247.5).
Why this recipe:
hicache-ratio 1.2, write-back, direct I/O,mem-fraction-static 0.85, 64 running requests per DP rank) instead of the Mooncake external linker.hicache-ratio 1.8, the host mirror of that larger pool exceeds node memory.mem-fraction-static 0.85, decode runs out of memory during CUDA graph capture; 0.75 leaves the headroom capture needs.disaggregation-decode-polling-intervalof 1; the retired key set 8. At 8, each admission batch stalled the decode loop about 2 ms.runners/srt-slurm/patches/629-dynamo-health-instances.patchbackports fix(dynamo): fall back to /health?instances=true when /health is status-only NVIDIA/srt-slurm#629. Since fix(http): make /health status-only by default; list instances only on request ai-dynamo/dynamo#16454, plain/healthomits theinstanceslist that the readiness wait counts. The patched probe asks/health?instances=trueonly when/healthanswers status-only, so older Dynamo frontends get the same single request.Aggregated row:
dsv41flash-fp4-gb300-dynamo-sglang-agentic-aggmoves to digest-pinnednightly-dev-cu13-20261029-c10e0e17(Dynamo1.7.0.dev20261028, SGLang0.0.0.dev1+gc10e0e177).54b428921749([Fix] docker: update NVIDIA megakernel pin to tested commit sgl-project/sglang#47145), which implements megakernel API 2, so TP4 c1 runs with--enable-nvidia-megakernelagain. Target and draft megakernels are active on all four TP ranks.SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1([Feat] [NVIDIA] runtime: Add a master switch for megakernel runtime optimizations sgl-project/sglang#47105). The switch also turns on the Qwen3.5/EAGLE runtime group, most of which is inactive on this path.SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD, the simulation settings,SGLANG_NVIDIA_MEGAKERNEL_SLOTS, the Engram host table and the NCCL settings stay explicit.Throughput gain at the same p90 interactivity (tok/s/GPU). Our points were measured on our GB300 cluster with the bound recipes of this PR and the AgentX client, not in this PR's sweep. The vLLM column linearly interpolates the currently published GB300 DeepSeek-V4.1-Flash Dynamo + vLLM and vLLM points on their Pareto frontier at our p90 interactivity. A blank cell means no gain.
At p90 107.1, the vLLM frontier lies between 1,011,514 @ 75.5 and 993,521 @ 145.6, which gives 1,003,413. The 1200 s values understate a 3600 s run by a factor of about 1.17-1.20 at these concurrencies, but even after that, c2884 and c4096 stay below vLLM's c10888 point (954,146 @ 196.5).
Validation details
All runs used the image above through srt-slurm on GB300 with the AgentX client (
reasoning_efforthigh, temperature 1) and the golden acceptance length 3.51. Each run's resolved recipe matches the recipe thatinfx generatebinds for this PR, apart from cluster NIC names, scheduler directives and the unused Mooncake services. The 1200 s runs had 0 request errors; the 3600 s run had 3 empty responses out of about 1.05M requests.中文
将 GB300 DeepSeek-V4.1-Flash Dynamo + SGLang AgentX 曲线换到包含 sgl-project/sglang#46404（DSpark 模拟接受的 bonus 改在模拟位置采样）的 nightly：分离式点用固定 digest 的
lmsysorg/sglang:nightly-dev-cu13-20261028-586355a8（Dynamo1.7.0.dev20261027、SGLang0.0.0.dev1+g586355a83），聚合式点用nightly-dev-cu13-20261029-c10e0e17（见下）。不添加 SGLang 引擎补丁。dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache，DEP2 prefill 加一个 DEP16 decode。dspark-disaggkey（65633129，早于 #46404）及其disagg-variants.yamlrecipe 退役。它已发布的三个点都被本 PR 的点覆盖（按tput_per_gpu与mean_intvty）：c4096 和 c2884 分别被 8 × DEP2 和 5 × DEP2 支配；c1228（367,046 @ 247.5）低于 agg TP4 c16 到 5 × DEP2 c2884 的连线（在 247.5 处约 573,000），因此不保留。hicache-ratio 1.2、write-back、direct I/O、mem-fraction-static 0.85、每个 DP rank 64 个运行请求）替代 Mooncake external linker。hicache-ratio 1.8时主机镜像超出节点内存，改为 1.2。mem-fraction-static 0.85时，CUDA graph capture 阶段 OOM，改为 0.75。629-dynamo-health-instances.patch，回移 fix(dynamo): fall back to /health?instances=true when /health is status-only NVIDIA/srt-slurm#629：/health只返回状态时才请求/health?instances=true。nightly-dev-cu13-20261029-c10e0e17（Dynamo1.7.0.dev20261028）。该镜像自带 sgl-nvidia-mega54b428921749（megakernel API 2，[Fix] docker: update NVIDIA megakernel pin to tested commit sgl-project/sglang#47145），TP4 c1 重新启用--enable-nvidia-megakernel，4 个 TP rank 上 target 和 draft megakernel 均生效。TP4 c1 原先逐个设置的 16 个 megakernel 运行时优化变量改为SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1（[Feat] [NVIDIA] runtime: Add a master switch for megakernel runtime optimizations sgl-project/sglang#47105）；该开关同时打开 Qwen3.5/EAGLE 那组优化，在此路径上大多不生效。SIM_DRAW_AHEAD、模拟设置、megakernel slots、Engram 和 NCCL 设置仍显式指定。放开 admission 上限；删除 TP2/EP2 c16，TP4 c4/c16 配置不变。同交互性（p90 interactivity）下相对 vLLM 的吞吐提升： 数据在我们的 GB300 集群上用本 PR 的配方测得，不是本 PR 的 sweep；在这些并发下，3600 s 结果约为 1200 s 的 1.17-1.20 倍。vLLM 用已发布的 Dynamo + vLLM 与 vLLM 点的 Pareto 前沿，在相同交互性处线性插值。
c2884 和 c4096 即使换算到 3600 s，也低于 vLLM 的 c10888 点（954,146 @ 196.5）。
AI model disclosure
aws/anthropic/eccn-claude-opus-5-5) investigated the bottleneck, ran the cluster experiments through delegated agents on the same model, wrote the recipe, master config, changelog entry and this description, and ran an independent review of the diff through a separate delegated agent on the same model.