Repository navigation
Conversation
…cy points - conc 1: 1P1D TP4/TP4 with six DSpark draft tokens. - conc 16: 1P1D TP4 prefill + TP8 decode with six DSpark draft tokens; the TP4 prefill adds the lmcache_offload host KV tier (dram-utilization 0.50). - gpu-memory-utilization, image and conc 64/128/256 unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Move conc 128 and 256 to dsv4-fp4-mi355x-atom-disagg-agentic-mega-mp-dspark on nightly_202610141644 and add conc 512. - Prefill: DP attention + EP8 + Mega MoE + EPLB, no TBO, KV offload to an LMCache MP server on the prefill node (1000 GB lazy L1). - Decode: DP attention without offload; dp_sticky routing on both roles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yhl-amd
changed the title
yhl-amd
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # total-cpu-dram-gb 1499 (dram-utilization 0.5) over the four prefill ranks. | ||
| lmcache.max_local_cpu_size: 374 |
There was a problem hiding this comment.
🔴 The conc-16 point records a DRAM budget about half of what its LMCache tier really uses, so results understate host memory by ~750 GB. A TP4 prefill at dram-utilization 0.50 gets only 4/8 of the node share, about 749 GB (generate.py:493-507). The recipe still sizes lmcache.max_local_cpu_size: 374 per rank against the full-node 1499 GB, so it allocates ~1496 GB. Fix: make the budget and the allocation agree. Either set dram-utilization: 1.0 for this TP4 point, which gives a 1499 GB share, or cut the per-rank size to about 187 GB. Then correct the comment.
Why this was flagged
The trigger is the c16 point in amd-master.yaml under dram-utilization: 0.50, with prefill: tp: 4 and kv-offloading: dram. agentic_dram_offload_gb (infx/matrix/generate.py:493-507) computes the prefill share as available * 0.50 * 4 / 8. That is about 749 GB, written to TOTAL_CPU_DRAM_GB (infx/srt_slurm/generate.py:59). The recipe comment at disagg-lmcache-dspark.yaml:134 assumes the whole 1499 GB goes to the four prefill ranks. It sets lmcache.max_local_cpu_size: 374 per rank, so 4 x 374 = 1496 GB actually lands on host DRAM. The published result reports allocated_cpu_dram_gb (infx/results/agentic/init.py:205) at roughly half the real usage. On the base branch the TP8 prefill used 187 x 8 = 1496 GB against a 1499 GB TP8 share, so the two matched. A repo precedent (perf-changelog line 11040) shows a TP4 share of 1499 GB only at dram-utilization 1.0. No validation compares the recipe's LMCache size with total-cpu-dram-gb.
Verification: Trigger: every run of the new c16 point in configs/amd-master.yaml. The new TP4 prefill gets 3000 * 0.5 * 4/8, which comes to about 749 GB. That is 4 x 374 = 1496 GB of LMCache CPU tier, about twice the 749 GB the matrix assigns and records. The only recipe-versus-matrix TOTAL_CPU_DRAM_GB check is in infx/srt_slurm/single_node.py:145, and it covers single-node recipes only.
| # Concurrency 128, 256 and 512 run on their own master-config key: prefill is | ||
| # DP attention with EP8, the Mega MoE backend and EPLB, without TBO, and | ||
| # offloads KV to one LMCache MP server on the prefill node (lazy 1000 GB L1 | ||
| # shared by the eight prefill ranks); decode is DP attention without offload. | ||
| # Both roles route dp_sticky. |
There was a problem hiding this comment.
🟡 (optional) Maintainers get new header comments that restate topology, which the repo's mandatory comment rules forbid. This class is comments describing a topology, override, section or config key. AGENTS.md "Comments (mandatory, every file)" bans header comments of that kind. Fix: delete these headers and keep only short per-line rationale such as line 236's port note, which covers the 5 sites listed. Same pattern at 5 sites (disagg-lmcache-dspark.yaml:87-89, disagg-lmcache-dspark.yaml:178-182, amd-master.yaml:1328-1330, amd-master.yaml:1361-1363, amd-master.yaml:1382-1386). [also at: inferencex-e2e/configs/amd-master.yaml:1386 - Maintainers get new header comments that restate topology, which AGENTS.md "Comments (mandatory, every file)" forbids.]
Why this was flagged
AGENTS.md:56 says "No header comments describing a file, topology, config, recipe, override, or section." The PR adds such comments above override_ep8_c128_mega_mp (recipe 178-182: "Concurrency 128, 256 and 512 run on their own master-config key: prefill is DP attention with EP8...") and above override_tp4_c1 (recipe 87-89). In amd-master.yaml it adds one above the new key dsv4-fp4-mi355x-atom-disagg-agentic-mega-mp-dspark (1382-1386) and rewrites two scenario-block headers (1328-1330, 1361-1363). These comments only restate the YAML and the topology. The repository keeps that record in perf-changelog.yaml and git. They will drift from the config on the next retune. The base branch's block comments were already of this kind, but this diff adds new ones and expands the existing ones. That breaks a rule CLAUDE.md applies to every file.
Verification: AGENTS.md:54-58 ("Comments (mandatory, every file)") says: "No header comments describing a file, topology, config, recipe, override, or section", and this PR does not change AGENTS.md, so the rule applies. The PR newly writes or rewrites each of the five headers, so they are new violations. Nothing breaks at runtime.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40623037268 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40623037268 |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
…num-seqs, meta image - Prefill keeps only prompt-end state checkpoints (interval -1, demand off) and rebuilds other hits by bounded state replay. - max-num-seqs sized per DP rank on both roles. - rocm/atom-dev:nightly_202610271520_meta for the mega-mp key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yhl-amd
removed
the
full-sweep-enabled
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yhl-amd
added
the
full-sweep-fail-fast
# Conflicts: # inferencex-e2e/perf-changelog.yaml
yhl-amd
added
full-sweep-enabled
# Conflicts: # inferencex-e2e/perf-changelog.yaml
Summary
Retopologize the DeepSeek-V4-Pro-0813 ATOM PD-disaggregated AgentX sweep on MI355X (
dsv4-fp4-mi355x-atom-disagg-agentic-lmcache-dspark) at low and high concurrency.Low concurrency (same key, same image):
num-speculative-tokens: 6).lmcache_offloadhost KV tier (same connector settings as before, 374 GB per rank at dram-utilization 0.50), so this point moves to thedram-utilization: 0.50block withkv-offloading: dram.High concurrency (new key
dsv4-fp4-mi355x-atom-disagg-agentic-mega-mp-dspark,rocm/atom-dev:nightly_202610141644):high-throughputall2all) and EPLB (window 100, interval 200, at most 4 rebalances); TBO off.lmcache-serverservice,lmcache_mpconnector withtp_rank_collapse, 1000 GB lazy L1, 900 s read TTL, LRU) replaces the in-processlmcache_offloadtier. dram-utilization 0.34 records the 1000 GB.dp_stickyon both roles.Unchanged:
gpu-memory-utilization(0.9), model, router and the conc 64 point.Files
inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/atom/mi355x-fp4/agentx/disagg-lmcache-dspark.yamlinferencex-e2e/configs/amd-master.yamlinferencex-e2e/perf-changelog.yamlValidation
infx.matrix.generate test-configemits the six expected points (c1 TP4/TP4, c16 TP4/TP8 dram, c64 unchanged, c128/256/512 EP8 prefill + DP decode on the new key)cluster:mi355x-amds中文
调整 DeepSeek-V4-Pro-0813 在 MI355X 上 ATOM PD 分离 AgentX 扫描（
dsv4-fp4-mi355x-atom-disagg-agentic-lmcache-dspark）的低并发和高并发点。低并发（原 key，镜像不变）：
num-speculative-tokens: 6）。lmcache_offload主机 KV 层（connector 设置不变，dram-utilization 0.50 下每个 rank 374 GB），因此该点移到dram-utilization: 0.50的配置块，kv-offloading: dram。高并发（新 key
dsv4-fp4-mi355x-atom-disagg-agentic-mega-mp-dspark，镜像rocm/atom-dev:nightly_202610141644）：high-throughputall2all）+ EPLB（窗口 100、间隔 200、最多 4 次），关闭 TBO。lmcache_offload改成 prefill 节点上的 LMCache MP server（lmcache-serverservice，lmcache_mpconnector，tp_rank_collapse，1000 GB lazy L1，读 TTL 900 s，LRU）。dram-utilization 0.34 对应这 1000 GB。dp_sticky路由。不变：
gpu-memory-utilization（0.9）、模型、router，以及 conc 64。🤖 Generated with Claude Code