Repository navigation
Conversation
Add kimik3-fp4-mi355x-atom-disagg-agentic: six Kimi-K3 MXFP4 AgentX points on MI355X with ATOM prefill/decode behind AToMesh and Mooncake P->D transfer. 1P1D TP8 DSpark7 c1, 1P2D TP8+DCP8 DSpark7 c16, 1P1D TP8+DCP8 DSpark4 c48/c72/c88 and 1P2D TP8+DCP8 DSpark4 c48. The DCP8 prefill keeps paged KV and KDA state images in an LMCache CPU tier; the DSpark draft stays BF16. 新增 kimik3-fp4-mi355x-atom-disagg-agentic：MI355X 上 6 个 Kimi-K3 MXFP4 AgentX 测试点，ATOM 预填充/解码分离，经 AToMesh 路由，Mooncake 负责 P->D 传输。包括 1P1D TP8 DSpark7 c1、1P2D TP8+DCP8 DSpark7 c16、1P1D TP8+DCP8 DSpark4 c48/c72/c88 和 1P2D TP8+DCP8 DSpark4 c48。DCP8 预填充 把分页 KV 和 KDA 状态镜像放在 LMCache CPU 层；DSpark 草稿模型保持 BF16。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
为 Kimi-K3 ATOM P/D 条目填入 #4311 链接。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| MC_GID_INDEX: '@fabric.rdma-gid-index' | ||
| NCCL_IB_DISABLE: '1' |
There was a problem hiding this comment.
🔴 All six points may see Mooncake prefill-to-decode KV transfers fail or stall if mi355x-amds rails are isolated. The ATOM disagg recipe already on this runner relies on ATOM_MOONCAKE_MATCHED_RAILS: auto. The shared &env block sets MC_GID_INDEX and NCCL_IB_DISABLE but leaves matched rails out. Without it, Mooncake falls back to dest-device affinity, which the sibling says rail isolation breaks. Fix: add ATOM_MOONCAKE_MATCHED_RAILS: auto to the &env anchor so prefill and decode both get it, or note why Kimi-K3 does not need it.
Why this was flagged
The condition is that mi355x-amds is rail-isolated, as the sibling recipe on the same runner states. Its comment at dsv4/atom/mi355x-fp4/agentx/disagg-lmcache-dspark.yaml:33-35 reads "Rail-isolated RDMA: matched rails replace dest-device affinity." and it sets ATOM_MOONCAKE_MATCHED_RAILS: auto. Both recipes use runner cluster:mi355x-amds, framework: atom-disagg and kv-p2p-transfer: mooncake. The new &env anchor (disagg-variants.yaml:40-63) copies that recipe's MC_GID_INDEX: '@ fabric.rdma-gid-index' and NCCL_IB_DISABLE: '1' lines but drops the matched-rails line. Both roles use this anchor, since decode takes env: *env. No override adds the setting back. The runners.yaml fabric entry for mi355x-amds (lines 810-814) has no rail flag that would supply it. So a prefill rank may post RDMA writes from a NIC that cannot reach the decode NIC. KV handoffs on every point would then error or hang, and the sweep would fail. The PR description does not mention this setting.
Verification: The new shared env: &env anchor (kimik3/atom/mi355x-fp4/agentx/disagg-variants.yaml lines 40-63) has no ATOM_MOONCAKE_MATCHED_RAILS. Decode uses env: *env (line 88), so both roles lack the setting. The sibling recipe is dsv4/atom/mi355x-fp4/agentx/disagg-lmcache-dspark.yaml. The new recipe kept lines 34 and 36 of its lines 33-36 and dropped line 35, ATOM_MOONCAKE_MATCHED_RAILS: auto.
Jasen2201
added
the
full-sweep-fail-fast
Keep the Kimi-K3 ATOM P/D changelog entry at the physical end after main's new entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jasen2201
added
full-sweep-enabled
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=39261889638 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=39261889638 |
Keep the Kimi-K3 ATOM P/D changelog entry at the physical end after #4295. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ugepages On mi355x-amds a 4 KiB LMCache pool registered with hipHostRegister stays movable. Each page the kernel migrates (compaction, reclaim) stalls the rank's GPU queues until the driver re-validates the pool, and GLM-5.2 ATOM prefills with ~1 TiB pools hung in the post-pool barrier on 3 of 14 timed starts there. The K3 DCP8 prefill pins 8 x 224 GiB (8 x 128 GiB at c16). Set LMCACHE_LOCAL_CPU_USE_HUGEPAGES on the prefill so LMCache maps each pool with MAP_HUGETLB 2 MiB pages bound to its NUMA node. host_setup reserves the pages on the head node (the prefill's) through the cluster's hugepage_sanity.sh, checks each NUMA node's share and that 400 GiB of other memory stays free for weight loading, and teardown releases them. The c1 point has no LMCache and is unchanged. 把 Kimi-K3 ATOM prefill 的 LMCache 池改用 2 MiB 大页，host_setup 在 prefill 节点预留、teardown 释放，避免 4 KiB 池页面迁移导致启动 barrier 卡死。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jasen2201
added
non-canary-full-sweep-enabled
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
before sweeping, u need to fix the mooncake kvcache counters for atomesh. chun and seungrok are working on a code change to fix it iirc. tho the MVP code change needs some serious changes to make it high quality
functionstackx
removed
the
non-canary-full-sweep-enabled
|
oh nvm, this is Mooncake for RDMA & LMCache for kvcache, chun & @seungrokj does lmcache atomesh with mooncake rdma have this issue or is it mooncake store that has this issue? |
Summary
Adds
kimik3-fp4-mi355x-atom-disagg-agentic: six Kimi-K3 MXFP4 AgentX points on MI355X with ATOM prefill/decode disaggregation behind AToMesh and Mooncake RDMA from prefill to decode. The existing ATOM entry,kimik3-fp4-mi355x-atom-agentic-mtp, serves prefill and decode on one node; these points split them.lmcache_offloadbeside Mooncake, so the paged KV and the KDA state images share one LMCache CPU pool. On the DSpark4 points,OFFLOAD_STATE_CPU_SIZE=128caps the state images within the 224 GB pool andATOM_STATE_CHECKPOINT_DEMAND=1adds checkpoints at observed shared prefixes; the fixed-interval grid stays off.dram-utilization0.598 and 0.343 budget 1793 GB and 1028 GB for the eight prefill ranks. Decode runs no CPU tier. ReplaySSM is on for the DCP8 points.Inferact/Kimi-K3-DSparkis served as it ships, in BF16: theonline_quant_configexclusions cover every draft layer (layers.*andcontext_proj), as in the single-node ATOM recipe. The draft is read from/it-share/data/Inferact-Kimi-K3-DSpark, the copy the vLLM P/D recipe mounts. The srt driver supplies the golden AL.policy: random, so a 1P2D request picks a decode at random.rocm/atom-dev:nightly_202610190554, pinned by digest (ROCm/ATOM daa4f1bb8).Risks.
mi355x-amds(3 of 8 starts). These prefills pin up to 1.75 TiB.Reference: ATOM CI. The same topologies, server flags and golden ALs on ROCm/ATOM's MI355X CI (a different cluster), with one difference: the CI quantizes the DSpark draft to FP8, which this recipe does not. Expect these points to land somewhat lower. Total tok/s per GPU (input incl. cached, plus output), p90 TPOT:
AI model disclosure
claude-opus-5-5(Claude Opus 5.5), running in Claude Code.infx.srt_slurm.workload, loaded it with srtctl and checked the rendered ATOM command lines and golden AL against the CI runs; checked draft-layer coverage against the CI's online-quantization records.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
概要
新增
kimik3-fp4-mi355x-atom-disagg-agentic：MI355X 上 6 个 Kimi-K3 MXFP4 AgentX 测试点，采用 ATOM 预填充/解码分离，经 AToMesh 路由，Mooncake RDMA 负责预填充到解码的传输。现有的 ATOM 条目kimik3-fp4-mi355x-atom-agentic-mtp在单节点上同时做预填充和解码；这些测试点把两者拆开。lmcache_offload，分页 KV 和 KDA 状态镜像共用一个 LMCache CPU 池。DSpark4 测试点用OFFLOAD_STATE_CPU_SIZE=128把状态镜像限制在 224 GB 池内的 128 GB，并用ATOM_STATE_CHECKPOINT_DEMAND=1在观察到的共享前缀处补充 checkpoint；固定间隔网格保持关闭。dram-utilization取 0.598 和 0.343，为 8 个预填充 rank 分别预算 1793 GB 和 1028 GB。解码不使用 CPU 层。DCP8 测试点开启 ReplaySSM。Inferact/Kimi-K3-DSpark按发布时的 BF16 精度提供服务：online_quant_config的排除列表覆盖全部草稿层（layers.*和context_proj），与单节点 ATOM 配方一致。草稿从/it-share/data/Inferact-Kimi-K3-DSpark读取，即 vLLM P/D 配方挂载的同一份拷贝。golden AL 由 srt 驱动提供。policy: random，因此 1P2D 的请求随机选择一个解码节点。rocm/atom-dev:nightly_202610190554，按 digest 固定（ROCm/ATOM daa4f1bb8）。风险。
mi355x-amds上发现，LMCache 预填充注册 1 TiB pinned 池后可能在启动 barrier 卡住（8 次启动中 3 次）。这些预填充最多 pin 1.75 TiB。参考：ATOM CI。 在 ROCm/ATOM 的 MI355X CI（另一套集群）上运行相同拓扑、服务参数和 golden AL，唯一区别是 CI 把 DSpark 草稿量化为 FP8，而本配方不量化。因此这些测试点的结果预计会略低。吞吐为每 GPU 总 token/s（输入含命中缓存的部分，加输出），延迟为 p90 TPOT；数据和运行链接见上表。
AI 模型使用说明
claude-opus-5-5（Claude Opus 5.5），运行于 Claude Code。infx.srt_slurm.workload绑定每个变体并用 srtctl 加载，核对生成的 ATOM 命令行和 golden AL 与 CI 运行一致；依据 CI 的在线量化记录核对草稿层的排除覆盖。