[Klaud Cold] Update DeepSeek-V4-Pro FP4 MI355X ATOM AgentX DSpark image to nightly_202610261558 / [Klaud Cold] 将 DeepSeek-V4-Pro FP4 MI355X ATOM AgentX DSpark 镜像更新至 nightly_202610261558 by Klaud-Cold · Pull Request #4577 · SemiAnalysisAI/InferenceX

Klaud-Cold · 2026-10-27T12:23:01Z

Goal: Update ATOM image from rocm/atom-dev:nightly_202610141644 to rocm/atom-dev:nightly_202610261558.
Baseline: 2026-10-15 · rocm/atom-dev:nightly_202610141644
Mean latency · Sources: API 1, API 2, API 3, API 4

Point	Total tok/s/GPU ↑	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
AgentX c1 TP8 EP1 75f53e	2,171.83	16.32	866.73	4.12
AgentX c2 TP8 EP1 291413	2,341.01	19.39	768.66	4.28
AgentX c4 TP8 EP1 8095cb	3,370.15	24.79	759.46	4.53
AgentX c8 TP8 EP1 0e432a	6,212.64	46.67	737.31	5.29
AgentX c16 TP8 EP1 2470c9	11,762.87	79.84	745.71	7.63
AgentX c48 TP8 EP8 25a62b	24,735.84	169.77	9,583.28	9.68
AgentX c64 TP8 EP8 673532	32,033.31	248.01	9,147.42	11.97
AgentX c96 TP8 EP8 e825cf	42,959.86	359.08	9,109.92	16.05
AgentX c128 TP8 EP8 d3fb44	48,673.29	422.19	9,645.63	18.3
AgentX c256 TP8 EP8 d7adce	77,043.89	588.06	20,295.79	22.97
AgentX c384 TP8 EP8 09e4fe	88,430	690.54	20,269.24	42.38

Eval	Score ↑	Samples
gsm8k/em_strict · c16	97.57%	1,319
gsm8k/em_strict · c384	97.73%	1,319

Stop: Close this PR at any time to stop this candidate now; Klaud cancels its runs and deletes its branch.

中文

**目标：**将 ATOM 镜像从 rocm/atom-dev:nightly_202610141644 更新至 rocm/atom-dev:nightly_202610261558。
**基线：**2026-10-15 · rocm/atom-dev:nightly_202610141644
平均延迟 · 来源： API 1, API 2, API 3, API 4；数值及异常说明见上表。
**停止：**随时关闭此 PR 即可立即停止该候选：Klaud 会取消其运行并删除其分支。

…2610261558 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-27T12:23:09Z

Thanks for the contribution!

Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

**审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
**PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
**合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

Klaud-Cold · 2026-10-27T12:25:14Z

Initial attempt · Failed · Run 40503383390 / attempt 1 · 2026-10-27 13:36 UTC
rocm/atom-dev:nightly_202610261558@sha256:57477e7a4b5156fcc7b21bab082eba98577752465f58479ac2ead8514f882a61 · ccd4fcd15796 · Mean latency
Change: Measured ref e7060af: Pin the master image and atomesh frontend to rocm/atom-dev:nightly_202610261558 (sha256:57477e7a, ATOM 0.1.9.dev47+ge893780f4, was 0.1.8.dev55+g6e9b3e1c5); no recipe flags changed. Source diff ROCm/ATOM@6e9b3e1...e893780 keeps every used flag/env except ATOM_MEGA_MASK_PAD_ROWS (removed in ROCm/ATOM#2725, now a no-op); AITER/MORI build from HEAD so their exact commits are unverified.

Point	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
AgentX c1 TP8 EP1 75f53e	N/A	N/A	N/A
AgentX c16 TP8 EP1 2470c9	N/A	N/A	N/A
AgentX c48 TP8 EP8 25a62b	N/A	N/A	N/A
AgentX c256 TP8 EP8 d7adce	N/A	N/A	N/A
AgentX c384 TP8 EP8 09e4fe	N/A	N/A	N/A

Note: AgentX c1 TP8 EP1 75f53e, AgentX c256 TP8 EP8 d7adce: cancelled.
Note: All rows: request errors unavailable.
Note: All rows: Δ N/A: point unavailable.
Note: AgentX c16 TP8 EP1 2470c9, AgentX c384 TP8 EP8 09e4fe: unavailable.
Note: AgentX c48 TP8 EP8 25a62b: failed.

Eval	Score ↑	Samples
gsm8k/em_strict · c16	97.65% (+0.08 pp)	1,319 each
gsm8k/em_strict · c384	97.5% (-0.23 pp)	1,319 each

Next: Repair 1: disable EPLB on the DEP points that enable it, keeping every other flag, and rerun the smoke.

中文

初次尝试 · 失败 · Run 40503383390 / attempt 1 · 2026-10-27 13:36 UTC
rocm/atom-dev:nightly_202610261558@sha256:57477e7a4b5156fcc7b21bab082eba98577752465f58479ac2ead8514f882a61 · ccd4fcd15796 · 平均延迟
**变更：**测量提交 e7060af：将主配置镜像和 atomesh 前端固定到 rocm/atom-dev:nightly_202610261558（sha256:57477e7a，ATOM 0.1.9.dev47+ge893780f4，原为 0.1.8.dev55+g6e9b3e1c5）；未修改配方参数。源码对比 ROCm/ATOM@6e9b3e1...e893780 显示所用参数和环境变量均保留，仅 ATOM_MEGA_MASK_PAD_ROWS 在 ROCm/ATOM#2725 中移除（现已无作用）；AITER/MORI 从 HEAD 构建，确切提交无法验证。实测数值及异常说明见上表。
**下一步：**修复 1：在启用 EPLB 的 DEP 点上关闭 EPLB，其他参数不变，并重新运行 smoke。

Klaud-Cold · 2026-10-27T12:35:55Z

parked · Session 1 is waiting on 40503383390. Recovery reruns preempted runs and starts the next session once they end.

中文

parked · 第 1 个会话正在等待 40503383390。恢复流程会重跑被抢占的运行，并在运行结束后启动下一个会话。

Klaud-Cold · 2026-10-27T13:33:09Z

resumed · The session ended before finishing; session 2 continues it in a new run of its planner.

中文

resumed · 会话未完成即已结束；第 2 个会话将在其规划运行的新运行中继续。

The first EPLB rebalance hangs the DEP8 server on rocm/atom-dev:nightly_202610261558 (ROCm/ATOM#2725 routing rewrite). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Klaud-Cold · 2026-10-27T13:37:52Z

Repair 1/5 · Failed · Run 40513607186 / attempt 1 · 2026-10-27 18:39 UTC
rocm/atom-dev:nightly_202610261558@sha256:57477e7a4b5156fcc7b21bab082eba98577752465f58479ac2ead8514f882a61 · ccd4fcd15796 · Mean latency
Change: Measured ref 09dc099: remove enable-eplb, eplb-config and ATOM_EPLB_MAX_REBALANCES from the c48/c96/c128/c256/c384 DEP variants (the c64 variant already ran without EPLB); the first rebalance hung the DEP8 server under ROCm/ATOM#2725's routing rewrite, which lists EPLB and MegaMoE as untested end to end. Image, topology and all other flags are unchanged.

Point	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
AgentX c1 TP8 EP1 75f53e	16.42 (+0.6%)	662.75 (-23.5%)	4.18 (+1.5%)
AgentX c16 TP8 EP1 2470c9	N/A	N/A	N/A
AgentX c48 TP8 EP8 25a62b	178.32 (+5%)	3,125.64 (-67.4%)	14.74 (+52.3%)
AgentX c256 TP8 EP8 d7adce	568.44 (-3.3%)	21,315.27 (+5%)	24.25 (+5.6%)
AgentX c384 TP8 EP8 09e4fe	N/A	N/A	N/A

Note: AgentX c16 TP8 EP1 2470c9, AgentX c384 TP8 EP8 09e4fe: unavailable.
Note: AgentX c16 TP8 EP1 2470c9, AgentX c384 TP8 EP8 09e4fe: request errors unavailable.
Note: AgentX c16 TP8 EP1 2470c9, AgentX c384 TP8 EP8 09e4fe: Δ N/A: point unavailable.

Eval	Score ↑	Samples
gsm8k/em_strict · c16	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c384	97.42% (-0.30 pp)	1,319 each

Note: gsm8k/em_strict · c16: failed.
Note: gsm8k/em_strict · c16: Δ N/A: eval score unavailable.

Next: Infrastructure retry 1: rerun the failed c16 eval job on the same head, since this unexplained illegal memory access did not occur on the identical c16 config in the initial attempt.

中文

修复 1/5 · 失败 · Run 40513607186 / attempt 1 · 2026-10-27 18:39 UTC
rocm/atom-dev:nightly_202610261558@sha256:57477e7a4b5156fcc7b21bab082eba98577752465f58479ac2ead8514f882a61 · ccd4fcd15796 · 平均延迟
**变更：**测量提交 09dc099：从 c48/c96/c128/c256/c384 DEP 变体中移除 enable-eplb、eplb-config 和 ATOM_EPLB_MAX_REBALANCES（c64 变体原本就未启用 EPLB）；在 ROCm/ATOM#2725 的路由重写下，首次 rebalance 使 DEP8 服务挂起，该 PR 注明 EPLB 和 MegaMoE 尚未端到端测试。镜像、拓扑和其他参数均不变。实测数值及异常说明见上表。
**下一步：**基础设施重试 1：在同一提交上重跑失败的 c16 评测任务，因为相同的 c16 配置在初次尝试中未出现这一无法解释的非法内存访问。

Klaud-Cold · 2026-10-27T13:38:08Z

parked · Session 2 is waiting on 40513607186. Recovery reruns preempted runs and starts the next session once they end.

中文

parked · 第 2 个会话正在等待 40513607186。恢复流程会重跑被抢占的运行，并在运行结束后启动下一个会话。

Klaud-Cold · 2026-10-27T18:36:04Z

resumed · The session ended before finishing; session 3 continues it in a new run of its planner.

中文

resumed · 会话未完成即已结束；第 3 个会话将在其规划运行的新运行中继续。

Klaud-Cold · 2026-10-27T18:39:51Z

Infrastructure retry 1/2 · Passed · Run 40513607186 / attempt 2 · 2026-10-28 03:06 UTC
rocm/atom-dev:nightly_202610261558@sha256:57477e7a4b5156fcc7b21bab082eba98577752465f58479ac2ead8514f882a61 · ccd4fcd15796 · Mean latency
Change: Same head and recipe as repair 1 (measured ref 09dc099, EPLB off on DEP points); only the failed c16 gsm8k eval job is rerun, reusing the run's passed c1/c48/c256 benchmarks and c384 eval.

Point	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
AgentX c1 TP8 EP1 75f53e	16.42 (+0.6%)	662.75 (-23.5%)	4.18 (+1.5%)
AgentX c16 TP8 EP1 2470c9	N/A	N/A	N/A
AgentX c48 TP8 EP8 25a62b	178.32 (+5%)	3,125.64 (-67.4%)	14.74 (+52.3%)
AgentX c256 TP8 EP8 d7adce	568.44 (-3.3%)	21,315.27 (+5%)	24.25 (+5.6%)
AgentX c384 TP8 EP8 09e4fe	N/A	N/A	N/A

Note: AgentX c16 TP8 EP1 2470c9, AgentX c384 TP8 EP8 09e4fe: unavailable.
Note: AgentX c16 TP8 EP1 2470c9, AgentX c384 TP8 EP8 09e4fe: request errors unavailable.
Note: AgentX c16 TP8 EP1 2470c9, AgentX c384 TP8 EP8 09e4fe: Δ N/A: point unavailable.

Eval	Score ↑	Samples
gsm8k/em_strict · c16	97.8% (+0.23 pp)	1,319 each
gsm8k/em_strict · c384	97.42% (-0.30 pp)	1,319 each

Next: Smoke benchmarks and both selected evals passed: append the perf-changelog entry, merge main and dispatch the full final sweep.

中文

基础设施重试 1/2 · 已通过 · Run 40513607186 / attempt 2 · 2026-10-28 03:06 UTC
rocm/atom-dev:nightly_202610261558@sha256:57477e7a4b5156fcc7b21bab082eba98577752465f58479ac2ead8514f882a61 · ccd4fcd15796 · 平均延迟
**变更：**与修复 1 相同的提交和配方（测量提交 09dc099，DEP 点关闭 EPLB）；仅重跑失败的 c16 gsm8k 评测任务，复用该运行已通过的 c1/c48/c256 基准测试和 c384 评测。实测数值及异常说明见上表。
**下一步：**smoke 基准测试和两项所选评测均已通过：追加 perf-changelog 条目，合并 main 并发起完整的最终扫描。

Klaud-Cold · 2026-10-27T18:40:05Z

parked · Session 3 is waiting on 40513607186. Recovery reruns preempted runs and starts the next session once they end.

中文

parked · 第 3 个会话正在等待 40513607186。恢复流程会重跑被抢占的运行，并在运行结束后启动下一个会话。

Klaud-Cold · 2026-10-28T03:04:40Z

resumed · The session ended before finishing; session 4 continues it in a new run of its planner.

中文

resumed · 会话未完成即已结束；第 4 个会话将在其规划运行的新运行中继续。

…920b0a-6b1902ae19b3dc0a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Klaud-Cold · 2026-10-28T03:08:45Z

Final full sweep · Running · Run 40624895148 / attempt 1 · 2026-10-28 03:08 UTC
rocm/atom-dev:nightly_202610261558@sha256:57477e7a4b5156fcc7b21bab082eba98577752465f58479ac2ead8514f882a61 · 591909cd8a8c · Mean latency
Change: Final sweep at 591909c: same image and recipe as repair 1 (EPLB off on DEP points), with the perf-changelog entry appended after merging main.

Point	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
AgentX c1 TP8 EP1 75f53e	N/A	N/A	N/A
AgentX c2 TP8 EP1 291413	N/A	N/A	N/A
AgentX c4 TP8 EP1 8095cb	N/A	N/A	N/A
AgentX c8 TP8 EP1 0e432a	N/A	N/A	N/A
AgentX c16 TP8 EP1 2470c9	N/A	N/A	N/A
AgentX c48 TP8 EP8 25a62b	N/A	N/A	N/A
AgentX c64 TP8 EP8 673532	N/A	N/A	N/A
AgentX c96 TP8 EP8 e825cf	N/A	N/A	N/A
AgentX c128 TP8 EP8 d3fb44	N/A	N/A	N/A
AgentX c256 TP8 EP8 d7adce	N/A	N/A	N/A
AgentX c384 TP8 EP8 09e4fe	N/A	N/A	N/A

Note: All rows: pending.
Note: All rows: request errors unavailable.
Note: All rows: Δ N/A: point unavailable.

Next: Wait for all 11 AgentX points and 6 default evals of the final sweep, then verify artifacts against the frozen baseline and finish.

中文

最终完整 sweep · 运行中 · Run 40624895148 / attempt 1 · 2026-10-28 03:08 UTC
rocm/atom-dev:nightly_202610261558@sha256:57477e7a4b5156fcc7b21bab082eba98577752465f58479ac2ead8514f882a61 · 591909cd8a8c · 平均延迟
**变更：**在 591909c 上运行最终扫描：镜像和配方与修复 1 相同（DEP 点关闭 EPLB），合并 main 后追加了 perf-changelog 条目。实测数值及异常说明见上表。
**下一步：**等待最终扫描的全部 11 个 AgentX 点和 6 项默认评测完成，然后对照冻结的基线核验产物并结束。

Klaud-Cold · 2026-10-28T03:08:58Z

parked · Session 4 is waiting on 40624895148. Recovery reruns preempted runs and starts the next session once they end.

中文

parked · 第 4 个会话正在等待 40624895148。恢复流程会重跑被抢占的运行，并在运行结束后启动下一个会话。

github-actions · 2026-10-29T03:09:03Z

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40624895148

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40624895148

perf(amd): update DSv4-Pro FP4 MI355X ATOM AgentX image to nightly_20…

e7060af

…2610261558 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-project-automation Bot added this to InferenceX Board Oct 27, 2026

github-actions Bot added the klaud-parked Klaud parked until its queued or running benchmark runs end label Oct 27, 2026

github-actions Bot added klaud-in-progress Klaud is working on this candidate or will resume it and removed klaud-parked Klaud parked until its queued or running benchmark runs end labels Oct 27, 2026

fix(amd): disable EPLB on DSv4-Pro FP4 MI355X ATOM DEP points

09dc099

The first EPLB rebalance hangs the DEP8 server on rocm/atom-dev:nightly_202610261558 (ROCm/ATOM#2725 routing rewrite). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added klaud-parked Klaud parked until its queued or running benchmark runs end and removed klaud-in-progress Klaud is working on this candidate or will resume it labels Oct 27, 2026

github-actions Bot added klaud-in-progress Klaud is working on this candidate or will resume it and removed klaud-parked Klaud parked until its queued or running benchmark runs end labels Oct 27, 2026

github-actions Bot added klaud-parked Klaud parked until its queued or running benchmark runs end and removed klaud-in-progress Klaud is working on this candidate or will resume it labels Oct 27, 2026

github-actions Bot added klaud-in-progress Klaud is working on this candidate or will resume it and removed klaud-parked Klaud parked until its queued or running benchmark runs end labels Oct 28, 2026

github-actions Bot and others added 2 commits October 28, 2026 03:06

Merge remote-tracking branch 'origin/main' into klaud/auto-4565dac9cf…

4211e6b

…920b0a-6b1902ae19b3dc0a

perf-changelog: add DSv4-Pro FP4 MI355X ATOM AgentX image update entry

591909c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Klaud-Cold added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 28, 2026

github-actions Bot added the klaud-parked Klaud parked until its queued or running benchmark runs end label Oct 28, 2026

github-actions Bot removed the klaud-in-progress Klaud is working on this candidate or will resume it label Oct 28, 2026

Conversation

Klaud-Cold commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 27, 2026

Uh oh!

Klaud-Cold commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Klaud-Cold commented Oct 27, 2026

Uh oh!

Klaud-Cold commented Oct 27, 2026

Uh oh!

Klaud-Cold commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Klaud-Cold commented Oct 27, 2026

Uh oh!

Klaud-Cold commented Oct 27, 2026

Uh oh!

Klaud-Cold commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Klaud-Cold commented Oct 27, 2026

Uh oh!

Klaud-Cold commented Oct 28, 2026

Uh oh!

Klaud-Cold commented Oct 28, 2026

Uh oh!

Klaud-Cold commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Klaud-Cold commented Oct 27, 2026 •

edited

Loading

Klaud-Cold commented Oct 27, 2026 •

edited

Loading

Klaud-Cold commented Oct 27, 2026 •

edited

Loading

Klaud-Cold commented Oct 27, 2026 •

edited

Loading