Repository navigation
[Klaud Cold] Update DeepSeek-V4-Pro FP4 GB300 Dynamo-TRT AgentX MTP image to TensorRT-LLM 1.4.0rc1 / [Klaud Cold] 将 DeepSeek-V4-Pro FP4 GB300 Dynamo-TRT AgentX MTP 镜像更新至 TensorRT-LLM 1.4.0rc1 - #4583
Conversation
…0rc1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
…LM 1.4.0rc1 update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Klaud-Cold
added
the
full-sweep-fail-fast
|
Initial attempt · Failed · Run 40539806174 / attempt 1 · 2026-10-27 16:48 UTC Next: c24 prefill exited at startup: the pinned Dynamo fork (rev 2cbbdc86) rejected --publish-metrics with 'Unrecognized argument', which srtctl ae3d687 now passes by default (NVIDIA/srt-slurm be5ab92b, backends/trtllm.py publish_metrics=True); fail-fast cancelled the other points, so repair 1 sets engine.publish_metrics: false. 中文初次尝试 · 失败 · Run 40539806174 / attempt 1 · 2026-10-27 16:48 UTC |
|
parked · Session 1 is waiting on 40539806174. Recovery reruns preempted runs and starts the next session once they end. 中文parked · 第 1 个会话正在等待 40539806174。恢复流程会重跑被抢占的运行，并在运行结束后启动下一个会话。 |
github-actions
Bot
added
the
klaud-parked
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40629721463 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40629721463 |
|
resumed · The session ended before finishing; session 2 continues it in a new run of its planner. 中文resumed · 会话未完成即已结束；第 2 个会话将在其规划运行的新运行中继续。 |
github-actions
Bot
added
klaud-in-progress
Klaud-Cold
removed
the
full-sweep-fail-fast
… Dynamo-TRT AgentX Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2b5e25-5f20d20eaa35dce7 # Conflicts: # inferencex-e2e/perf-changelog.yaml
Klaud-Cold
added
the
full-sweep-fail-fast
|
Repair 1/5 · Failed · Run 40544295139 / attempt 1 · 2026-10-27 17:48 UTC Next: Node fault: runners gb300-nv_18 and gb300-nv_19 (batch3) inject DYN_TCP_RPC_HOST=enP22s22f0np0 (an interface name, set by neither the repo, srt-slurm ae3d687 nor the image), so Dynamo 2cbbdc86 failed with 'Invalid TCP bind address' (lib/runtime/src/pipeline/network/manager.rs:272) on c2626 and the c24 eval, while c388 and c24 on gb300-nv_00/_01 became healthy before fail-fast; the failed jobs rerun on the same head after 90 minutes as infrastructure retry 1. 中文修复 1/5 · 失败 · Run 40544295139 / attempt 1 · 2026-10-27 17:48 UTC |
|
parked · Session 2 is waiting on 40544295139. Recovery reruns preempted runs and starts the next session once they end. 中文parked · 第 2 个会话正在等待 40544295139。恢复流程会重跑被抢占的运行，并在运行结束后启动下一个会话。 |
github-actions
Bot
added
klaud-parked
|
resumed · The session ended before finishing; session 3 continues it in a new run of its planner. 中文resumed · 会话未完成即已结束；第 3 个会话将在其规划运行的新运行中继续。 |
github-actions
Bot
added
klaud-in-progress
|
parked · Session 3 is waiting for recovery to rerun the failed jobs of 40544295139 after 2026-10-27 19:18 UTC. Recovery starts the next session once they end. 中文parked · 第 3 个会话正在等待恢复流程在 2026-10-27 19:18 UTC 后重跑 40544295139 的失败作业。运行结束后，恢复流程将启动下一个会话。 |
github-actions
Bot
added
klaud-parked
|
resumed · The session ended before finishing; session 4 continues it in a new run of its planner. 中文resumed · 会话未完成即已结束；第 4 个会话将在其规划运行的新运行中继续。 |
github-actions
Bot
added
klaud-in-progress
|
Infrastructure retry 1/2 · Failed · Run 40544295139 / attempt 2 · 2026-10-27 20:26 UTC Next: Not transient: c736 and the c2626 eval again landed on batch3 runner gb300-nv_18 (nodes im-gb300-r03-*), where DYN_TCP_RPC_HOST=enP22s22f0np0 is inherited from the node environment (set by neither this repo nor srt-slurm ae3d687); Dynamo fork 2cbbdc86 (upstream f5de1f4f, 2026-08-07) treats it as a literal address and exits with 'Invalid TCP bind address' (lib/runtime/src/utils/ip_resolver.rs, network/manager.rs:272), while the c24/c388 evals on gb300-nv_00/_01 (host=10.3.10.x) passed. Interface names in DYN_TCP_RPC_HOST are supported upstream since ai-dynamo/dynamo#11870, so repair 2 moves the recipe Dynamo source pin to upstream v1.6.1 (0d9c24a8), which also contains #12162. 中文基础设施重试 1/2 · 失败 · Run 40544295139 / attempt 2 · 2026-10-27 20:26 UTC |
github-actions
Bot
removed
the
klaud-parked
Klaud-Cold
removed
the
full-sweep-fail-fast
…namo 1.6.1 Dynamo 1.6.1 accepts --publish-metrics (ai-dynamo/dynamo#12162), and the recipe requires trtllm_kv_cache_utilization in the frontend metrics export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2b5e25-5f20d20eaa35dce7 # Conflicts: # inferencex-e2e/perf-changelog.yaml
Klaud-Cold
added
the
full-sweep-fail-fast
|
Repair 5/5 · Failed · Run 40618262722 / attempt 1 · 2026-10-28 03:58 UTC
Note: AgentX c4 PTP4x1/DTP8x4 122428, AgentX c24 PTP4x1/DTP4x6 0c3e2a, AgentX c736 PTP8x2/DTP32x1 1ee938, AgentX c1152 PTP8x3/DTP16x1 d7555c, AgentX c2626 PTP8x5/DTP16x1 1c3ebd: cancelled.
Next: c388 decode (TP32, MEGAMOE_DEEPGEMM) hung at 02:19 in CUDA-graph warmup for batch size 2: every rank hit the DeepGEMM grid sync timeout (1800s, deep_gemm/comm/barrier.cuh:43), which then surfaced as CUDA 719 and fail-fast cancelled the rest. The same TP32 decode warmed up and passed the c388 and c736 eval jobs in this run, so this is a nondeterministic hang, not a config error. Main has since appended perf-changelog entries, so the next step merges main, keeps our entry at the tail and redispatches the final sweep as the single infrastructure retry; a repeat of this hang ends the candidate. 中文修复 5/5 · 失败 · Run 40618262722 / attempt 1 · 2026-10-28 03:58 UTC |
|
parked · Session 7 is waiting on 40618262722. Recovery reruns preempted runs and starts the next session once they end. 中文parked · 第 7 个会话正在等待 40618262722。恢复流程会重跑被抢占的运行，并在运行结束后启动下一个会话。 |
github-actions
Bot
added
klaud-parked
|
resumed · The session ended before finishing; session 8 continues it in a new run of its planner. 中文resumed · 会话未完成即已结束；第 8 个会话将在其规划运行的新运行中继续。 |
github-actions
Bot
added
klaud-in-progress
…2b5e25-5f20d20eaa35dce7 # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Final full sweep · Passed · Run 40629721463 / attempt 1 · 2026-10-28 15:38 UTC
Note: AgentX c24 PTP4x1/DTP4x6 0c3e2a: 1 request error.
Next: Mark ready for maintainer review. 中文最终完整 sweep · 已通过 · Run 40629721463 / attempt 1 · 2026-10-28 15:38 UTC |
|
parked · Session 8 is waiting on 40629721463. Recovery reruns preempted runs and starts the next session once they end. 中文parked · 第 8 个会话正在等待 40629721463。恢复流程会重跑被抢占的运行，并在运行结束后启动下一个会话。 |
github-actions
Bot
added
klaud-parked
|
validated · Repairs: unknown · Runs: 40539723267, 40539806174, 40544020308, 40544244228, 40544295139, 40578326154, 40578393219, 40581645270, 40581729738, 40581795226, 40598580993, 40598638914, 40618234134, 40618262722, 40629721463 中文validated · 修复次数：未知 · 运行：40539723267, 40539806174, 40544020308, 40544244228, 40544295139, 40578326154, 40578393219, 40581645270, 40581729738, 40581795226, 40598580993, 40598638914, 40618234134, 40618262722, 40629721463 |
github-actions
Bot
added
klaud-finished
|
Claude finished @Klaud-Cold's task in 29s —— View job ReviewLGTM - no blocking issues found.
For maintainers, not a code defect: the final sweep at high concurrency is clearly worse than the 1.3.0rc24 baseline:
Only c4 and c24 got better. The TTFT jump fits turning SWA scratch reuse off on prefill. The changelog description doesn't mention that override. Before merging, please decide whether this regression is acceptable. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # 1.4.0rc1 always enforces the SWA scratch resize check (debug-only in 1.3.0rc24), | ||
| # which c388 prefill tripped; DeepSeek-V4 model defaults turn scratch reuse on. | ||
| enable_swa_scratch_reuse: false |
There was a problem hiding this comment.
🟡 nit (optional): The new recipe comments record history, which AGENTS.md "Comments" forbids. Examples are "(debug-only in 1.3.0rc24)", "which c388 prefill tripped" and "Upstream v1.6.1: resolves ..." at lines 13-14. Fix: cut each block to one short technical line about why the value is set. For example: "# TRT-LLM 1.4 enforces the SWA scratch resize check; DeepSeek-V4 defaults enable reuse." Leave the version history and past results to git and perf-changelog.yaml.
Why this was flagged
AGENTS.md:57 says comments must not record history, meaning how code evolved, what it replaced, or past results. AGENTS.md:58 allows one short technical line per line or block. The comment added at disagg-variants.yaml:61-62 compares behaviour with the old 1.3.0rc24 image and names a past run failure (c388 prefill). The comment at disagg-variants.yaml:13-15 lists upstream PRs that fixed problems in the fork it replaces. Each of these blocks is three lines long. The base did not have these comments. The result is a convention slip, not a runtime fault.
Verification: nit. This is a repository-rule violation and nothing breaks at runtime. AGENTS.md:57 says: "No history: how code evolved, what it replaced, where it came from, or past results." At disagg-variants.yaml:61-62, the parenthetical "(debug-only in 1.3.0rc24)" describes the image this PR replaces, so it is history.
Goal: Update TensorRT-LLM image from nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc24 to nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc1.
Baseline: 2026-09-03 ·
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc24Mean latency · Sources: API 1, API 2, API 3, API 4
Stop: Close this PR at any time to stop this candidate now; Klaud cancels its runs and deletes its branch.
中文
**目标：**将 TensorRT-LLM 镜像从 nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc24 更新至 nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc1。
**基线：**2026-09-03 ·
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc24平均延迟 · 来源： API 1, API 2, API 3, API 4；数值及异常说明见上表。
**停止：**随时关闭此 PR 即可立即停止该候选：Klaud 会取消其运行并删除其分支。