[weireweire]

Update inferencex-e2e/utils/srt-slurm from v2.59.1 (ae3d6877) to v2.65.2 (59920ea1), and synchronize the documented pin in RECIPES.md and CONFIGS.md.

The main reason is NVIDIA/srt-slurm#629. Dynamo frontends since ai-dynamo/dynamo#16454 (Dynamo 1.7 dev builds) answer /health with status only ({"status":"healthy","stage":"ready"}) and list registered instances only for /health?instances=true. srtctl's readiness wait counts workers from that list, so on images that ship Dynamo 1.7 it never sees a worker and polls until the health-check timeout. With #629, DynamoFrontend.probe_ready reads /health as before and asks /health?instances=true only when the answer has no instances key. Older frontends still get the same single request. With this pin, configs on Dynamo 1.7 images need no backport patch.

Three other upstream changes reach existing recipes at launch. They were found by rendering the full srun plan of all 2023 bound recipes under both pins with srt-slurm's mock launcher:

• KV-event ports ([NV] DSR1 FP8 GB200 Dynamo TRT #617): the base port moves from 5200 to 3000. This applies to 490 recipes: every Dynamo vLLM worker's DYN_VLLM_KV_EVENT_PORT and the SGLang and vLLM --kv-events-config endpoints. The highest allocated port is about 3060 plus the DP block, below NATS (4222) and OTEL (4317). Nothing in InferenceX hardcodes 5200.
• Dynamo frontend srun_options (chore: add conc to multinode benchmarks job names for viz #624): the frontend step now receives the recipe's srun_options, as the workers already do.
  • All 513 Dynamo recipes: the frontend gets --container-workdir=/infmax-workspace.
  • 252 of them: the frontend also gets their container-remap-root and/or mem: 0.
  • 20 recipes: the frontend also gets --cpu-bind=none. These are dsv4-fp4-gb200-dynamo-vllm-agentic-dspark-{agg,disagg} and dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-{agg,disagg}, and frontend CPU placement can move their results.
• Sidecar metrics ([NV] dsr1 fp4 b200 trt agg update #620): in the five dynamo.sidecar recipes (dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-{agg,disagg}), vllm-rs binds HTTP and gRPC on 0.0.0.0 instead of 127.0.0.1, and Tachometer adds engine scrape targets. AIPERF_SERVER_METRICS_URLS is unchanged.

No node assignment or GPU mask changes in any render. #312 makes an uneven multi-node GPU split a hard error, and no current recipe has one. The other changes in the range are opt-in or unused by InferenceX: #614, #615, #616, #618, #623, #624 reuse_container, sa-bench, dsight and mcp.

573-participating-gpus.patch stays: it still applies to v2.65.2, and src/srtctl/core/power is unchanged between the two pins, so the reason in its README row still holds. No other patch is affected. No benchmark config changes, so there is no changelog entry.

Validation details

• infx generate --config-key '*' writes 2023 bound recipes, and the pinned srtctl validates every one. The output is byte-identical between v2.59.1 and v2.65.2.
• Every patch in runners/srt-slurm/patches/ applies to a fresh v2.65.2 checkout with git apply --check, as the launcher applies them.
• Rendered the srun plan of all 2023 bound recipes under both pins with srt-slurm's mock launcher and each cluster's defaults. All render under both pins with the same exit codes; the 608 plans that differ after normalizing step order differ only by the three items above.
• python -m infx.matrix.changed --base origin/main lists no changed config keys.

中文

将 inferencex-e2e/utils/srt-slurm 从 v2.59.1（ae3d6877）升级到 v2.65.2（59920ea1），并同步 RECIPES.md 和 CONFIGS.md 中的版本说明。

主要原因是 NVIDIA/srt-slurm#629。自 ai-dynamo/dynamo#16454（Dynamo 1.7 dev 版本）起，Dynamo frontend 的 /health 只返回状态，只有 /health?instances=true 才列出已注册的实例。srtctl 的 readiness 等待按这个列表数 worker，所以在带 Dynamo 1.7 的镜像上永远等不到 worker，直到健康检查超时。#629 之后，DynamoFrontend.probe_ready 照旧先读 /health，只有返回里没有 instances 时才请求 /health?instances=true，旧版 frontend 收到的请求不变。升级后，使用 Dynamo 1.7 镜像的配置不再需要回移补丁。

另外三项上游改动会影响现有 recipe 的启动。这是用 srt-slurm 的 mock launcher 在新旧两个版本下渲染全部 2023 个 recipe 的 srun 计划后发现的：

• KV-event 端口（[NV] DSR1 FP8 GB200 Dynamo TRT #617）： 基础端口从 5200 改为 3000，涉及 490 个 recipe。最高分配端口约为 3060 加 DP 块，低于 NATS（4222）和 OTEL（4317）。
• Dynamo frontend 的 srun_options（chore: add conc to multinode benchmarks job names for viz #624）： frontend 现在也会收到 recipe 的 srun_options。所有 513 个 Dynamo recipe 的 frontend 多了 --container-workdir=/infmax-workspace。其中 20 个还会多 --cpu-bind=none，即 dsv4-fp4-gb200-dynamo-vllm-agentic-dspark-{agg,disagg} 和 dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-{agg,disagg}，frontend 的 CPU 绑核可能影响它们的结果。
• Sidecar 指标（[NV] dsr1 fp4 b200 trt agg update #620）： 5 个 dynamo.sidecar recipe（dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-{agg,disagg}）中，vllm-rs 改为在 0.0.0.0 上监听，Tachometer 增加 engine 抓取目标。AIPERF_SERVER_METRICS_URLS 不变。

所有渲染结果中节点分配和 GPU mask 都没有变化。其余改动对 InferenceX 是可选的或未使用。

保留 573-participating-gpus.patch：它仍能应用于 v2.65.2，且两个版本间 src/srtctl/core/power 没有变化，README 中保留它的理由仍然成立。其他补丁不受影响。没有 benchmark 配置改动，因此不加 changelog 条目。

验证： infx generate --config-key '*' 生成 2023 个绑定后的 recipe，全部通过固定版本 srtctl 的校验，且新旧版本的输出逐字节相同；runners/srt-slurm/patches/ 中的所有补丁都能用 git apply --check 应用到全新的 v2.65.2；python -m infx.matrix.changed --base origin/main 未列出任何改动的 config key。

AI model disclosure

• Claude Opus 5.5 (runtime identifier aws/anthropic/eccn-claude-opus-5-5) prepared the submodule bump, checked the patches and the generated recipes against the new pin, and wrote this description. A delegated agent on the same model reviewed the upstream changes in the range and rendered the srun plans under both pins to find their runtime impact on InferenceX launches.

PR checklist and change type

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe): dependency (srt-slurm submodule) bump

Checklist

• I have completed the AI model disclosure and kept it current
• I have tested my changes locally
• I have updated documentation if necessary
• If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

🔴 Every multi-node srt-slurm benchmark switches to a new srtctl (v2.65.2), yet no sweep has to run it before merge. The cause is the missing perf-changelog entry. The bump changes what each launch runs: GPU allocation (#312), KV-event ports (#617), trtllm-serve srun options (#623) and readiness probing (#629). AGENTS.md:87 says infx.matrix.changed cannot see launcher changes, so an empty list proves nothing here. Fix: add one entry at the end of perf-changelog.yaml covering the affected multi-node config keys ("*" as in the v2.39.1→v2.43.4 bump), then get an accepted /use <run_id> before merging.

Why this was flagged

The srt driver clones this pinned submodule for every NVIDIA srt-slurm multi-node launch and runs its srtctl (infx/launch/drivers/srt/checkout.py:82-97). Moving from ae3d6877 to 59920ea1 changes how jobs are placed and started. Per the PR's own list, that covers multi-node worker GPU allocation (#312), KV-event port choice (#617), trtllm-serve disagg srun options (#623) and frontend readiness probing (#629). AGENTS.md:87 says a change to what a benchmark runs needs a perf-changelog.yaml entry. It also says python -m infx.matrix.changed 'does not see script or launcher code, so judge script and launcher changes by what they run'. The PR leaves out the entry only because that tool listed no keys. AGENTS.md:89 says main never runs sweeps, and the entry is what triggers result collection and the /use gate. So the new orchestrator merges without any hardware sweep. A regression would first show up in some unrelated later PR's results, and nothing would tie it to this bump. An earlier pin bump (perf-changelog.yaml:9344-9347, v2.39.1 to v2.43.4) did add a "*" entry.

Verification: Trigger: any NVIDIA srt-slurm multi-node config launched after merge. It runs the new srtctl, and no sweep has run that srtctl in this repo first. inferencex-e2e/infx/launch/drivers/srt/checkout.py:82-97 clones repository_root() / SUBMODULE, so the submodule pointer decides which srtctl runs. AGENTS.md:87 requires a perf-changelog.yaml entry. The PR adds no entry.