Repository navigation
CollectiveX: run the MI300X and MI325X pools on the vLLM ROCm image / CollectiveX：在 vLLM ROCm 镜像上运行 MI300X 与 MI325X 资源池 - #4581
Merged
Merged
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Oseltamivir
force-pushed
the
cx-results-barite
branch
from
October 28, 2026 02:04
f91fd33 to
a16cdbf
Compare
… pools Barite's MI300X and MI325X nodes carry eight 400 GbE Broadcom RoCE v2 rails each. Cross-node RDMA reaches line rate on matching and crossed rails (ib_write_bw, about 390 Gb/s), so the no-fabric limit of the retired TensorWave pools does not apply. Both SKUs gain MoRI and UCCL-EP EP16, MoRI low-latency EP16 (vLLM's InterNodeV1LL) and UCCL-EP low-latency EP8. CollectiveX：在 Barite MI300X 与 MI325X 资源池上运行 EP16 与低延迟模式 Barite 的 MI300X 与 MI325X 节点各有 8 条 400 GbE Broadcom RoCE v2 通道。跨节点 RDMA 在同通道与交叉 通道上都达到线速（ib_write_bw 约 390 Gb/s），因此已退役的 TensorWave 资源池"无网络"的限制不再适用。 两个 SKU 新增 MoRI 与 UCCL-EP 的 EP16、MoRI 低延迟 EP16（vLLM 的 InterNodeV1LL）以及 UCCL-EP 低延迟 EP8。 (cherry picked from commit c0191e8)
MoRI's shmem bootstrap chose its own interface. On Barite's per-rail /31 links that interface has no kernel route to the root rank, so every second-node rank failed PhoneHomeProtocol and EP16 never started. The network profile now exports MORI_SOCKET_IFNAME alongside NCCL's, Gloo's and UCCL's. CollectiveX：让 MoRI 的 bootstrap 使用注册表中的 socket 接口 MoRI 的 shmem bootstrap 会自行选择网络接口。在 Barite 的每通道 /31 链路上，该接口没有到 root rank 的内核路由，因此第二个节点的所有 rank 都在 PhoneHomeProtocol 阶段失败，EP16 无法启动。网络配置现在会与 NCCL、Gloo 和 UCCL 一样导出 MORI_SOCKET_IFNAME。 (cherry picked from commit 7caeb5c)
…ards Barite's hosts run Broadcom's out-of-tree bnxt_re driver (237.1.137.0, kernel ABI 8) with Broadcom's matching provider. The image's stock rdma-core bnxt_re provider supports only ABI 1, so libibverbs listed no device and MoRI asserted "no rdma device found". A registry network field, host_verbs_provider, now names a host provider that multi-node shards bind read-only over the image's file of the same name. CollectiveX：将 Barite 的 Broadcom verbs provider 绑定进多节点分片 Barite 主机运行 Broadcom 的树外 bnxt_re 驱动（237.1.137.0，内核 ABI 8）及配套 provider。镜像自带的 rdma-core bnxt_re provider 只支持 ABI 1，因此 libibverbs 找不到设备，MoRI 断言 "no rdma device found"。 新增注册表网络字段 host_verbs_provider，指定一个主机 provider，多节点分片以只读方式将其绑定到镜像中同名文件上。 (cherry picked from commit 8677dd1)
…ainer The bind from the previous commit could not work: Broadcom's host libbnxt_re (237.1.137.0, the only provider that speaks the hosts' bnxt_re kernel ABI 8) is linked on Ubuntu 24.04 and needs GLIBC_2.38, while the ROCm images ship glibc 2.35, and their rdma-core 39 libibverbs loads only the providers in its own directory. Multi-node shards now bind the host provider under /opt/collectivex/host-verbs, and backend preparation installs it into the node's writable container over the image's file. Its one GLIBC_2.38 symbol, __isoc23_strtol, is unversioned and served by a shim that forwards to strtol, and elf_vernaux.py retargets the unused GLIBC_2.38 requirement to GLIBC_2.2.5. Preparation fails unless the result loads and libibverbs lists every registry RDMA device. MoRI JIT-compiles its device kernels for the NIC it finds, so it picks bnxt on its own. CollectiveX：在容器内安装 Barite 的 Broadcom verbs provider 上一个提交中的绑定方式无法生效：Broadcom 的主机 libbnxt_re（237.1.137.0，唯一支持主机 bnxt_re 内核 ABI 8 的 provider）在 Ubuntu 24.04 上链接，需要 GLIBC_2.38，而 ROCm 镜像只有 glibc 2.35，且其 rdma-core 39 的 libibverbs 只加载自身目录中的 provider。 多节点分片现在将主机 provider 绑定到 /opt/collectivex/host-verbs，由后端准备步骤将其安装到该节点可写 容器中、覆盖镜像中的同名文件。其唯一需要 GLIBC_2.38 的符号 __isoc23_strtol 被去除版本，并由转发到 strtol 的 shim 提供；elf_vernaux.py 将不再使用的 GLIBC_2.38 依赖改为 GLIBC_2.2.5。只有结果能够加载、 且 libibverbs 列出注册表中所有 RDMA 设备时，准备步骤才会成功。MoRI 会针对检测到的 NIC 即时编译设备 内核，因此会自动选择 bnxt。 (cherry picked from commit ce1bd1c)
The vLLM ROCm image ships Broadcom's provider (bnxt-rocelib 235.2.86.0) under /usr/local/lib, which lists every Barite RDMA device as-is, and it has no ibv_devices. The install step now lists devices through libibverbs from Python, keeps the image's provider when it lists every registry device, and otherwise installs the patched host provider into rdma-core's provider directory, which libibverbs searches before the loader path. CollectiveX：镜像自带的 verbs provider 可用时予以保留 vLLM ROCm 镜像在 /usr/local/lib 下自带 Broadcom 的 provider（bnxt-rocelib 235.2.86.0），可直接列出 Barite 的全部 RDMA 设备，且镜像中没有 ibv_devices。安装步骤现在通过 Python 调用 libibverbs 列出设备：若镜像的 provider 已列出注册表中的全部设备则保留，否则将修补后的主机 provider 安装到 rdma-core 的 provider 目录， libibverbs 会先于加载器路径搜索该目录。 (cherry picked from commit 66a2709)
run.sh staged the resolved matrix as .shards/matrix-<sku>.json and each shard control as .shards/<shard>.json, so two invocations from one checkout (one per backend, run in parallel) overwrote each other's files and ran the other's shards. Each invocation now stages under .shards/run-<pid>, removed on exit. CollectiveX：为每次 run.sh 调用使用独立的分片目录 run.sh 此前将解析后的矩阵写入 .shards/matrix-<sku>.json，每个分片的控制文件写入 .shards/<shard>.json，因此同一检出目录中的两次调用（例如按后端并行运行）会互相覆盖文件并运行对方的分片。 现在每次调用都在 .shards/run-<pid> 下暂存，并在退出时删除。 (cherry picked from commit 8ef9db6)
salloc --no-shell does not notice a job that ends between its grant and salloc's own poll: on h100 a node prolog fails the job ten seconds in, and the launcher then waited for hours. The launcher now watches the queued job, stops salloc once Slurm no longer lists it without a grant, and retries the allocation (three attempts, as for validation failures). CollectiveX：作业已结束时不再继续等待资源分配 salloc --no-shell 不会察觉在授予分配与其自身轮询之间结束的作业：在 h100 上，节点 prolog 会在作业开始 10 秒后使其失败，启动器随后会等待数小时。启动器现在会监视排队中的作业，当 Slurm 不再列出该作业且未授予 分配时停止 salloc，并重试分配（与验证失败相同，最多三次）。 (cherry picked from commit d990d13)
The Barite MI300X and MI325X pools ran EP in an MI35x-tagged rocm/sgl-dev build. They move to vLLM's release image vllm/vllm-openai-rocm:v0.32.0 (ROCm 7.2.3, MoRI v1.2.3.post1 built for every NIC backend). Its Broadcom verbs provider matches the hosts' bnxt_re driver, so the launcher keeps the image's own provider. The other pools move once they are rerun. CollectiveX：在 vLLM v0.32.0 ROCm 镜像中运行 mi300x 与 mi325x 的 EP Barite 的 MI300X 与 MI325X 资源池此前在带 MI35x 标签的 rocm/sgl-dev 构建中运行 EP。现改用 vLLM 的发布镜像 vllm/vllm-openai-rocm:v0.32.0（ROCm 7.2.3，MoRI v1.2.3.post1，为所有 NIC 后端构建）。该镜像自带的 Broadcom verbs provider 与主机的 bnxt_re 驱动匹配，因此启动器保留镜像自带的 provider。其余资源池在重跑后跟进。
Twenty MoRI cases measured by hand on the Barite MI300X and MI325X pools in vllm/vllm-openai-rocm:v0.32.0: normal decode and prefill at EP8 and EP16, and low-latency decode at EP16, in bf16 and fp8. Every case completed and passed the correctness oracle. CollectiveX：发布手动运行的 mi300x 与 mi325x MoRI 结果 在 Barite MI300X 与 MI325X 资源池上，于 vllm/vllm-openai-rocm:v0.32.0 中手动测量了二十个 MoRI 用例：EP8 与 EP16 的 normal decode 和 prefill，以及 EP16 的 low-latency decode，覆盖 bf16 与 fp8。所有用例均已完成并通过正确性校验。
…blocks results UCCL-EP normal mode is green at EP8 and EP16 in bf16 and fp8 on both Barite SKUs. Low-latency EP8 hangs until the run timeout kills it, while low-latency EP16 bf16 is green, so the registry now lists UCCL-EP low-latency at EP16 only. Low-latency fp8 fails the oracle (max relative error 0.07-0.11) at every size; mi300x keeps that result as a red cell, and the mi325x run was killed before writing one. CollectiveX：发布 mi300x 和 mi325x 手动运行的 UCCL-EP 与 swap-blocks 结果 UCCL-EP normal 模式在两个 Barite SKU 上 EP8 和 EP16、bf16 和 fp8 全部通过。low-latency EP8 会一直挂起直到被运行超时终止，而 low-latency EP16 bf16 通过，因此注册表现在只在 EP16 列出 UCCL-EP low-latency。low-latency fp8 在所有规模上都未通过校验（最大相对误差 0.07-0.11）；mi300x 将该结果保留为红色单元格，mi325x 的运行在写出结果前被终止。
The first run had a pair period spread of up to 2x between p50 and p99 and ran slower than bf16 at every size. The rerun is tight (p99 within 3% of p50) and sits below bf16, as fp8 should. CollectiveX：替换 mi300x MoRI EP16 fp8 prefill 的噪声结果 首次运行的 pair period p99 最高达 p50 的 2 倍，且在每个规模下都慢于 bf16。重跑结果稳定（p99 与 p50 相差不到 3%），并低于 bf16，符合 fp8 的预期。
…nt squeue failures from ending a queued allocation run.sh records the source commit once but every shard restages the checkout, so a checkout updated mid-run stamped later shards with a commit they did not run. It now stops before the next shard when HEAD or the collectivex/ diff has moved. The salloc watcher treated a failed squeue as the job having ended; only an empty listing or a purged job id counts now. The MI300X/MI325X results leave this change and are rerun from the merged commit. run.sh 只记录一次源提交，但每个分片都会重新暂存检出内容；运行中途更新检出会让后续分片带上并未运行的提交。 现在若 HEAD 或 collectivex/ 的改动发生变化，会在下一个分片前停止。salloc 监视器曾把 squeue 查询失败当作作业结束； 现在只有空列表或已清除的作业号才算结束。MI300X/MI325X 结果从此变更中移除，并将从合并后的提交重新运行。
Oseltamivir
force-pushed
the
cx-results-barite
branch
from
October 28, 2026 06:35
a16cdbf to
c3a422b
Compare
运行时注释中按 SKU 描述 Broadcom 资源池。
Oseltamivir
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Setup for the MI300X and MI325X pools on the vLLM
v0.32.0ROCm image. The results that used to be in this PR are gone. Every result file claimed a source commit that is not onmain, and seven MoRI EP16 files claimed a commit that did not run them, because the checkout changed while the run was going. Both pools are being rerun from this PR's merge commit, and those results will arrive in a separate PR.Changes:
vllm/vllm-openai-rocm:v0.32.0, and register EP16 and low-latency for these pools (ll_backends: MoRI and UCCL-EP at EP16).run.sh:collectivex/diff has changed since the run started. Every shard restages the checkout, so a change mid-run used to stamp later shards with a commit they did not run.squeuequery no longer counts as "job gone". Only an empty listing or a purged job id does.Validation
pytest collectivex/tests: 238 passed, 12 skipped.cases.jsonmatches a fresh resolve.AI model disclosure
claude-opus-5-5, run through Claude Code): the changes, the hand runs, the checks and this description.中文
摘要
在 vLLM
v0.32.0ROCm 镜像上运行 MI300X 与 MI325X 资源池所需的环境改动。本 PR 原有的结果已全部移除：所有结果文件记录的源提交都不在main上，其中 7 个 MoRI EP16 文件记录的提交并不是实际运行它们的提交，因为运行期间检出内容发生了变化。两个资源池正在基于本 PR 的合并提交重新运行，结果将在另一个 PR 中提交。改动：
vllm/vllm-openai-rocm:v0.32.0中运行 mi300x/mi325x 的 EP，并为这两个资源池注册 EP16 与低延迟（ll_backends：MoRI 与 UCCL-EP 的 EP16）。run.sh：collectivex/的改动发生变化，在下一个分片之前停止。每个分片都会重新暂存检出内容，因此运行中途的改动曾让后续分片带上并未运行的提交。squeue查询失败不再被视为“作业已结束”，只有空列表或已清除的作业号才算。验证
pytest collectivex/tests：238 通过，12 跳过。cases.json与重新解析的结果一致。