Repository navigation
CollectiveX: keep PCI discovery in the bundled hwloc for aws-ofi-nccl / CollectiveX：在 aws-ofi-nccl 自带的 hwloc 中保留 PCI 发现 - #4612
Conversation
…uild The bundled hwloc was configured without I/O discovery, so aws-ofi-nccl found no NIC beside any GPU, its RDMA protocol failed to start (topo group size 0), and GIN, which runs only on that protocol, was unavailable: every nccl-ep EP16 case on H100 failed. hwloc now reads the PCI tree through a static libpciaccess (0.16, hash-pinned) linked into the plugin, and a plugin built on the bundled hwloc keys its own cache so the broken build is replaced. 内置 hwloc 曾在关闭 I/O 发现的情况下配置，导致 aws-ofi-nccl 找不到任何 GPU 旁的网卡，其 RDMA 协议无法启动 （拓扑组大小为 0），而 GIN 只能运行在该协议上，因此不可用：H100 上所有 nccl-ep EP16 用例均失败。现在 hwloc 通过静态链接进插件的 libpciaccess（0.16，固定哈希）读取 PCI 树；基于内置 hwloc 构建的插件使用独立缓存键， 以替换此前损坏的构建。
libtool packed libpciaccess.a into the plugin's convenience archive as an unreadable member, so the aws-ofi-nccl link failed. Its objects now join libhwloc.a, and hwloc's .la is dropped so libtool links the archive as is. libtool 曾把 libpciaccess.a 作为无法读取的成员打包进插件的便利库，导致 aws-ofi-nccl 链接失败。现在其目标文件 并入 libhwloc.a，并删除 hwloc 的 .la，使 libtool 直接链接该静态库。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and found no bugs. It changes how the aws-ofi-nccl plugin's static hwloc is built, adds a new third-party source download, and touches the CODEOWNER-covered collectivex/ path, so a human look would still be worthwhile.
What was reviewed:
- The libpciaccess fetch is pinned by sha256 the same way as the hwloc and aws-ofi-nccl tarballs, and a failure in any step returns 1 before the cache is marked ready.
- The
ar x/ar rsmerge: libpciaccess's object names (common_*.o,linux_*.o) don't collide with hwloc's (topology*.o,pci-common.o, ...), soar rsadds them rather than replacing hwloc members. - The cache key: the new variant applies only when
/usr/include/hwloc.his missing, and the/tmpfallback path uses the same key, so stale plugins built with--disable-ioget rebuilt.
Extended reasoning...
The change is about 40 lines across two CollectiveX shell scripts. It adds a sha256-pinned libpciaccess 0.16 build, turns hwloc I/O discovery back on, merges libpciaccess's objects into libhwloc.a and drops libhwloc.la, and adds the bundled-build variant to the aws-ofi-nccl cache key. It has no auth or data-exposure surface. It does add a new external source download, which is checksum-verified. I deferred rather than approved because collectivex/ is covered by a CODEOWNERS entry and because the libtool archive-merge workaround is a build-chain design choice that the code owners should confirm.
Summary
nccl-ep EP16 on h100-dgxc failed with "NCCL EP requires NCCL Device API support". The bundled aws-ofi-nccl plugin logged "Unexpected topo group size of 0": its static hwloc was configured with
--disable-io, so hwloc saw no PCI devices, the plugin could not group NICs with GPUs, and NCCL GIN got no RDMA protocol.libhwloc.aand drop hwloc's.la. libtool otherwise packedlibpciaccess.ainto the plugin's convenience archive as an unreadable member and the link failed.Validated on h100-dgxc:
h100-dgxc-nccl-ep-ll-bf16-n2now succeeds, with the oracle green on every rung.中文
h100-dgxc 上的 nccl-ep EP16 报错 "NCCL EP requires NCCL Device API support"。自带的 aws-ofi-nccl 插件提示 "Unexpected topo group size of 0"：其静态 hwloc 使用了
--disable-io，看不到任何 PCI 设备，插件无法将网卡与 GPU 分组，NCCL GIN 因此没有 RDMA 协议。libhwloc.a并删除 hwloc 的.la，否则 libtool 会把libpciaccess.a作为无法读取的成员打包进插件便利库，导致链接失败。已在 h100-dgxc 上验证：
h100-dgxc-nccl-ep-ll-bf16-n2成功，每一档 oracle 均通过。AI disclosure
Drafted and tested with an AI coding assistant; reviewed by the author.