Skip to content

Fail when a requested RDMA device is not found - #4180

Open
erwinzhang7 wants to merge 1 commit into
ml-explore:mainfrom
erwinzhang7:fix-3777-jaccl-missing-device
Open

Fail when a requested RDMA device is not found#4180
erwinzhang7 wants to merge 1 commit into
ml-explore:mainfrom
erwinzhang7:fix-3777-jaccl-missing-device

Conversation

@erwinzhang7

Copy link
Copy Markdown
Contributor

Fixes #3777.

create_connections() walks device_names and, for each non-empty name, searches the
enumerated verbs devices. If the name matches nothing the inner loop simply falls out and the
next name is processed — nothing is appended and nothing is reported. It throws when
open_device fails, but not when the device is absent entirely.

The result is a vector shorter than device_names. MeshGroup and RingGroup size
themselves from device_names (size_(device_names.size())), so every index from
connections.size() upward reads past the end, and the garbage is passed to ibv_reg_mr as a
protection domain.

This is why it is not a missing null check: the read is out of bounds before there is a
pointer to check. With -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_DEBUG the same run
aborts at vector.h:406, libc++ Hardening assertion __n < size() failed: vector[] index out of bounds.

Reproduction

No RDMA hardware and no second node are needed — the absence of a verbs device is what
triggers it. On macOS 26.6 with rdma_ctl status reporting disabled, librdma.dylib still
loads and every ibv_* symbol resolves, so jaccl::is_available() returns true while
ibv_get_device_list reports zero devices.

std::vector<std::string> names = {"", "rdma_bogus0"};
jaccl::AllGatherFn agf = [](const char* s, char* d, size_t n) { std::memcpy(d, s, n); };
jaccl::MeshGroup mesh(0, names, jaccl::SideChannel(0, names.size(), agf));

Single process, stub all-gather, no coordinator. The stub is never called: the constructor
dies earlier, inside MeshGroup::allocate_buffers().

is_available()            = true
device_names.size()       = 2
create_connections().size = 1

EXC_BAD_ACCESS (code=1, address=0x4)
libibverbs.dylib`ibv_reg_mr_iova2 + 44

After this change the same program throws instead:

[jaccl] Could not find device rdma_bogus0 (0 available)

Fixing it in create_connections rather than at the call sites means the invariant the
callers already rely on — that the returned vector is as long as device_names — is restored
at its source, so no bounds or null guards are needed downstream.

Checked

Valid inputs are unaffected: an empty list returns 0, and lists of empty self-slots return one
connection each (1 → 1, 3 → 3). Absent names now throw whether they appear first or last.

Notes

The existing open_device and MESH_MAX_PEERS failures also throw without calling
free_device_list, and I matched that rather than changing the surrounding behaviour. Happy
to add the cleanup on all three paths if you would prefer.

I did not add an automated test: there is no jaccl test target today — tests links only
mlx and doctest, and jaccl is a PRIVATE link inside mlx, so create_connections is
not reachable from it — and jaccl itself only builds on macOS SDK ≥ 26.2. Adding that
scaffolding seemed larger than the fix. Glad to add a guarded test if you would like one.

Out of scope: @cdvankammen's report on this issue of the same crash with rdma_en2
present on both hosts is not addressed here and I could not verify it — I have no
RDMA-enumerating hardware. This PR only covers the fail-fast path for an absent device.

create_connections() skipped names that matched no enumerated device without
reporting anything, so it returned fewer connections than it was given names.
Its callers size themselves from device_names, so the extra indices read past
the end of the vector and the garbage was passed to ibv_reg_mr as a protection
domain.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant