You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(racer): GPUDirect reads into client host and GPU memory - #989
Adds host-read and gpu-read RPCs to the Racer client socket. The client passes a memfd or an exported CUDA allocation with SCM_RIGHTS, and the dataplane writes the object straight into it instead of streaming bytes over the socket. For gpu-read, the dataplane checks CRC64 and the AES-GCM tag and decrypts on the GPU, reads aligned disk pages with cuFile when it can, and lands peer pages in GPU memory over RDMA using dma-buf registered device pools. The on-disk record format moves to v2 so that ciphertext starts 4 KiB aligned; v1 records still read. Also adds a Python client, a racer-vllm loader plugin, NVIDIA library injection for the dataplane, and an opt-in label that mounts the client socket into app pods. When GPUDirect Storage is not available, Racer turns off cuFile's compatibility mode and reads disk pages through its own host path, which is faster. New metrics count the disk pages that cuFile reads straight into GPU memory. cuFile reads run on four reader threads and the cuFile driver opens at startup, and the vLLM loader ends its reads on page boundaries with four in flight, so a 7B model loads from one NVMe drive at close to the drive's speed.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Descriptor accumulation and CUDA context, synchronization, and allocation-lifetime defects can disrupt the dataplane or cause invalid GPU accesses during model loading.
Close weights in a finally block before leaving with client. If model.load_weights() raises after receiving a tensor, the exception traceback can retain the suspended generator, delaying the executor shutdown and GPU fences in weights.py:116-126 until the traceback is released. Background reads can then continue after the client closes. A consumer that returns without exhausting the iterator also leaves cleanup pending. Explicitly closing the iterator makes its cleanup run before client shutdown in both cases.
Include cuFile and dependencies in the CUDA runtime image
images/racer-dataplane/Containerfile:95
Include the cuFile user-space library and its dependencies in the CUDA-enabled runtime image. The default ubuntu:noble final stage installs only libgcc-s1 and optional RDMA packages (lines 104-114), while this step adds only the CUDA shim. NVIDIA compute/utility injection supplies driver libraries, not the toolkit's libcufile.so.0. Consequently, the shim's dlopen (gpu/native/gpu_cuda.c:640-643) fails and the helper disables cuFile (src/gpu.rs:895-898), so even a GPUDirect Storage-capable host falls back to host-memory disk reads. Package cuFile in the final image or provide an explicit supported injection path.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
GPU fallback device selection and CDI request wiring have unresolved correctness issues.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Preserve destination CUDA device for staging registration
clients/python/racer/src/racer/tensors.py:44
Preserve the wrapped tensor's CUDA device on this target. Both host and stream fallback call _dest_device(dest), which reads dest.device; without it, a target on cuda:3 requests new staging registration on GPU 0 (clients/python/racer/src/racer/client.py:44-50,211-219,639,680). Registration retains that GPU's primary context (cuda.py:206), so the read can fail if GPU 0 is unavailable or held by another process in exclusive-process mode, even when the destination GPU is usable. This also affects the vLLM torch-memory fallback. Expose tensor.device so staging uses the destination GPU.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Native GPU cryptography and memory lifetimes require hardware-backed validation and human review, alongside resolution of the retained correctness issues.
The int() conversions silently truncate malformed dimensions and byte offsets. For example, a U8 entry with shape: [1.9] and data_offsets: [0.9, 1.9] is accepted as shape (1,) and offsets [0, 1]; a negative dimension of -0.5 becomes zero. These altered ranges determine reads and tensor construction in clients/python/racer-vllm/src/racer_vllm/weights.py:86,109. Require integer arrays, excluding booleans, rather than coercing the values. Add rejection tests for fractional values, numeric strings, and booleans so malformed headers raise SafetensorsError instead of loading an altered tensor layout.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Native GPU cryptography and cross-device memory lifetimes need hardware-backed validation and human review, alongside resolution of the verified cleanup failure.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Descriptor association remains unresolved, and native CUDA/RDMA lifetimes and cryptographic paths need hardware-backed validation and final human review.
…over RDMA
Requester::request_landing threads a gpu-read's GPU ordinal down to the
peer transport. When the hop is RDMA and the rail's landing buffer is
on that GPU, accept_native grants a landing slot (else host memory as
usual). A completed landing write comes back as RelayResponse::Landed,
whose signed head is checked for exactly this page (landed_response_head)
and verified against the request binding (Forwarding::verify_landed).
…path
Fill::acquire_sealed now tries land_from_peer after local disk staging:
one CopyOnly request to the first provider with the request's GPU
ordinal. A landed page becomes SealedSource::Landed and is decrypted on
the GPU; it never joins single flight, memory, or disk. A host answer
to the landing request is used once instead of fetching again. Any
other outcome takes the usual path. Adds GPU helper tests for landing
buffers and decrypting landed ciphertext.
A page landed by RDMA has no host copy or disk mapping for its owner to
drop, so routing its failed verdict to the owner only consumed a retained
verdict slot. Record the corrupt miss locally instead.
…ved with
A descriptor-aware head read could return a plain HEAD and the start of a
following descriptor-bearing request from one recvmsg. Every descriptor was
attached to the first decoded head, so the HEAD was rejected for carrying an
unexpected descriptor and neither request was served.
Linux unix_stream_read_generic merges skbs into one stream read but stops
after an skb carrying SCM_RIGHTS. A client sends a head's descriptors with
the start of that head, so descriptors from one receive belong to the last
head that starts in its bytes. The receive loop now keeps the newest chunk's
descriptors apart: earlier chunks belong to the current head, and the last
chunk does too unless bytes remain after the head ends. Otherwise they are
carried on the lease with the read-ahead and the same rule is applied when
the next head is decoded from it. Carried descriptors still count toward
MAX_HEAD_DESCRIPTORS, close with the lease or a rejected head, and a
non-descriptor head read that finds them closes them and rejects.
The http layer still rejects pipelining at exchange completion, so a
pipelined host-read after a HEAD now gets the HEAD served and the connection
closed with the carried descriptor released.
Also make the client disconnect cancellation test pump until the hangup is
observed instead of a fixed pump count; it failed under parallel load.
cuFile rejects descriptors with O_NOFOLLOW (error 5019), and Racer opens
every slab file and block device with it, so cuFile was turned off on the
first direct read. Reopen the open file through /proc/self/fd before
registering it.
Without nvidia-fs or P2PDMA, cuFile's compatibility mode reads through a
bounce buffer on the GPU helper thread. Racer's host path does the same
work on I/O threads, so prefer it unless CUFILE_ALLOW_COMPAT_MODE is set.
cuFile classifies each file's block device through udev and fails to
register files it cannot classify (error 5030). Mount the host /run/udev
read-only when NVIDIA injection is on, and install libnuma1, which cuFile
loads at runtime.
The memory cache now keeps only ciphertext, and the range probe decrypts
cached or disk ciphertext on the CPU. Sealed reads probe live plaintext
only, and acquire_sealed stages cached ciphertext from memory before
trying cuFile or a peer, so the GPU decrypts it.
Make the simulation thread safe and add GpuFile::detach and FileReader so
a prepared cuFile read can run on another thread while the helper keeps
decrypting.
The GPU helper thread ran each cuFile read synchronously, so reads and
decryption took turns and one read at a time left the drive mostly idle.
Reads now run on four reader threads. The helper keeps file
registration and frees a handle only after its last read finishes.
Opening the driver probes every GPU and took about 0.8s on an eight-GPU
B200 node. It ran on the first sealed page, so every first model load paid
for it. Open it on a separate thread at startup instead. A failed open
turns cuFile off, as a failed stage does.
Neighboring chunks used to read the page they share twice. Chunks now end
on page boundaries where they can, and a tensor that crosses one is copied
out of each chunk into its own memory.
With page-aligned chunks and parallel cuFile reads, four staging buffers load a 7B model about 7% faster than two on one NVMe drive, close to the drive's GPUDirect limit.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Adds host-read and gpu-read RPCs to the Racer client socket. The client passes a memfd or an exported CUDA allocation with SCM_RIGHTS, and the dataplane writes the object straight into it instead of streaming bytes over the socket. For gpu-read, the dataplane checks CRC64 and the AES-GCM tag and decrypts on the GPU, reads aligned disk pages with cuFile when it can, and lands peer pages in GPU memory over RDMA using dma-buf registered device pools. The on-disk record format moves to v2 so that ciphertext starts 4 KiB aligned; v1 records still read. Also adds a Python client, a racer-vllm loader plugin, NVIDIA library injection for the dataplane, and an opt-in label that mounts the client socket into app pods. When GPUDirect Storage is not available, Racer turns off cuFile's compatibility mode and reads disk pages through its own host path, which is faster. New metrics count the disk pages that cuFile reads straight into GPU memory. cuFile reads run on four reader threads and the cuFile driver opens at startup, and the vLLM loader ends its reads on page boundaries with four in flight, so a 7B model loads from one NVMe drive at close to the drive's speed.