cuda-core: define boolean storage and element padding/packing policy #1503

Activity

1. elibol

   added

   interopInterop with external compilers / frontendsInterop with external compilers / frontends

   cutile-rsTile programming model: cutile crates, compiler, macros, IR, kernels, and bookTile programming model: cutile crates, compiler, macros, IR, kernels, and book

   cuda-oxideSIMT programming model: rustc backend, cargo oxide, cuda-device, cuda-host, bookSIMT programming model: rustc backend, cargo oxide, cuda-device, cuda-host, book

   runtimeShared runtime crates: cuda-bindings, cuda-core, cuda-core-derive, cuda-async; Tile and SIMT co-ownShared runtime crates: cuda-bindings, cuda-core, cuda-core-derive, cuda-async; Tile and SIMT co-own

   on Oct 13, 2026
2. elibol commented on Oct 13, 2026

   

   elibol

   on Oct 13, 2026

   CollaboratorAuthor

   More actions

   Consolidating #1513 into this issue. Its body follows, with the proposed direction at the end.

   ---

   The host crates model every device buffer as [T]: a count of elements, each size_of::<T>() bytes, one element per address step. That holds for byte-and-wider scalars and for bool, which is one byte. It does not hold for sub-byte element types, and it leaves the layout rules for aggregate element types implicit. Both gaps surface as soon as we exchange tensors with other frameworks, so this issue is about writing the storage model down and fixing the few places where the code is ambiguous.

   What the tree does today

   • Sub-byte types. Tile IR's i4 and f4E2M1FN exist in kernels (cutile::ElementType) but are not DTypes, because one element is not one byte. Host storage for FP4 is f4e2m1fnx2 in cuda-core/src/dtype.rs, a byte holding two values with the first value in the low nibble, and kernels move between the two with pack and unpack. DTypeId::F4E2M1FNX2.size_in_bytes() is 1, which is the size of the storage element, not of a logical FP4 value. A Tensor<f4e2m1fnx2> therefore has its shape in byte pairs, and an odd logical extent cannot be expressed.
   • Booleans. One byte per element holding 0x00 or 0x01. Tile IR specifies this for pointer-tile loads and stores (operations, cuda_tile.load_ptr_tko 9.6.3 and cuda_tile.store_ptr_tko 9.6.6), and it matches the Rust bool layout.
   • Aggregate element types. #[derive(DeviceCopy)] in cuda-core-derive checks that every field is DeviceCopy and nothing else. It accepts repr(Rust) structs and structs with padding. This is sound for the SIMT path because the same rustc lays out both sides, and host-to-device copies hand a raw pointer and a byte count to the driver without ever forming a &[u8] over the padding. It is not a layout other compilers or frameworks can rely on.

   What other systems do

   • DLPack (C API) describes a dtype as (code, bits, lanes) and keeps shape and strides in elements. Sub-byte types are "packed in memory" by default, little bit-endian: (D >> (i * bits)) & mask is the i-th element. A producer can set DLPACK_FLAG_BITMASK_IS_SUBBYTE_TYPE_PADDED to say each value instead occupies a whole byte. Booleans are bits = 8 "as per common array library convention". There are codes for float4_e2m1fn, the FP6 formats, and the FP8 formats including e8m0fnu.
   • PyTorch (tensor attributes) exposes torch.float4_e2m1fn_x2, which "represents two 4-bit values packed into a byte"; shape and stride operations "operate at the byte boundary and will not unpack/repack sub-byte values". That is the same choice as our f4e2m1fnx2: the storage pair is the element, and the shape is in pairs.
   • ml_dtypes (README) takes the opposite choice for NumPy: int4, uint4, and float4_e2m1fn are stored unpacked, "padded up to a byte in memory", with the value in the low bits. That is DLPack's padded flag.
   • Apache Arrow (columnar format) bit-packs booleans eight to a byte with least-significant-bit numbering, so an Arrow boolean column is neither our layout nor DLPack's default.

   So there are three conventions for the same FP4 data (packed with the shape in logical elements, packed with the shape in byte pairs, one per byte) and two for booleans (one per byte, bit-packed), and nothing in our API says which ones we speak.

   What needs deciding

   1. Shape semantics for packed types. Keep PyTorch's model, where the byte pair is the element and the shape is in pairs, or DLPack's, where the shape is in logical elements and the storage width is a separate fact. The first is what the code does; the second is what every import and export has to translate to.
   2. Nibble order. f4e2m1fnx2::from_nibbles(low, high) puts the first value in the low nibble, which is DLPack's little bit-endian order. That should be stated as a guarantee rather than left as an implementation detail.
   3. Odd extents and padded sub-byte storage. Neither can be represented today. A DLPack import with an odd last dimension, or with the padded flag set, has no target type.
   4. Padding and layout in DeviceCopy aggregates. Decide whether DeviceCopy keeps tolerating padding, and whether the derive should require repr(C) or repr(transparent). bytemuck's Pod forbids padding; its AnyBitPattern, which Safe element casts can produce invalid bool values (Tensor::reinterpret, DeviceBuffer::cast_elem/cast_chunks) #1506 adopts for the safe casts, tolerates it, so casting into a padded struct stays allowed under either answer. The question is what layout we promise to anything that is not rustc.
   5. DTypeId::size_in_bytes. It currently means the storage element size. It should say so, and interop needs the DLPack bits value next to it.

   Proposed direction

   • Keep byte-addressable storage types as the only DTypes, as PyTorch does. f4e2m1fnx2 stays the FP4 host type, its shape stays in pairs, and logical sub-byte types stay kernel-only. Document this in dtype.rs and the book, including the low-nibble-first order.
   • Specify the DLPack mapping once and treat it as the reference for every other framework: FP4 with bits = 4 and the default packing becomes f4e2m1fnx2 with the last dimension halved, odd extents and the padded flag are rejected with a clear error until there is a type for them, bool is bits = 8, the FP8 and e8m0fnu types map one to one, and Arrow booleans must be unpacked by the caller.
   • Have #[derive(DeviceCopy)] require repr(C) or repr(transparent), keep tolerating padding, and document the copy rule: padding bytes are moved as raw bytes and never read as values. Point at Safe element casts can produce invalid bool values (Tensor::reinterpret, DeviceBuffer::cast_elem/cast_chunks) #1506 for how this interacts with safe casts.
   • Add DTypeId::bits() alongside size_in_bytes() so the DLPack dtype can be produced without a side table, and a note on each DTypeId variant stating whether it is a logical type or a packed storage type.
3. nihalpasham

   added this to the cuda oxide v0.4.0 milestone on Oct 25, 2026
4. elibol

   added a commit that references this issue on Oct 30, 2026

   cuda-core: safe element casts require AnyBitPattern; bool is DeviceCo…

   ...

   dfea116