guppy

Implementing a Marlin-esque w4a16 (INT4 weight, FP16 activations) matmul kernel for Blackwell (sm_120). Same waters, smaller fish.

Kernels

Kernel 1: Naive

One CUDA thread computes one output element output[m][n]. Each thread independently loops over the k dimension and MACs.

Kernel 2: Tiled

Threads in a block cooperatively load weights, activations, and scales into shared memory before computing. Threads still own one output element each, but now read shared memory instead of redundantly hitting global memory per MAC.

Kernel 3a: Register tiling

Each thread now computes a $TM \times TN$ subtile of output instead of a single element. Increases FMA per read. Fully unrolled accumulator array keeps everything in registers rather than local memory.

Kernel 3b: Vectorised Dequant

Same tiling as 3a, but the scalar unpack path is replaced with lop3 (unpacking two weights per instruction) and the MAC loop with hfma2 on packed half2 register (two FMAs per instruction). Stores partials in fp16 which are then written to a fp32 accumulator to reduce aggregate rounding error.

Resources

• Marlin paper
• GPTQ paper
• Siboehm's matmul worklog
• Aleksa Gordić's anatomy of high performance matmul kernels
• Wafer's GPU perf engineering resources
• Pranjal Shankhdhar's Outperforming cuBLAS on H100 worklog
• The Software Frontier's Mastering CUDA and HPC Series
• MLC's Modern GPU Programming For MLSys