Exact LoRA trainable-parameter counts and fine-tuning VRAM plans for Llama-style decoder models. Dependency-free Python, no network calls, no estimation, every byte constant named rather than buried.
Two settings fix the trainable parameter count: the rank r and the set of
target modules. lora_alpha, dropout, learning rate and schedule change the
model you get, not the memory the run needs — so they are not inputs here.
conda install -c freedomluckystrike lora-plan
Llama-3.1-8B, r=64, all seven projections:
from lora_plan import (ModelShape, trainable_parameters, adapter_bytes,
memory_plan, total_bytes, fits_on_card, human_bytes, GIB)
shape = ModelShape.from_config({
"num_hidden_layers": 32, "hidden_size": 4096, "intermediate_size": 14336,
"num_attention_heads": 32, "num_key_value_heads": 8,
})
t = trainable_parameters(shape, rank=64) # 167_772_160 (2.09% of 8.03B)
human_bytes(adapter_bytes(t)) # '320.00 MiB' adapter at bf16
plan = memory_plan(8_030_261_248, t, "lora16")
human_bytes(total_bytes(plan)) # '18.46 GiB'
fits_on_card(plan, 24 * GIB) # (True, 5.54 GiB spare)
human_bytes(total_bytes(memory_plan(8_030_261_248, t, "qlora4"))) # '7.36 GiB'
human_bytes(total_bytes(memory_plan(8_030_261_248, t, "full"))) # '120.66 GiB'For a linear layer of shape (in, out), LoRA adds the two factor matrices
A (r x in) and B (out x r), so r * (in + out) trainable parameters.
module_shapes() gives the seven per-block shapes and is grouped-query aware:
k_proj and v_proj are hidden_size x (num_key_value_heads * head_dim), which
on Llama-3.1-8B is 4096 x 1024, not 4096 x 4096. Getting that wrong inflates
the adapter.
| term | bytes per trained parameter |
|---|---|
| trainable weights, fp16 | 2 |
| gradients, fp16 | 2 |
| fp32 master copy | 4 |
Adam m and v |
8 (sgdm 4, sgd 0) |
The frozen base is 2 bytes per weight in lora16, and NF4 in qlora4 — 4 bits
per weight, or 4.127 with double quantisation for the quantised absmax constants.
activation_bytes defaults to 1 GiB and is an input, not a guess: nothing in
this library can know your recomputation setting or sequence length. Measure yours
and pass it in.
The constants, the fit verdicts and where they come from are published at https://ml0x.com/calculators/lora-hyperparameters.html. The test suite pins this library to that page's worked example: 167,772,160 trainable parameters, 18.46 / 7.36 / 120.66 GiB for LoRA / QLoRA / full fine-tuning, 5.54 GiB spare on a 24 GB card.
python -m unittest discover -s tests
26 tests, no dependencies.
MIT.