LoRA, QLoRA and Parameter-Efficient Fine-Tuning

Fine-tuning a giant is mostly overhead

Supervised fine-tuning changes every weight. For a model with $N$ parameters, training needs about 16 bytes per parameter for weights, gradients and Adam's states (training memory), and the result is a new $N$-parameter model. For GPT-3 175B, the LoRA authors note, each fine-tuned instance would be a full 175B-parameter model, prohibitively expensive to deploy for many tasks Established.

Earlier, Houlsby and colleagues inserted small trainable adapter layers into a frozen BERT and came within 0.4% of full fine-tuning on GLUE while adding 3.6% parameters per task Established. But the extra layers add a little computation at every step of inference.

A low-rank update

LoRA freezes the pretrained weights and injects trainable rank-decomposition matrices: the update to a weight matrix is $\Delta W = BA$, with $A$ initialized randomly and $B$ at zero, so the model starts out unchanged, and the update is scaled by $\alpha / r$ Established:

$$h = W_0 x + \frac{\alpha}{r} B A x, \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d, k).$$

Tiny example. A 4,096 × 4,096 projection has 16.8 million weights. A rank-8 update trains $8 \times (4{,}096 + 4{,}096) = 65{,}536$ numbers: 0.39% of the matrix. Adam's states are needed only for those.

Rank r8

This matrix, full fine-tuning

16,777,216

This matrix, LoRA

65,536 (0.39%)

Llama 3 8B, query + value in all 32 layers

3.4M (0.043% of 8B)

Trainable numbers are r × (d + k) per matrix. B starts at zero, so training begins from the unchanged model. Llama 3 8B’s value projection is 4,096 × 1,024 because of grouped-query attention. Counts are our arithmetic from the published architecture; Adam’s states are needed only for the trainable part.

Compared with fine-tuning GPT-3 175B with Adam, LoRA reduced the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times, while matching or beating fine-tuning quality on the models tested Established. With rank 4 applied only to the query and value projections, the checkpoint shrank from 350 GB to about 35 MB; storing 100 adapted models takes about 354 GB instead of 35 TB Established.

No extra latency, or many tasks at once

For serving, $W = W_0 + BA$ can be computed and stored, so inference runs as usual with no added latency; switching tasks means subtracting one $BA$ and adding another Established. Alternatively, keep the adapters separate and let one copy of the base model serve requests for many adapters.

QLoRA: a 4-bit base

LoRA still keeps the frozen base in 16 bits. QLoRA backpropagates through a frozen 4-bit base model into LoRA adapters, which reduced memory enough to fine-tune a 65B-parameter model on a single 48 GB GPU while preserving 16-bit fine-tuning task performance Established. It introduced the 4-bit NormalFloat (NF4) data type for normally distributed weights, double quantization of the quantization constants, and paged optimizers to absorb memory spikes Established. See quantization for why the constants matter.