Quantization

A coarser ruler

A 16-bit number can take about 65,000 values. A 4-bit integer can take 16. Quantization picks a grid of levels for a group of weights and rounds each weight to the nearest level.

The simplest scheme, absmax, makes the grid symmetric around zero and just wide enough for the largest weight in the group:

$$s = \frac{\max_i |w_i|}{2^{b-1} - 1}, \qquad q_i = \text{round}\!\left(\frac{w_i}{s}\right), \qquad \hat w_i = q_i \, s .$$

Tiny example. Four weights $[0.12, -0.50, 0.31, 0.04]$ in 4 bits: the integer levels run from −7 to 7, so $s = 0.50 / 7 \approx 0.0714$. Dividing and rounding gives $q = [2, -7, 4, 1]$, which decode to $[0.143, -0.500, 0.286, 0.071]$. Each weight now needs 4 bits plus a share of the scale.

Why one large number is a problem

The scale is set by the largest value. If one weight in a group is 40 times larger than the others, the grid stretches to fit it, and almost every ordinary weight lies closer to zero than to the first level: it rounds to zero.

Dettmers and colleagues found that large language models develop rare but systematic outlier features with very large magnitudes, which ruin quantization precision once they appear in all Transformer layers at about 6.7B parameters Established. LLM.int8() uses separate scaling constants for each inner product in a matrix multiplication and keeps the outlier dimensions in a 16-bit multiplication while more than 99.9% of values are multiplied in 8 bits; with it, models of up to 175B parameters ran in 8 bits without performance degradation Established.

The general fix is smaller groups: one scale per block of, say, 64 weights confines an outlier's damage to its own block. Scales cost memory: with 32-bit scales and blocks of 64, the scales add 32/64 = 0.5 bits per parameter, which QLoRA's double quantization reduces to about 0.127 Established.

Rounding smarter

Rounding each weight to its nearest level independently ignores that errors add up in a layer's output.

• GPTQ quantizes weights a block of columns at a time and updates the remaining weights to compensate for the rounding error, using approximate second-order information; it quantized 175B-parameter models to 3–4 bits in about four GPU hours with negligible accuracy loss Established.
• AWQ observes that protecting about 1% of salient weights, identified from the activations rather than the weights, greatly reduces quantization error, and protects them by scaling those channels up before quantizing Established.

Floating point with fewer bits

Integers are not the only option. FP8 comes in two encodings, E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 and 2), and matched 16-bit training quality on the models tested, up to 175B parameters Established. Meta served Llama 3 405B with FP8 quantization of most parameters and activations in the feed-forward layers, leaving the self-attention layers unquantized, and reported up to 50% higher throughput in the prefill stage Established.

What it buys, and what it doesn't

A decode step reads every weight (prefill and decode), so halving the bytes per weight roughly halves that part of the step, if the hardware has fast kernels for the format. Memory falls in step: Llama 3 70B needs 140 GB in 16 bits, 70 GB in 8 bits and about 39 GB at 4 bits with 64-weight groups.