Classifier-Free Guidance

From a classifier to no classifier

Dhariwal and Nichol introduced classifier guidance: add the gradient of an image classifier, trained on noisy images, to the diffusion model's prediction, with a scale that trades diversity for fidelity. Established It worked, but needed a second model trained specially for noisy inputs.

Ho and Salimans showed that guidance needs no classifier: jointly train a conditional and an unconditional model (in practice one network, with the condition dropped at random during training) and combine their predictions to get a similar trade-off between sample quality and diversity. Established

The rule

With prompt $c$ and guidance scale $s$:

$$\tilde\varepsilon = \hat\varepsilon(x_t) + s\,\big(\hat\varepsilon(x_t, c) - \hat\varepsilon(x_t)\big).$$

The difference $\hat\varepsilon(x_t, c) - \hat\varepsilon(x_t)$ is the direction the prompt pushes. Scaling it by $s > 1$ goes further in that direction than the model itself would. In the paper's notation this is $(1+w)\hat\varepsilon(x_t,c) - w\,\hat\varepsilon(x_t)$, so $w = s - 1$; tools usually expose $s$.

Because $\hat x_0$ is a linear function of $\hat\varepsilon$ at a fixed $x_t$, guiding the noise prediction and guiding the clean-data prediction are the same thing. The lab guides $\hat x_0$, which is easier to draw.

Tiny example (lab numbers)

The lab's prompted model is deliberately imperfect: it reads the prompt only half the time. With the prompt "ring", 25 steps and 240 samples:

For the ring, guidance mostly helps. Try "plus" and the cost appears: at scale 8 the samples crowd toward the middle of the plus, only 17% of its blobs get a sample, and fewer than half of the samples land on any shape. The guidance paper itself notes that as guidance strength grows, each conditional places probability mass farther from the other classes, and sample diversity decreases. Established

Why it can distort

Guidance is extrapolation. Where the prompted and unprompted predictions differ a lot, a large scale overshoots. The Imagen paper reports that a large guidance weight improves image–text alignment but damages fidelity, producing highly saturated and unnatural images, and introduces dynamic thresholding to allow higher weights. Established (Imagen uses the same convention as this page: a weight of 1 means no extra guidance.) A guidance scale is a dial between "typical of the data" and "unmistakably the prompt", not a quality knob that is always better higher. Interpretation It plays a role similar to a low temperature in text decoding.

Variations

• Negative prompts replace the unconditional prediction with one conditioned on what to avoid, so the extrapolation pushes away from it.
• Cost: two network calls per step (often batched together). Distillation can bake guidance into a single-call student.
• Text encoders: Imagen found that scaling its frozen text encoder improved fidelity and image–text alignment more than scaling the image diffusion model. Established Guidance amplifies whatever the text encoder understood, and cannot add what it missed.

Mini experiment

In the lab, keep "ring" and 25 steps. Set "Model reads the prompt" to 100% and compare scale 1 with scale 3. Then set it to 30%. When is guidance doing real work, and when is it only reducing diversity? Then switch to "plus" and find the smallest scale at which coverage drops below 60%.