Diffusion Models

Creating noise is easy

Take a clean data point $x_0$ (an image, or in the lab a point in 2-D) and mix it with Gaussian noise $\varepsilon$:

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon, \qquad \varepsilon \sim \mathcal N(0, I).$$

The schedule $\bar\alpha_t$ falls from 1 (clean) to nearly 0 (pure noise) as $t$ goes from 0 to 1. There is nothing to learn here: any noise level can be produced in one line. The idea of learning to reverse a gradual noising process comes from Sohl-Dickstein and colleagues in 2015. Established

Learning to undo it

Train a network $\varepsilon_\theta(x_t, t)$ to recover the noise that was added:

$$\mathcal L = \mathbb E_{x_0,\,\varepsilon,\,t}\,\big\|\varepsilon - \varepsilon_\theta(x_t, t)\big\|^2 .$$

That is ordinary supervised regression with labels you make yourself, which is why diffusion trains stably. DDPM (Ho, Jain and Abbeel, 2020) used this simplified objective with T = 1000 noise levels and reached an FID of 3.17 on CIFAR-10. Established Predicting the noise is equivalent to predicting the clean data: $\hat x_0 = (x_t - \sqrt{1-\bar\alpha_t}\,\hat\varepsilon)/\sqrt{\bar\alpha_t}$.

What the best denoiser does. With squared error, the best possible prediction is an average: $\hat x_0 = \mathbb E[x_0 \mid x_t]$, the mean of all clean data that could have produced this noisy point. Near the data that average is sharp. From pure noise, every clean point is equally plausible, so the best guess is the average of the whole dataset, which is usually not a valid sample at all. This one fact explains why generation needs many steps.

Generating step by step

Start with pure noise and repeat: predict $\hat x_0$, then move to a slightly lower noise level, keeping the part of $x_t$ that the prediction says was noise:

$$x_{t'} = \sqrt{\bar\alpha_{t'}}\,\hat x_0 + \sqrt{1-\bar\alpha_{t'}-\sigma^2}\;\hat\varepsilon + \sigma z .$$

With $\sigma = 0$ this is the deterministic DDIM sampler; with the DDPM choice of $\sigma$ it adds fresh noise $z$ at each step. DDIM uses the same trained model as DDPM and produced high-quality samples 10× to 50× faster in wall-clock time in its experiments. Established Each early step commits only to coarse structure; later steps, at lower noise, fill in detail.

Tiny example (lab numbers)

The lab uses a known two-dimensional distribution (points on a ring, a plus and four corner clusters), so the ideal denoiser can be computed exactly, with no network:

• 1 step: every one of the 240 samples lands on the same point, the average of all the data, at the centre, which lies on no shape.
• 5 steps: fewer than half the samples are on a shape; the rest sit between shapes, where averages of nearby possibilities fall.
• 25 steps: 96% of samples are on a shape.

The lab's denoiser is perfect, so its errors come only from taking too few steps. A real network adds its own approximation errors on top.

Relatives and practicalities

• Score-based view. Song and colleagues showed that diffusion and score-based models are discretizations of one stochastic differential equation, with an equivalent deterministic ODE. Established
• Flow matching trains a network to predict a velocity that carries noise to data along chosen paths; diffusion paths are one special case.
• Fewer steps. Better samplers, distillation into few-step students, and consistency-style training all attack the cost of many network calls.
• Memorization. Carlini and colleagues extracted over a thousand training images from state-of-the-art diffusion models, with duplicated training images most at risk. Established The lab shows the extreme case in miniature: its exact denoiser reproduces its known distribution perfectly and can never produce anything outside it.

Mini experiment

In the lab, set the prompt to "No prompt" and the steps to 1, 2, 5, 10 and 25, pressing "Skip to the end" each time. Record the share on a shape. Then switch to DDPM at 25 steps. Explain why one step gives a single dot at the centre, and why the share on a shape rises with steps even though the denoiser never changes.