Latent Diffusion and Text-to-Image

Compress first, then diffuse

Rombach and colleagues observed that pixel-space diffusion models often consume hundreds of GPU days to train, and moved diffusion into the latent space of a pretrained autoencoder. Established The autoencoder handles what they call perceptual compression (textures and pixel detail that a decoder can restore); the diffusion model spends its capacity on semantic content: what is in the picture and where.

The paper compared spatial downsampling factors f from 1 (pixels) to 32 and found small factors trained slowly while large ones lost fidelity; factors of 4 to 8 worked best. Established With f = 8, a 512 × 512 image becomes a 64 × 64 grid of latent positions, 64 times fewer than its pixels.

Text as a condition

A text encoder (CLIP's, or a large language model's encoder) turns the prompt into a sequence of token embeddings. Cross-attention layers inside the denoiser let every latent position attend to those embeddings at every step, the same attention as in Chapter 7 with queries from the image and keys and values from the text. Latent diffusion introduced these cross-attention layers so that text, bounding boxes or other inputs could condition generation. Established Classifier-free guidance then strengthens the prompt's pull.

Other routes exist. DALL·E 2 first generated a CLIP image embedding from the caption, then decoded an image from that embedding with a diffusion model. Established

From U-Nets to Transformers

Early latent diffusion used a convolutional U-Net as the denoiser. Peebles and Xie's Diffusion Transformer (DiT) runs a Transformer over patches of the latent and found that models with more compute per forward pass consistently reached lower FID. Established That turned image generation into another place where the scaling habits of LLMs apply, and it is the architecture OpenAI's Sora report names for video.

Editing

Generation from noise is one use. To edit an existing image:

• Partial noising. SDEdit adds noise to an input image or sketch and denoises it with the diffusion model; how much noise you add balances faithfulness to the input against realism. Established
• Inpainting. Denoise only a masked region while keeping the rest fixed to the (noised) original.
• Instruction editing. InstructPix2Pix trained a diffusion model on edit examples generated by GPT-3 and Stable Diffusion, so it edits from a written instruction in one forward pass per step, without per-image fine-tuning. Established

Tiny example

Several of the latent diffusion paper's own configurations encode a 256 × 256 RGB image (196,608 numbers) with f = 8 into a latent of shape 32 × 32 × 4 (4,096 numbers): 48 times fewer values for the denoiser to process at every step. Fifty denoising steps with guidance are about 100 denoiser calls on that small latent, plus a single decoder call at the end to return to pixels.

Mini experiment

Using the SDEdit idea in the diffusion lab's terms: if you started the reverse process from a point already on the plus, noised to 30% instead of 100%, would a "ring" prompt be able to move it to the ring? What does that suggest about the noise level as an editing-strength knob?