CLIP and Image–Text Embeddings

Pick the partner out of a line-up

Contrastive learning in Chapter 12 trained a text encoder on question–passage pairs. CLIP makes the two sides different modalities. Take a batch of $B$ pictures with their captions. Encode every picture with an image encoder and every caption with a text encoder, normalize each vector, and fill a $B \times B$ grid with cosine similarities $s_{ij}$. Training pushes the diagonal (true pairs) up and everything else down:

$$\mathcal{L} = \frac{1}{2B}\sum_{i=1}^{B}\left[-\log\frac{e^{s_{ii}/\tau}}{\sum_j e^{s_{ij}/\tau}} \;-\; \log\frac{e^{s_{ii}/\tau}}{\sum_j e^{s_{ji}/\tau}}\right]$$

The first term asks each picture to find its caption; the second asks each caption to find its picture. The temperature $\tau$ sharpens the softmax. CLIP trained on 400 million image–text pairs collected from the internet, with a minibatch of 32,768 and a learnable temperature initialized to the equivalent of 0.07. Established A bigger batch means more wrong answers to beat in every row, which is part of why contrastive training favours huge batches.

Labels written at test time

Once trained, classification needs no classifier. To sort a photo into dog, cat or bird, embed the texts "a photo of a dog", "a photo of a cat" and "a photo of a bird", and pick the one with the highest similarity to the photo's embedding. The label set is just text, chosen after training.

The CLIP paper reports matching the original ResNet-50's ImageNet accuracy zero-shot, without using any of the 1.28 million labelled training examples that model was trained on. Established The same paper benchmarks over 30 existing datasets, and the model transfers non-trivially to most of them but not uniformly. Established

Tiny example (lab numbers)

The lab trains a toy CLIP in your browser: 12 × 12 pictures of one coloured shape, captions like "a red circle", a small image network and a bag-of-words text encoder. Four colour–shape combinations are never shown in training, though each colour and each shape appears with other partners.

After 400 batches on the default seed: pictures of seen combinations get the right caption out of 16 about 93% of the time, never-seen combinations about 75%. Asked to choose among the bare words "red", "green", "blue", "yellow", which never appeared alone in training, it is right on every test picture. Across 12 seeds the never-seen score ranges from 55% to 90%: what a model learns about a concept it has seen only in some combinations depends on the run.

What the shared space cannot see

The lab's text encoder adds word vectors, so "a red circle and a blue square" and "a blue circle and a red square" get exactly the same vector. Real CLIP text encoders are Transformers and could see order, yet the problem survives. Yuksekgonul and colleagues tested vision-language models on more than 50,000 cases and found poor relational understanding, mistakes in linking objects to their attributes, and a severe lack of sensitivity to word order. Established Their explanation is the training signal: retrieval on typical datasets succeeds without order information, so the contrastive loss puts little pressure on encoding it. Captions that differ only in order, used as hard negatives, supply that pressure. Interpretation

Other limits follow from the data: concepts absent from the web pairs, small text in images, and the social biases of web captions all carry into the embedding.

Where it shows up

• Vision-language models often use a CLIP-style image encoder as their eyes (vision-language models).
• Image generators condition on text through such embeddings, and DALL·E 2 generated images from CLIP image embeddings (latent diffusion).
• Evaluation: CLIPScore rates how well an image matches text by CLIP similarity (multimodal evaluation).
• Search: the same nearest-neighbour index you built for text in Chapter 12 works for images.

Mini experiment

In the lab, train at τ = 0.1, then retrain at τ = 0.05 and τ = 0.3 and compare the grid's diagonal and the never-seen score. Then choose "Shape words" and find a never-seen combination the model gets wrong. Is the error about colour or about shape, and why might a picture-encoder that saw triangles only in three colours struggle with the fourth?