Concept · Chapter 5: Vision, Speech & Reinforcement Learning
Vision Transformer (ViT)
A Vision Transformer cuts an image into 16×16 patches, turns each into a token vector, and runs a standard Transformer over the sequence.
The problem
Transformers had taken over language, but vision still depended on convolution's built-in locality.
The solution
Treat patches as tokens: flatten and linearly project each patch, add position embeddings, and let self-attention relate every patch to every other.
The consequence
With enough pretraining data, one architecture now serves text and images, which made multimodal models much simpler to build.
You should understand first
- Vectors
- Tensors and Shapes
- Images as Tensors
- Dot Product
- Convolution
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- The Chain Rule
- The Forward Pass
- Computational Graphs and Autodiff
- Backpropagation
- Convolutional Neural Networks
- Embeddings
- Attention
- Self-Attention
- Vision Transformer (ViT)
An image as a sentence
Split a 224×224 image into non-overlapping 16×16 patches: 14 across, 14 down, 196 patches. Flatten each patch's 16 × 16 × 3 = 768 numbers and multiply by a learned matrix to get a token vector. Add a learned position embedding so the model knows where each patch came from, prepend a special classification token, and feed the 197 vectors to an ordinary Transformer encoder. That is the whole idea.
This page belongs to Chapter 5 but depends on Chapter 7: if self-attention is new, read that first and come back.
The trade-off
A CNN assumes locality from the first layer. A ViT does not: any patch can attend to any other from layer one, and it must learn from data which patches matter to each other. The ViT paper found that with mid-sized pretraining data strong CNNs did better, while with very large pretraining datasets the ViT matched or exceeded them at lower training cost Established. This is often summarised as "weaker inductive bias needs more data but scales further" Interpretation, a theme that recurs throughout the LLM chapters.
What to remember
- 224×224 image, 16×16 patches → 14 × 14 = 196 tokens.
- Patch embedding is a linear layer; equivalently, a 16×16 convolution with stride 16.
- Less built-in structure than a CNN: it needs more data, and catches up or wins when data is plentiful.
Key papers
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer et al. · 2020 · ICLR 2021
Showed that a nearly unmodified Transformer, reading an image as a sequence of patches, can match strong CNNs when pretrained on enough data. Vision and language began to share one architecture.
How to read it: The key result is the comparison across pretraining dataset sizes: with less data CNNs win, with more the ViT catches up.