Concept · Chapter 5: Vision, Speech & Reinforcement Learning
Detection and Segmentation
Detection finds and boxes every object in an image; segmentation labels every pixel.
The problem
Real applications need to know where things are and how many, not just one label for the whole picture.
The solution
Reuse a CNN backbone and add heads that predict boxes and classes (detection) or an upsampling decoder that predicts a class per pixel (segmentation).
The consequence
Vision moved from tagging photos to measuring scenes: self-driving perception, medical imaging, photo editing.
You should understand first
- Vectors
- Tensors and Shapes
- Images as Tensors
- Dot Product
- Convolution
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- The Chain Rule
- The Forward Pass
- Computational Graphs and Autodiff
- Backpropagation
- Convolutional Neural Networks
- Pooling and Downsampling
- Detection and Segmentation
Three questions about one image
| Task | Output | Example |
|---|---|---|
| Classification | one label | "cat" |
| Detection | a box + label per object | "cat at (40, 60, 180, 200); dog at …" |
| Segmentation | a label for every pixel | a cat-shaped mask |
Detection
A detector must output a variable number of boxes. Two-stage detectors such as Faster R-CNN (2015) first propose candidate regions with a small network that shares the backbone's feature maps, then classify and refine each proposal Established. One-stage detectors (YOLO, SSD and successors) predict boxes and classes directly on a grid of positions, trading a little accuracy for speed. Both are trained with a classification loss plus a regression loss on box coordinates.
Segmentation
Pooling threw away resolution; segmentation needs it back. U-Net (2015) pairs a contracting encoder with an expanding decoder and copies high-resolution encoder features across to the matching decoder stage Established. The encoder works out what is in the image; the skip connections remember where. The same U shape later became the denoising network inside many diffusion image generators (Chapter 15).
What to remember
- Classification: one label per image. Detection: a box and label per object. Segmentation: a label per pixel.
- Two-stage detectors (Faster R-CNN) propose regions then classify them; one-stage detectors (YOLO, SSD) predict boxes directly.
- U-Net: shrink to understand, grow back to localise, with skip connections carrying fine detail.
Key papers
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Shaoqing Ren, Kaiming He et al. · 2015 · NeurIPS 2015
A standard two-stage detector: one CNN proposes candidate boxes and the same features classify and refine them.
U-Net: Convolutional Networks for Biomedical Image Segmentation
Olaf Ronneberger, Philipp Fischer, Thomas Brox · 2015 · MICCAI 2015
A labelling for every pixel: an encoder that shrinks the image and a decoder that grows it back, with skip connections carrying fine detail across. The U shape later became the backbone of many diffusion image generators.