Skip to content
Road to Intelligence

Concept · Chapter 5: Vision, Speech & Reinforcement Learning

Convolutional Neural Networks

Must knowKnow well18 minDifficulty

A CNN stacks convolution, nonlinearity and pooling layers so that early layers detect edges and later layers combine them into parts and objects.

The problem

One layer of filters only sees small patches; recognising an object needs patterns built from patterns across a large region.

The solution

Stack many convolutional layers with nonlinear activations and occasional downsampling, then finish with a classifier, and train all filters by backpropagation.

The consequence

With enough labelled data and compute, CNNs learned visual features that beat hand-designed pipelines, beginning with ImageNet in 2012.

You should understand first

  1. Vectors
  2. Tensors and Shapes
  3. Images as Tensors
  4. Dot Product
  5. Convolution
  6. The Turing Test
  7. Symbolic AI
  8. Logic and Rules
  9. Expert Systems
  10. Knowledge Representation
  11. The Knowledge-Acquisition Bottleneck
  12. From Rules to Learning
  13. Supervised, Unsupervised and Self-Supervised Learning
  14. Features, Labels and Tasks
  15. Loss Functions
  16. Derivatives and Gradients
  17. Gradient Descent
  18. Linear Regression
  19. Probability and Distributions
  20. Entropy
  21. Softmax
  22. Cross-Entropy Loss
  23. Logistic Regression
  24. The Perceptron
  25. Activation Functions
  26. The Artificial Neuron
  27. Matrix Multiplication
  28. Multilayer Perceptron (MLP)
  29. The Chain Rule
  30. The Forward Pass
  31. Computational Graphs and Autodiff
  32. Backpropagation
  33. Convolutional Neural Networks

Intuition: patterns of patterns

One convolutional layer answers local questions: is there a vertical edge here? A curve? A patch of red? The next layer convolves over those answers, not over raw pixels, so it can ask: are there two edges meeting at a corner? A third layer can look for arrangements of corners. Each layer sees a wider region of the original image than the last.

The standard recipe

  1. Convolve

    A layer applies F filters, producing F feature maps (channels). Each filter spans all channels of its input.

  2. Nonlinearity

    Apply ReLU. Without it, stacked convolutions collapse into one big linear filter (the same argument as for MLPs).

  3. Downsample

    Every few layers, pool or use a stride of 2 to halve the height and width, usually while increasing the number of channels.

  4. Classify

    Flatten or average the final feature maps and pass them through a small fully connected head with softmax.

Receptive field: tiny numeric example

A 3×3 filter sees a 3×3 patch. Stack a second 3×3 layer: each of its outputs combines a 3×3 block of first-layer outputs, each of which saw 3×3 pixels, so it sees 5×5 pixels. A third layer sees 7×7. In general, LL stacked 3×3 layers with stride 1 see (2L+1)×(2L+1)(2L+1)\times(2L+1) pixels, and every downsampling step makes later growth faster. Two 3×3 layers use 2×9=182\times 9 = 18 weights per channel pair versus 25 for one 5×5 layer, with an extra nonlinearity in between: the design logic behind the VGG networks of 2014.

Training

Nothing new is needed: the loss is cross-entropy, the gradients come from backpropagation, and each filter weight's gradient is summed over every position where it was used. LeCun and colleagues trained such a network end to end on handwritten zip codes in 1989 Established, and descendants read cheques in deployed systems by the late 1990s.

Visualisations of trained CNNs show early filters resembling oriented edges and colour blobs, and deeper units responding to textures and object parts Established. Calling this a hierarchy of concepts is a useful summary, though individual units rarely map cleanly onto human-nameable ideas Interpretation.

Why should I care?

As a researcher

CNNs are the clearest case of representation learning and of how inductive bias interacts with data scale, the comparison the Vision Transformer later made explicit.

As an engineer

A pretrained CNN backbone (ResNet, EfficientNet and relatives) is still a strong, cheap default for image tasks and transfer learning.

Modern systems that depend on it

  • ResNet
  • object detectors
  • U-Net
  • the image encoders of many multimodal systems

Historical context

Before

Hand-crafted features such as SIFT or HOG, followed by a separate classifier such as an SVM.

After

End-to-end learned feature hierarchies: pixels in, labels out, every filter trained from data.

Used today

Widely used in production vision, especially on devices; Transformers dominate the largest vision models, often with convolutional ideas mixed back in.

What to remember

  • Pattern: [conv → ReLU] × a few → pool, repeated; then a small fully connected head.
  • Each layer outputs many channels, one feature map per filter.
  • Stacking widens the receptive field: two 3×3 layers see 5×5, three see 7×7.
  • Early layers learn edge- and colour-like filters; deeper layers respond to larger, more object-like patterns.
  • LeNet (1989–1998) read digits; AlexNet (2012) won ImageNet; ResNet (2015) went to 152 layers.

Key papers

Essential

Gradient-based learning applied to document recognition

Yann LeCun, Léon Bottou et al. · 1998 · Proceedings of the IEEE

The LeNet paper: convolutional networks trained end-to-end with gradient descent for handwriting recognition, deployed commercially for reading cheques.

How to read it: Long (46 pages). Sections I–II explain why learned features beat hand-designed ones — the heart of Chapter 4.

~1 h 30 min readdoi:10.1109/5.726791✓ verified 2026-09-26
Important

Backpropagation Applied to Handwritten Zip Code Recognition

Y. LeCun, B. Boser et al. · 1989 · Neural Computation

Trained a network with shared local weights end to end by backpropagation on real handwritten digits: the convolutional network as we know it.

~30 min readdoi:10.1162/neco.1989.1.4.541✓ verified 2026-09-26

Watch