Concept · Chapter 5: Vision, Speech & Reinforcement Learning
Convolution
A convolution slides one small grid of weights (a filter) across the input, taking a weighted sum at every position to produce a feature map.
The problem
Fully connected layers need an enormous number of weights for images and must relearn the same pattern separately at every location.
The solution
Use a small filter that looks only at a local patch, and reuse the same filter weights at every position (weight sharing).
The consequence
A layer detects a pattern anywhere with a handful of weights, and shifting the input shifts the output: the basis of every CNN.
You should understand first
Intuition: a stencil you slide
Hold a small stencil over a corner of the picture, multiply each pixel under it by the number printed on the stencil, and add up. Write that sum into the output. Move the stencil one pixel and repeat. The stencil is the filter (or kernel); the grid of sums is the feature map.
If you work with SQL, this is a rolling window aggregate in two dimensions: the same window function applied at every row, except the weights inside the window are learned.
Tiny numeric example (1-D)
Take a row of pixels that steps from dark to bright, [0, 0, 1, 1, 1], and the filter [-1, 0, 1] ("right minus left"):
Position 1
Window[0, 0, 1]: −1·0 + 0·0 + 1·1 = 1. An edge.Position 2
Window[0, 1, 1]: −1·0 + 0·1 + 1·1 = 1. Still on the edge.Position 3
Window[1, 1, 1]: −1·1 + 0·1 + 1·1 = 0. Flat, so nothing to report.
Five inputs, a 3-wide filter, three outputs: . The filter fires where brightness changes and is silent where it is flat. The 2-D vertical-edge filter in the lab is the same idea, averaged over three rows.
The equation
Output size: with input width n, kernel k, zero-padding p on each side and stride s, the output width is . With n = 16, k = 3, p = 0, s = 1, that is 14.
Try it
Try it · toy model
Slide a 3×3 filter over a tiny picture, see the nine multiplications behind every output, edit the weights, then add ReLU and pooling.
Why sharing weights is the whole point
- Few parameters. A 3×3 filter has 9 weights no matter how big the image is.
- Equivariance. Because every position uses the same weights, moving the input moves the feature map by the same amount. Tick "Move the picture" in the lab.
- Local first. Each output depends on a small neighbourhood. Stacking layers widens the view (see CNNs).
Hubel and Wiesel's recordings of edge-selective cells with small receptive fields in the visual cortex inspired this design Established, but a CNN is a loose engineering analogy to the visual cortex, not a model of it Interpretation.
Why should I care?
As a researcher
Weight sharing and locality are the textbook example of an inductive bias: structure built into a model that trades flexibility for data efficiency.
As an engineer
Convolutions still run image, audio and video pipelines on phones and edge devices, and 1-D convolutions appear inside many sequence models.
Modern systems that depend on it
- CNNs
- pooling
- U-Net segmentation
- WaveNet audio
Historical context
Before
Vision systems used hand-designed feature detectors (edge filters, gradient histograms) followed by a classifier.
After
The filters themselves became learnable weights, trained end to end by backpropagation.
Used today
In mobile and real-time vision, in the patch-embedding step of Vision Transformers, and in audio models.
What to remember
- Output = sum over a 3×3 patch of (pixel × weight), plus a bias, at every position.
- The same few weights are reused everywhere: 9 weights instead of tens of thousands.
- Shift the input and the feature map shifts with it (translation equivariance).
- A valid n×n convolution with a k×k filter gives (n − k + 1) × (n − k + 1); padding keeps the size.
- Deep-learning 'convolution' is technically cross-correlation (no kernel flip); since the kernel is learned, it does not matter.
Key papers
Receptive fields, binocular interaction and functional architecture in the cat's visual cortex
D. H. Hubel, T. N. Wiesel · 1962 · The Journal of Physiology
Neuroscience, not machine learning, but it described visual neurons that respond to oriented edges in a small patch of the visual field: the idea behind local, edge-detecting filters.
How to read it: Read the summary and look at the receptive-field figures; the physiology detail is optional for our purposes.
Backpropagation Applied to Handwritten Zip Code Recognition
Y. LeCun, B. Boser et al. · 1989 · Neural Computation
Trained a network with shared local weights end to end by backpropagation on real handwritten digits: the convolutional network as we know it.
Watch
3Blue1Brown
But what is a convolution?
A visual introduction to discrete convolution that starts with adding dice and ends with image kernels and FFTs.