Vision, Speech & Reinforcement Learning

The problem

Images are grids of numbers

Chapter 4 ended with networks that learn their own features. This chapter is a branch off the main road to language models. It follows neural networks into two other kinds of data, pictures and sound, and then into a different kind of learning altogether, where nobody supplies the right answer.

Start with a photo. To a computer it is a grid of pixels, each holding three numbers for red, green and blue. A modest 224 × 224 colour image is already 150,528 numbers. Feed that into a fully connected layer and two things go wrong. The first layer needs an enormous number of weights. Worse, flattening the grid into a list throws away the one thing that matters most about images: which pixels are next to which. A fully connected network would have to learn what a vertical edge looks like separately in every corner of the picture.

The idea

One small filter, slid everywhere

The fix builds two facts about images into the layer. Useful patterns are local: an edge lives in a few neighbouring pixels. And they can appear anywhere: an ear is an ear in either corner. So use one small grid of weights, a filter, that looks at a 3 × 3 patch, and slide the same filter across every position. At each stop, multiply the patch by the filter element by element and add up the products. The grid of results is a feature map.

That sum is a dot product between the filter and one patch. The filter [-1, 0, 1] run along the pixels [0, 0, 1, 1, 1] gives [1, 1, 0]: it fires where brightness steps up and stays silent where it is flat. Try the two-dimensional version below. Select a cell of the feature map to see the nine multiplications behind it, or edit the filter yourself.

Two consequences follow from reusing one filter everywhere. The layer needs 9 weights instead of tens of thousands. And if the picture moves, the feature map moves with it: tick "Move the picture" and watch. A pattern learned in one place is recognised in every place.

The architecture

Stacking filters into a network

One layer can only ask local questions. A convolutional neural network (CNN) stacks many. Each layer applies dozens of filters, passes the results through ReLU, and every few layers pools: it keeps the strongest response in each 2 × 2 block, halving the height and width. Layer two convolves over layer one's feature maps, so it can find corners made of edges. Layer three finds arrangements of corners. With each layer, every unit sees a wider patch of the original image. A small fully connected head turns the final features into class probabilities.

1. 1

   Convolve

   Many filters, one feature map each. Filters span all channels of their input.
2. 2

   ReLU

   Without a nonlinearity, stacked convolutions collapse into one big filter.
3. 3

   Pool

   Shrink the maps. Later layers see more of the image at a lower cost.
4. 4

   Classify

   Average or flatten the last maps and apply a classifier with softmax.

Nobody hand-writes these filters. They start random and are trained by backpropagation on labelled images, like any other weights. LeCun and colleagues trained such a network end to end to read handwritten zip codes in 1989 Established, and by the late 1990s its descendants were reading cheques. The design traces back through Fukushima's Neocognitron (1980) to Hubel and Wiesel's discovery of edge-sensitive cells in the visual cortex Established. The brain was an inspiration, not a blueprint Interpretation.

The breakthrough

The ImageNet years

CNNs worked on digits for two decades before they clearly beat hand-engineered systems on real photos. What they lacked was data and compute. ImageNet (2009) supplied millions of labelled photos, and its yearly challenge became the benchmark Established. In 2012 AlexNet, a deep CNN trained on GPUs with ReLU and dropout, cut the top-5 error to 15.3%, more than 10 points ahead of the next entry Established. Chapter 4 called this the start of the deep-learning era.

Networks then grew deeper until, around twenty layers, extra depth started to make them worse to train. ResNet (2015) added a shortcut around every pair of layers, computing x + f(x), and trained networks with 152 layers Established. You will meet that same residual connection around every sublayer of a Transformer.

More than labels

Beyond one label per image

Knowing that a photo contains a cat is rarely enough. Detection draws a box around every object and names it. Segmentation labels every pixel, which is what a medical scan or a self-driving car needs. Both reuse a CNN backbone and change the head. Segmentation has to recover the resolution that pooling threw away. U-Net does it with a U-shaped network whose skip connections carry fine detail from the shrinking half to the growing half.

Then, in 2020, vision borrowed language's architecture. A Vision Transformer cuts an image into 16 × 16 patches, turns each patch into a vector, and runs an ordinary Transformer over the sequence as if the patches were words. With modest pretraining data it trailed strong CNNs; with very large datasets it matched or beat them Established. A model with fewer built-in assumptions needs more data, but can go further when it gets it Interpretation. The same theme returns with language models.

Hearing

Sound becomes a picture

Speech arrives as a waveform: air pressure measured thousands of times a second. At 16 kHz, one second is 16,000 numbers, and the wiggles hide what matters. A vowel is defined by which frequencies are loud, and a word is a pattern of frequencies changing over time.

So slice the sound into short, overlapping windows, about 25 ms each. For each window, measure how much of every frequency it contains (a Fourier transform). Stack the results side by side and you have a spectrogram: time runs left to right, pitch bottom to top, and brightness shows strength. Speech becomes an image, and everything above applies.

The lab also shows a trade-off with no way around it. A short window pins down when something happened but blurs which frequencies it had. A long window separates close frequencies but smears events in time. Real systems pick about 25 ms as a compromise that suits speech.

From sound to text

Recognising speech is a sequence problem with a catch: 300 audio frames must become 11 letters, and nobody marks which frames belong to which letter. For decades, recognisers stitched together separately built acoustic, pronunciation and language models. Around 2012, deep networks overtook the statistical acoustic models Established, in the same years they overtook vision pipelines. CTC (2006) had already removed the alignment problem. The network may output a letter or a "blank" at every frame, and training adds up every alignment that spells the right transcript.

The latest step will look familiar from Chapter 6: treat recognition as translation, from audio frames to text, with an encoder, a decoder and attention. Whisper (2022) trained such a Transformer on 680,000 hours of audio paired with transcripts from the web, and it transcribes robustly without task-specific fine-tuning Established. Speech followed the same arc as vision: hand-built stages, then one learned model, then one much larger model trained on messier data.

A different kind of learning

Learning from consequences

Everything so far, from Chapter 3 on, learned from examples: an input paired with the right answer. Many problems have no right answers to copy. Which move should a Go player make? How should a robot shift its weight? What makes a helpful reply? We cannot label every step, but we can score how things turned out.

Reinforcement learning (RL) learns from that score. An agent takes an action, the environment responds with a new state and a reward, and the loop repeats. The agent's goal is the return: the total future reward, with each step's reward multiplied by a discount γ per step of delay, so that sooner rewards count for more. With γ = 0.9, a reward of 10 three steps away is worth 0.9² × 10 = 8.1 now.

The standard formalism is the Markov decision process: states, actions, transition rules, rewards and a discount. A policy says what to do in each state. A value function says how much return to expect from there. They are connected by one recursive idea, the Bellman equation: the value here is the reward now plus the discounted value of where you land.

Learning what actions are worth

Q-learning turns the Bellman equation into a learning rule. Keep an estimate Q(s, a) for every state and action. After each step, move the estimate a fraction α toward a better-informed target: the reward just received plus γ times the best Q value from the next state.

Value flows backward from rewards, one step per update. The first time the agent reaches the +10 goal, only the final move learns anything. The next time it passes the square before it, that move learns too, and so on back toward the start. Press One move in the lab to watch a single update, or +50 episodes to watch the table fill in.

The lab also shows RL's third difficulty in action. With neutral starting values, the wandering agent usually finds the nearby +1 coin first. From then on, the path to the coin looks better than every move it has never tried, and it stops looking. Random exploration (ε) rarely strings together enough lucky moves to reach the +10 goal. Switch to an optimistic start, where every untried move looks as good as the best reward on the map. Now the agent tries each move until experience talks it down, and it finds the goal every time. Then lower γ to 0.3 and retrain: with a steep enough discount, +1 in three moves really is better than +10 in five.

Learning the policy directly

A Q table needs one entry per state and action. That is impossible for a camera image, and awkward when the action is a continuous motor torque or a whole paragraph of text. Policy-gradient methods skip the table. The policy itself is a network that outputs action probabilities. Sample an action, see how it went, and nudge the parameters so that actions that did better than expected become more probable and the rest less probable:

The baseline b, often a learned value estimate called the critic, is what turns "rewarded" into "better or worse than usual". Look closely and this is Chapter 2's cross-entropy update with a twist. Supervised learning raises the log-probability of the correct answer. A policy gradient raises the log-probability of the answer the model chose, in proportion to how well it turned out.

With neural networks as their policies and values, these methods scaled. DQN learned 49 Atari games from pixels in 2015, and AlphaGo combined policy and value networks with tree search to beat Lee Sedol in 2016 Established. AlphaGo's search tree is a direct descendant of Chapter 1's search, with learned judgement where the hand-written evaluation used to be.

Why it matters

Why it matters

This chapter sits off the main road, but it supplies three pieces the road needs.

• The recipe. Vision proved, earlier than language, that large labelled data, GPUs and learned features beat hand engineering, and that a pretrained network can be reused for new tasks.
• Convergence. Images became patches and sound became frames, and both ended up as sequences of vectors processed by Transformers. That is why modern models can take pictures and audio as easily as text (Chapter 15).
• Learning beyond imitation. Supervised learning can only copy its examples. RL optimises outcomes, so it can improve on them. When language models were fine-tuned from human preferences (RLHF), the optimiser was a policy-gradient method, PPO Established, and recent reasoning models are trained with RL on problems whose answers can be checked automatically Active research.

What came next

Back to language

The main road continues with text. Chapter 6 asks the questions this chapter asked about pixels and sound: how do you turn words into numbers that keep their meaning, and how does a model remember what came fifty words ago? Watch for familiar moves. Text is a sequence like audio, word vectors are learned features like CNN filters, and the attention mechanism that ends Chapter 6 is what the Vision Transformer and Whisper were later built on.