Concept · Chapter 5: Vision, Speech & Reinforcement Learning
Deep RL, from Atari to AlphaGo to RLHF
Deep reinforcement learning uses neural networks as the policy and value functions, which let RL scale from toy tables to Atari, Go and, eventually, the fine-tuning of language models.
The problem
Tables cannot represent values or policies for problems with astronomically many states, such as game screens or board positions.
The solution
Replace tables with deep networks that generalise across similar states, and add stabilising tricks (replay, target networks, clipped updates, self-play).
The consequence
RL produced superhuman game players and became the tool that turns reward signals into better language-model behaviour.
You should understand first
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Probability and Distributions
- Expected Value and Variance
- Reinforcement Learning
- MDPs, Policies and Value
- Q-Learning
- Softmax
- Derivatives and Gradients
- Loss Functions
- Gradient Descent
- Policy Gradients
- Vectors
- Tensors and Shapes
- Images as Tensors
- Dot Product
- Convolution
- Features, Labels and Tasks
- Linear Regression
- Entropy
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- The Chain Rule
- The Forward Pass
- Computational Graphs and Autodiff
- Backpropagation
- Convolutional Neural Networks
- Deep RL, from Atari to AlphaGo to RLHF
Three landmarks
Atari, 2015
DQN used a CNN to estimate Q values from raw game frames. One architecture and one set of hyperparameters learned 49 Atari games from pixels and score, reaching human-level play on many of them Established.Go, 2016
AlphaGo combined a policy network (which moves look promising?) and a value network (who is winning?) with Monte Carlo tree search. It learned first from human expert games, then improved through self-play with reinforcement learning, and beat Lee Sedol 4–1 Established. The search tree is the direct descendant of Chapter 1's search: learning supplied the judgement that hand-written evaluation functions lacked.Language, 2017 →
Human preferences became a reward. Christiano and colleagues (2017) trained a reward model from human comparisons of behaviour clips and optimised a policy against it Established; InstructGPT (2022) applied the same recipe to GPT-3 Established. Since 2024, reasoning models have been trained with large-scale RL on problems whose answers can be verified, such as maths and code; DeepSeek-R1 (2025) documented one such recipe Active research.
Why RL keeps coming back
Supervised learning can only imitate: its ceiling is the quality of its examples. RL can improve on the examples, because it optimises the outcome rather than matching the demonstration. That is why RL tends to appear at the point where imitation runs out: after pretraining on human games in AlphaGo, and after pretraining on human text in language models Interpretation.
What to remember
- DQN (2015): a CNN estimates Q from Atari pixels.
- AlphaGo (2016): policy and value networks guide a tree search; trained on human games, then improved by self-play.
- RLHF (2017 onward): a reward model learned from human comparisons supplies the reward; PPO updates the policy.
- RL on checkable answers is a key ingredient of recent reasoning models (evolving).
Key papers
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu et al. · 2015 · Nature
A single deep network learned to play dozens of Atari games from raw pixels and score alone — deep learning meets reinforcement learning.
Mastering the game of Go with deep neural networks and tree search
David Silver, Aja Huang et al. · 2016 · Nature
AlphaGo combined learned intuition (neural networks) with classical search — and beat top professionals at a game long thought decades away.
How to read it: A perfect bridge between this chapter's two halves: symbolic search, guided by learned networks.
Deep reinforcement learning from human preferences
Paul Christiano, Jan Leike et al. · 2017 · NeurIPS 2017
Showed that agents can be trained from human comparisons between behaviours rather than a hand-written reward — the foundation of RLHF.
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu et al. · 2022 · NeurIPS 2022
InstructGPT: the supervised fine-tuning + reward model + RL recipe that turned GPT-3 into an instruction-following assistant, and the template for ChatGPT.
How to read it: Figure 2 is the three-step RLHF pipeline you'll meet in Chapter 10.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI et al. · 2025
An openly released reasoning model, with a detailed account of training long chains of reasoning mainly through reinforcement learning on verifiable problems.