Skip to content
Road to Intelligence

Concept · Chapter 5: Vision, Speech & Reinforcement Learning

Reinforcement Learning

Must knowKnow well16 minDifficulty

Reinforcement learning trains an agent to choose actions that maximise the total reward it collects over time, learning from the consequences of its own behaviour instead of from labelled answers.

The problem

For many tasks (playing a game, controlling a robot, holding a conversation) nobody can supply the correct action at every step, but we can score how things turned out.

The solution

Let an agent act in an environment, observe states and rewards, and adjust its behaviour to increase the expected discounted sum of future rewards.

The consequence

Machines can discover strategies no one demonstrated, but learning is slower and less stable than supervised learning, and the reward must be designed with care.

You should understand first

  1. The Turing Test
  2. Symbolic AI
  3. Logic and Rules
  4. Expert Systems
  5. Knowledge Representation
  6. The Knowledge-Acquisition Bottleneck
  7. From Rules to Learning
  8. Supervised, Unsupervised and Self-Supervised Learning
  9. Probability and Distributions
  10. Expected Value and Variance
  11. Reinforcement Learning

Intuition: learning a game with only the score

Imagine learning a video game where nobody explains the rules and the only feedback is the score. You try things, notice what raises the score, and do more of that. That is reinforcement learning: an agent acts in an environment; after each action it sees a new state and a numeric reward.

How it differs from supervised learning

SupervisedReinforcement
Feedbackthe correct answer for each inputa score for what happened
Timingimmediateoften delayed by many steps
Dataa fixed datasetproduced by the agent's own choices
Main riskoverfittingnever discovering the good behaviour at all

A move early in a chess game may win or lose it forty moves later. Deciding which of many earlier actions deserves the credit is the credit assignment problem.

The return: tiny numeric example

The agent maximises the return, the sum of future rewards, each discounted by a factor γ\gamma per step of delay:

Gt=rt+1+γ rt+2+γ2 rt+3+⋯G_t = r_{t+1} + \gamma\, r_{t+2} + \gamma^2\, r_{t+3} + \cdots

With rewards 0, 0, 10 and γ=0.9\gamma = 0.9: G=0+0.9⋅0+0.81⋅10=8.1G = 0 + 0.9\cdot 0 + 0.81\cdot 10 = 8.1. A reward of 10 three steps away is worth 8.1 now. With γ=0.3\gamma = 0.3 it is worth only 0.9, less than a reward of 1 right away. The lab below shows this choice flipping.

Try it

Try it · toy model

Learn from Consequences

Watch Q-learning fill in a value table move by move, and see an agent settle for a small reward until optimism or a gentler discount changes its mind.

Know well10 min

Where it appears in AI

Samuel's checkers player (1959) already improved from self-play Established; DQN learned Atari games from pixels (2015) and AlphaGo combined learned networks with search to beat a top professional at Go (2016) Established. Since 2022 the most influential use has been fine-tuning language models: InstructGPT used RL against a reward model trained on human preferences Established, and reasoning models are trained with RL on problems whose answers can be checked automatically Active research (Chapter 14).

Why should I care?

As a researcher

RL is the framework for sequential decisions under uncertainty, and the bridge from prediction to action that RLHF and reasoning models rely on.

As an engineer

Recommendation, bidding, robotics and LLM fine-tuning all have RL-shaped loops; knowing its failure modes (reward hacking, instability) saves projects.

Modern systems that depend on it

  • Q-learning
  • policy gradients
  • AlphaGo
  • RLHF
  • reasoning models

Historical context

Before

Supervised learning needs a correct label for every input; hand-written game players and controllers needed an expert to design the strategy.

After

An agent can improve from a scalar score alone, including beyond the level of any demonstrator.

Used today

Games and robotics, and, most visibly, fine-tuning language models from human preferences and from checkable answers (Chapters 10 and 14).

What to remember

  • The loop: observe state → choose action → receive reward and next state → repeat.
  • Goal: maximise the return G = r₁ + γr₂ + γ²r₃ + … (γ < 1 makes sooner reward count more).
  • No correct answers, only scores; rewards can be delayed, so credit assignment is hard.
  • The agent's own choices decide what data it sees: it must explore.
  • Supervised learning learns from examples; RL learns from consequences.

Key papers

Essential

Reinforcement Learning: An Introduction (2nd edition)

Richard S. Sutton, Andrew G. Barto · 2018 · MIT Press

The standard textbook, free online from the authors. Everything in this chapter's RL half (MDPs, value functions, Q-learning, exploration, policy gradients, actor–critic) is developed carefully there.

How to read it: Chapters 1, 3 and 6 cover the core: the problem, MDPs, and temporal-difference learning including Q-learning. Chapter 13 is policy gradients.

~10 h read✓ verified 2026-09-26

Watch