Skip to content
Road to Intelligence

Concept · Chapter 5: Vision, Speech & Reinforcement Learning

Exploration vs Exploitation

Must knowUnderstand10 minDifficulty

An agent must balance exploiting the best action it knows with exploring actions that might turn out better; too little exploration locks it into a mediocre habit.

The problem

An agent only learns about the actions it tries, so a greedy agent can settle on the first reward it finds and never discover a better one.

The solution

Deliberately try uncertain actions: take random moves some of the time (ε-greedy), or start with optimistic estimates so that untried actions look attractive.

The consequence

Learning is slower in the short run but finds better behaviour; how to explore efficiently is still an open research question in large problems.

You should understand first

  1. The Turing Test
  2. Symbolic AI
  3. Logic and Rules
  4. Expert Systems
  5. Knowledge Representation
  6. The Knowledge-Acquisition Bottleneck
  7. From Rules to Learning
  8. Supervised, Unsupervised and Self-Supervised Learning
  9. Probability and Distributions
  10. Expected Value and Variance
  11. Reinforcement Learning
  12. MDPs, Policies and Value
  13. Q-Learning
  14. Exploration vs Exploitation

The restaurant problem

Your favourite restaurant is good. The new place across the road might be better, or worse. Always going to the favourite (exploiting) guarantees a decent meal and guarantees you never learn about the new one. Trying the new place (exploring) risks a bad dinner for the chance of a better habit. The simplest version, choosing among slot machines with unknown payouts, is the multi-armed bandit problem.

Two simple strategies

  • ε-greedy. With probability ε, take a random action; otherwise take the best-looking one. Easy, and enough in many small problems, but the random moves are undirected: to find a reward five steps away, several lucky random moves must line up.
  • Optimistic initial values. Start every estimate above anything achievable. Every untried action then looks wonderful, the agent tries it, is "disappointed", and moves on to the next untried one. Exploration becomes systematic, and fades on its own as estimates come down to reality.

What the lab shows

In the gridworld, a +1 coin sits three moves from the start and a +10 goal five moves away. With all values starting at 0, the first reward the wandering agent stumbles on is usually the coin; after that the path to the coin looks better than every unexplored move, and small ε rarely strings together enough random moves to find the goal. In this site's unit tests, with the lab's default settings, a neutral start ends up preferring the coin in at least 35 of 50 random seeds, while an optimistic start finds the shortest route to the goal in all 50 within 50 episodes. That is a toy result about one small map, not a general law.

What to remember

  • Exploit: pick the best-looking action. Explore: try something else to learn more.
  • ε-greedy: with probability ε take a random action; simple but undirected.
  • Optimistic start: initial values above anything achievable make the agent try everything once.
  • In the lab, a neutral start settles for the +1 coin in most runs; an optimistic start finds the +10 goal every time.

Key papers

Essential

Reinforcement Learning: An Introduction (2nd edition)

Richard S. Sutton, Andrew G. Barto · 2018 · MIT Press

The standard textbook, free online from the authors. Everything in this chapter's RL half (MDPs, value functions, Q-learning, exploration, policy gradients, actor–critic) is developed carefully there.

How to read it: Chapters 1, 3 and 6 cover the core: the problem, MDPs, and temporal-difference learning including Q-learning. Chapter 13 is policy gradients.

~10 h read✓ verified 2026-09-26

Watch