Concept · Chapter 5: Vision, Speech & Reinforcement Learning
Actor–Critic Methods
Actor–critic methods train two parts together: an actor (the policy) that chooses actions and a critic (a value estimate) that judges them, giving the actor a lower-noise learning signal.
The problem
Pure policy gradients wait for whole-episode returns and are very noisy; pure value methods struggle with large or continuous action spaces.
The solution
Learn a value function alongside the policy and use it as the baseline, so the actor is updated by the advantage: how much better the action was than the critic expected.
The consequence
Faster, steadier learning; most modern deep RL algorithms, including PPO as usually implemented, are actor–critic.
You should understand first
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Probability and Distributions
- Expected Value and Variance
- Reinforcement Learning
- MDPs, Policies and Value
- Softmax
- Derivatives and Gradients
- Loss Functions
- Gradient Descent
- Policy Gradients
- Q-Learning
- Actor–Critic Methods
A player and a coach
The actor plays. The critic watches each move and says "that went better than I expected" or "worse". The actor uses that judgement, rather than waiting for the final score, to decide what to reinforce. The critic improves its predictions from the same experience.
The signal
After a step from to with reward , the critic's TD error doubles as an estimate of the advantage:
If , the action beat expectations, so the actor raises its probability, exactly as in the policy-gradient update with replaced by . The critic, meanwhile, moves toward , the same temporal-difference idea as Q-learning.
Using the critic's one-step estimate instead of the full noisy return trades a little bias for a large reduction in variance. A3C (2016) ran many actors in parallel updating one shared actor–critic network Established, and PPO is typically implemented with a learned value function as its critic Established. In RLHF for language models, the critic predicts the reward-model score a partial response will end up with.
What to remember
- Actor: π(a | s), decides. Critic: V(s), predicts the return.
- Advantage ≈ r + γV(s′) − V(s): better or worse than the critic expected.
- The critic is trained like Q-learning's TD update; the actor like a policy gradient.
Key papers
Asynchronous Methods for Deep Reinforcement Learning
Volodymyr Mnih, Adrià Puigdomènech Badia et al. · 2016 · ICML 2016
A3C: a widely used deep actor–critic method, with many parallel actors sharing one network.
Proximal Policy Optimization Algorithms
John Schulman, Filip Wolski et al. · 2017
PPO: a simple, robust actor–critic policy-gradient method. It became the default RL algorithm in many labs and was the optimiser in InstructGPT-style RLHF.