Concept · Chapter 5: Vision, Speech & Reinforcement Learning
Policy Gradients
Policy-gradient methods adjust the parameters of a policy directly, making actions that led to higher-than-expected reward more probable and the rest less probable.
The problem
Value-based methods need a max over actions, which is awkward for huge or continuous action spaces (robot torques, whole sentences), and they only indirectly produce a policy.
The solution
Parameterise the policy as a network that outputs action probabilities and follow the gradient of expected return: reward (minus a baseline) times the gradient of the log-probability of the action taken.
The consequence
Any differentiable policy can be trained from reward, including a language model generating text: this is the machinery underneath RLHF and RL-trained reasoning models.
You should understand first
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Probability and Distributions
- Expected Value and Variance
- Reinforcement Learning
- MDPs, Policies and Value
- Softmax
- Derivatives and Gradients
- Loss Functions
- Gradient Descent
- Policy Gradients
Intuition: do more of what worked
Sample an action from the current policy. If the outcome was better than you expected, nudge the parameters so that action becomes more likely in that state; if worse, less likely. No value table and no max over actions are required: the policy network is the thing being trained.
Tiny numeric example
A policy chooses between two actions with a softmax over two logits, both starting at 0, so each has probability 0.5.
Act
Sample action A. It earns reward 1. The baseline (the average reward so far) is 0.4, so the advantage is 1 − 0.4 = 0.6.Gradient of log-probability
For a softmax, the gradient of log p(A) is 1 − p(A) = 0.5 for A's logit and −p(B) = −0.5 for B's logit.Update
With step size 1: logits become 0 + 0.6 × 0.5 = 0.3 and 0 − 0.6 × 0.5 = −0.3. Now p(A) = 1 / (1 + e^(−0.6)) ≈ 0.646.
If the reward had been 0 (advantage −0.4), the same arithmetic would have made A less likely. The baseline is what turns "rewarded" into "better or worse than usual".
The equation (REINFORCE)
Williams introduced this estimator (REINFORCE) in 1992 Established. Compare it with supervised learning: if the "sampled" action is always the correct label and its reward is always 1, the update is exactly the cross-entropy gradient. Policy gradients generalise "increase the log-probability of the right answer" to "increase it in proportion to how good the answer turned out to be".
Making it stable: PPO
Raw policy gradients are noisy, and one large step can wreck a good policy. Proximal Policy Optimization (2017) clips the objective so each update gains nothing from moving the policy's action probabilities too far from those of the policy that collected the data, which lets the same batch be reused for several updates Established. InstructGPT used PPO to fine-tune GPT-3 against a learned reward model of human preferences Established; for a language model, the state is the prompt, the action is the whole response, and the reward is the reward model's score (Chapter 10).
Why should I care?
As a researcher
The score-function (log-derivative) trick behind policy gradients is a general tool for optimising through random choices, and the variance of its estimates drives much RL research.
As an engineer
PPO-style policy-gradient training and its descendants are how preference and reward signals are turned into language-model updates.
Modern systems that depend on it
- actor–critic methods
- PPO
- RLHF
- RL for reasoning
Historical context
Before
Learn values, then act greedily with respect to them; policies were a by-product.
After
The policy is the model being trained, and reward shapes it directly through gradients.
Used today
In robotics and games (PPO and relatives) and in post-training of language models, where the 'action' is a generated response.
What to remember
- ∇J(θ) = E[ ∇ log π_θ(a | s) · (G − b) ]: push up the log-probability of an action in proportion to how much better than expected it turned out.
- Subtracting a baseline b does not bias the gradient but greatly reduces its variance.
- If you only ever reinforce the correct label with reward 1, the update is the ordinary cross-entropy gradient.
- PPO clips each update so the new policy does not move too far from the one that collected the data.
Key papers
Reinforcement Learning: An Introduction (2nd edition)
Richard S. Sutton, Andrew G. Barto · 2018 · MIT Press
The standard textbook, free online from the authors. Everything in this chapter's RL half (MDPs, value functions, Q-learning, exploration, policy gradients, actor–critic) is developed carefully there.
How to read it: Chapters 1, 3 and 6 cover the core: the problem, MDPs, and temporal-difference learning including Q-learning. Chapter 13 is policy gradients.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J. Williams · 1992 · Machine Learning
REINFORCE: the basic policy-gradient estimator. Make the actions that led to high reward more probable. The same estimator sits inside modern RL fine-tuning of language models.
Proximal Policy Optimization Algorithms
John Schulman, Filip Wolski et al. · 2017
PPO: a simple, robust actor–critic policy-gradient method. It became the default RL algorithm in many labs and was the optimiser in InstructGPT-style RLHF.