Reinforcement Learning from Human Feedback

Let the policy try

In Chapter 5, a policy chose actions and received rewards. For a causal language model, the state is the prompt plus tokens generated so far, and an action is the next token. A completed response can receive a score from a learned reward model. A policy-gradient algorithm uses that signal to change the probabilities of the sampled actions.

InstructGPT first trained on demonstrations, then learned a reward model from ranked responses, then used PPO for policy optimization. This is an influential recipe, not the definition of every assistant.

Reward model + PPODPO

1. 01

   Show an answer

   Demonstrations train an SFT policy. Keep a frozen reference copy.

2. 02

   Compare answers

   Judges rank candidates. A reward model learns to predict those comparisons.

3. 03

   Sample and improve

   The policy generates responses. PPO uses reward and a penalty for drifting from the reference.

These are common teaching pipelines, not mandatory stages for every assistant. DPO-based systems can still use a reward model elsewhere, for example to select demonstrations.

Why keep a reference?

A judge is easiest to exploit far from examples it knows. One restraint is a KL penalty against a frozen reference, commonly the SFT checkpoint. A simplified sequence-level objective is

$$\max_\theta\;\mathbb E_{y\sim\pi_\theta(\cdot\mid x)}[r(x,y)] -\beta D_{\mathrm{KL}}\big(\pi_\theta(\cdot\mid x)\;\|\;\pi_{\mathrm{ref}}(\cdot\mid x)\big).$$

Read it as reward for the response, minus a cost for changing the distribution. Larger positive β makes deviation more expensive in this idealized objective. Real implementations estimate terms from samples and may add other objectives.

Three answers are enough to see the problem

Our invented reference chooses a brief correct answer 50% of the time, a detailed correct one 30%, and a confident false one 20%. The flawed judge assigns rewards 0, 1 and 2. At β=1, the optimal distribution is approximately 17.9%, 29.2%, 52.9%. The proxy improved and correctness fell from 80% to 47.1%.

This lab solves a three-action objective exactly; it does not run PPO. Reducing β makes the bad judge's favorite answer even more likely. Penalizing the unsupported claim changes the target and reverses the effect.

+Deep diveThe finite-action optimumShould know

For actions with reference probabilities $q_i$, reward $r_i$ and positive β, the optimum is

$$p_i^*=\frac{q_i e^{r_i/\beta}}{\sum_j q_j e^{r_j/\beta}}.$$

The reference supplies a starting distribution; exponentiated reward tilts it. If the reference gives an action zero probability, finite forward KL keeps it outside the support. This analytic solution explains the lab; large neural policies require optimization instead.

Restraint is not truth

KL can limit a distribution change. It cannot determine whether a date was invented, a refusal was appropriate or a summary preserved uncertainty. Those require better feedback and separate evaluation. This is why monitoring reward alone is insufficient.

Keep the reference and the policy used to collect the latest rollout distinct. The latter appears in PPO's importance ratio; it need not be the frozen SFT model. The PPO concept follows those two roles explicitly.