RL with Verifiable Rewards

The checker supplies the signal

An arithmetic answer can be compared with a reference. A program can be run against tests. Each check turns a task into an environment that hands out rewards with no human in the loop. Reinforcement learning then does what it did in Chapter 5: try, score, and make high-scoring behaviour more likely.

Reinforcement learning with verifiable rewards uses an executable or rule-based check, such as agreement with a known answer or passing specified tests, to produce training feedback. Established

Compare the alternatives from Chapter 10. RLHF learns a reward model from human preferences, which can judge open-ended answers but is itself a model with exploitable errors. Supervised fine-tuning imitates solutions someone wrote, so it cannot exceed them. A verifiable reward is narrower than either, but cheap, consistent and not a neural network that can be flattered.

The loop

1. Sample: for each training problem, generate one or more complete responses from the current model.
2. Check: compute each response's reward (1 if the answer matches, perhaps a small reward for the right format).
3. Compare: estimate how much better or worse each response is than expected, for example against the group average (GRPO).
4. Update: change the weights to make better-than-expected responses more likely, with limits on how far one step can move the policy.

Inference-time selection (best-of-N) uses the same check but stops after step 2. RL is what makes the next attempt better.

The exact gradient, in a tiny case

The reward lab has four fixed responses with logits $\theta_i$, probabilities $p_i = \operatorname{softmax}(\theta)_i$ and rewards $r_i$. Expected reward is $J = \sum_i p_i r_i$, and

$$\frac{\partial J}{\partial \theta_i} = p_i\,(r_i - J).$$

What it does. A response gains logit in proportion to how likely it already is and how much better than average it scored. Numbers. Start uniform, $p_i = 0.25$. With outcome reward $r = [1, 1, 0, 0]$, $J = 0.5$, so each correct response's logit rises by $0.25 \times 0.5 = 0.125$ and each wrong one's falls by $0.125$. After one update the correct responses each have probability 0.281.

Real training cannot compute $J$ exactly: it estimates the gradient from sampled token sequences, which is noisy, and handles thousands of tokens per response. The lab removes that noise on purpose to isolate one question: what does this reward make more likely?

What R1 reported

The January 2025 DeepSeek-R1 report described R1-Zero, trained with large-scale RL using rule-based rewards directly from a base model without supervised fine-tuning first, and R1, which added cold-start data and further stages; it also described distilling R1's reasoning into smaller models. Established The report attributed behaviours such as self-verification and longer reasoning to the RL stage. “RL alone” still begins from a large pretrained model, which supplies the language and much of the mathematics.

How much RL with verifiable rewards creates new problem-solving ability, versus making the base model's existing good samples more likely, is debated and depends on model, task and evaluation budget. Active research One useful comparison: the base model's accuracy when allowed many samples versus the trained model's accuracy with one.

What a check cannot see

• Incomplete tests reward code that passes them without meeting the specification.
• Answer-only checks reward lucky or memorised answers; see process supervision.
• Exploitable checkers: if the model can affect the checker (edit the test file, print the expected string), it may learn to.
• Coverage: tasks without checks, such as writing, research design or judgement calls, get no reward at all.

Keep the training checks separate from the evaluation set, and inspect samples, not only the reward curve.

Mini experiment

In the lab, train with Correct final answer for 60 updates and note the valid-derivation probability. Reset, train with Valid steps, and note it again. Then write down a reward for a coding task that would let a model score well without writing a correct program, and one extra check that would close the gap.