Group Relative Policy Optimization (GRPO)

Better than what?

A reward of 1 says a response passed. To learn, the policy needs to know whether it did better than expected: that is the advantage. Policy-gradient methods push up responses with positive advantage and push down those with negative advantage.

PPO estimates “expected” with a learned value model (a critic) that predicts the reward from the prompt and partial response. For a large language model the critic is usually another network of comparable size, trained alongside the policy.

DeepSeekMath introduced Group Relative Policy Optimization, a PPO variant that drops the critic and instead estimates the baseline from the scores of a group of responses sampled for the same prompt, reducing memory use. Established

The group baseline

For one prompt, sample $G$ responses and score them $r_1, \dots, r_G$. Each response's advantage is

$$A_i = \frac{r_i - \operatorname{mean}(r)}{\operatorname{std}(r) + \epsilon}.$$

Tiny example. Four responses with rewards $[1, 0, 0, 1]$. The mean is 0.5 and the population standard deviation is 0.5, so the advantages are $[1, -1, -1, 1]$ (ignoring the tiny $\epsilon$). The two passing responses are pushed up equally and the two failing ones down.

What it does. The group plays the role of the critic: “expected” means “what this model typically scores on this prompt right now”. It is the same idea as comparing a pipeline run against its own recent runs instead of against a separately trained forecast.

Groups with nothing to say

If all four responses pass, $r = [1, 1, 1, 1]$: every $r_i - \operatorname{mean}(r)$ is zero and the standard deviation is zero too, so implementations must guard the division and the prompt contributes no gradient. The same happens if all four fail. Problems the model always solves, or never solves, waste compute.

So the curriculum matters: useful training problems sit where the model sometimes succeeds. This is the RL version of the observation in the consensus lab that more samples help only when the generator has a chance.

The rest of the objective

The advantage is one ingredient. GRPO, like PPO, also uses:

• a probability ratio between the current policy and the one that generated the samples, so several update steps can reuse one batch;
• clipping of that ratio, so one batch cannot move the policy too far;
• a KL penalty toward a reference model, discouraging drift into degenerate text;
• choices about averaging over tokens, which decide how much each token of a long or short response counts.

Reproducing a GRPO result requires all of these details, and published implementations differ.

The DeepSeek-R1 report used GRPO for its large-scale RL stages. Established That connection, an open model trained with a documented, critic-free method, is much of why GRPO spread quickly.

Mini experiment

Compute the advantages by hand for rewards $[1, 1, 1, 0]$ and $[0, 0, 0, 1]$. Which single response gets the larger push in each case, and why does a rare success get a bigger advantage than a common one? Then explain what happens to a prompt where all eight samples score 1.