Direct Preference Optimization

Compare two changes

For the same prompt, you have a chosen answer and a rejected answer. Ask how much the policy has increased each answer's probability relative to a fixed reference. DPO rewards a larger relative gain for the chosen answer.

Suppose reference log probabilities are −4 for the chosen response and −3 for the rejected one. The current policy assigns −3.5 and −3.2. The gains are +0.5 and −0.2, a difference of 0.7. At β=0.5, the scaled margin is 0.35, the implied pair preference is 58.7%, and the loss is 0.533 nats. At the reference, the margin is zero and loss is 0.693.

The equation follows the example

For preferred $y_w$, rejected $y_l$ and prompt $x$:

$$u=\beta\left[\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right],\qquad \mathcal L_{\mathrm{pair}}=-\log\sigma(u).$$

Average this over training pairs. A response log probability is a sum of its conditional token log probabilities. Length normalization would define a different variant. Prompt boundaries, role tokens and end tokens must be handled consistently in policy and reference scoring.

The original derivation links this objective to KL-regularized reward optimization. Vanilla DPO needs neither a separately trained reward model nor online sampling during preference fitting. That theoretical connection is not a guarantee that finite-data DPO and a particular PPO run produce identical models.

A counterexample worth remembering

Click Both probabilities fall. The policy now assigns −4.5 to both responses. The chosen response lost 0.5 log units relative to its reference; the rejected response lost 1.5. The pair margin improved even though both absolute probabilities decreased. DPO controls a comparison, not two independent promises about absolute likelihood.

What to evaluate

Track the pair objective, but also compare held-out outputs for task success, factuality, refusals and retained capability. Preference data can reward verbosity, confidence or agreement with a user's mistake. DPO does not remove those biases simply by avoiding an explicit reward head. The TRL documentation provides a practical companion to the paper's mathematics.