Process vs Outcome Supervision

Two places for a label

Compare two solutions to $6 \times 7 \div 2$:

• Valid: $6 \times 7 = 42$, then $42 \div 2 = 21$.
• Lucky: $6 \times 7 = 40$, then $40 \div 2 = 21$. Two wrong steps that happen to cancel.

An outcome label looks at the last line and marks both correct. A process label looks at each step and rejects the lucky solution at its first line. If you train on outcome labels, the lucky solution is as good as the valid one; the model has no reason to prefer valid steps when they lead to the same number.

In data-engineering terms: outcome supervision checks that the final report's total matches; process supervision checks every transformation in the pipeline. A matching total can hide two bugs that cancel.

The evidence

Lightman and colleagues compared outcome and process supervision for training reward models on the MATH dataset and found process supervision significantly better; their process-supervised model solved 78% of a representative subset of MATH test problems, and they released PRM800K, 800,000 step-level human labels. Established

Read the setup before generalising. The reward models were judged by how well they picked the best of many sampled solutions; the paper compares selectors, not RL training recipes. Active learning (choosing which solutions to label) also made the labelling budget go further.

The price of steps

• Labels are expensive. Someone, or something, must judge every step, and long solutions have many.
• Steps are ambiguous. Is “Let x be the number of trays” correct? Is an unnecessary but true step good? Labellers disagree.
• A learned PRM is a model. It generalises imperfectly and can be fooled, just like an outcome verifier.
• Valid is not the same as useful. A long chain of true but irrelevant steps passes a step checker.

Where steps are mechanically checkable (arithmetic, formal proofs, code that runs), process checks can be programs instead of models, which removes the labelling cost and most of the ambiguity.

Surface features are not process

“More detailed” sounds like “better process”, but detail can be rewarded by length alone. In the reward lab, choose More words: the verbose mistake, which repeats a wrong step with confident filler, climbs to about 97% probability within 60 updates, and the probability of a correct answer falls to about 2%. Choose Valid steps and the valid derivation reaches about 99%. Choose Correct final answer and the valid and lucky derivations rise together and stay exactly tied, each near 49%.

Mini experiment

Write three solutions to a two-step problem of your own: one valid, one with compensating errors, one verbose and wrong. Score each with three rules: final answer, every step valid, and number of lines. Which rule would you trust as a training reward, and which could a model satisfy without solving anything?