Reasoning Models

The problem

A little room to work

Start at 1. You may add 2, multiply by 2, or subtract 3. Reach 19 in at most six moves.

The rules fit in one sentence. Finding a route is another matter. Always stepping to whatever is closest to 19 feels sensible: 1 → 3 → 6 → 12 → 14 → 16 → 18. Six moves, no solution: from 6 onward every number is even, an even number becomes odd only by subtracting 3, and subtracting always looks like a step backwards. The locally attractive choice at 3 discarded a useful alternative: 1 → 3 → 5 → 10 → 20 → 17 → 19.

This small puzzle exposes a distinction we will use throughout the chapter: knowing the available operations is different from allocating enough work to combine them. You can write intermediate results, explore alternatives, check candidates, or learn which attempts are worth making.

Chapter 13 spent computation interacting with an environment. Here we spend it before settling on an answer. A reasoning model can also use tools and act as an agent; the two ideas overlap. But more tool calls, more intermediate tokens, and a better-trained model are three different interventions.

A useful engineering definition of a reasoning model is a model trained to make productive use of intermediate computation on multi-step tasks. It is a description of training and behavior, not a claim of consciousness or a separate species of neural network. Interpretation

Two places to spend compute

Recall pretraining and post-training. Training changes parameters using many examples. Inference uses the resulting parameters to answer a particular request.

Extra inference computation does not, by itself, update the weights. A longer response runs the model more times; it does not automatically teach the checkpoint a lasting lesson. Established

An ordinary autoregressive answer already takes repeated forward passes—roughly one per decoded token, ignoring serving optimizations. Reasoning does not turn “one forward pass” into “many” for the first time. It changes what those additional passes are used for. A brief answer may conceal a longer computation; a long answer may contain little useful work.

For a data engineer, think of training as investing in the system that generates query plans, and inference as the work spent on a particular query. Better planning and more execution effort can help together. Neither removes the need for a correct result.

The first move: intermediate work

Give the answer a scratchpad

Suppose six trays hold seven seedlings each, and half are moved outside. “21 remain” is the answer. “6 × 7 = 42; 42 ÷ 2 = 21” gives the next prediction useful intermediate state. The number 42 is now in the context instead of having to remain implicit in internal activations.

Chain-of-thought prompting demonstrated that examples containing intermediate steps improved performance on several reasoning tasks for sufficiently large models in the paper's experiments. Established A few months later, Kojima and colleagues found that adding only “Let's think step by step” before the answer, with no worked examples, raised one large model's accuracy on the MultiArith benchmark from 17.7% to 78.7%. Established Prompting a pretrained model with worked examples changes its input. Fine-tuning it on worked solutions changes its weights. RL changes those weights using a reward signal. Keep these three mechanisms separate even when they produce similar-looking text.

The probabilistic picture is the same autoregressive model from Chapter 8. Introduce intermediate text $z$ between question $x$ and answer $y$:

$$p(y\mid x)=\sum_z p(z\mid x)\,p(y\mid x,z).$$

This is a conceptual marginalization over possible intermediate sequences, not an algorithm that enumerates them all. One generated solution samples or decodes one such path. More tokens give additional computational steps and context, but also more chances to carry an early error forward.

Try the puzzle mentally again. The useful intermediate state is not “I should think carefully.” It is the current number, moves remaining, and alternatives you have not ruled out. Productive work changes what the next step can do.

Explore before committing

Chapter 1 introduced search over states. The same structure returns here: propose a continuation, evaluate it, retain promising possibilities and expand again. A beam keeps several states at each depth. A greedy search keeps only one.

Tree of Thoughts explored explicit search over intermediate text states, with evaluation and backtracking, on a small set of problem-solving tasks. Established That is an inference procedure around a model. It does not mean every model sold as a reasoning model runs an explicit tree search.

The lab uses arithmetic rather than generated text so every transition is inspectable. Press Play with one path. Then keep three paths. The extra branches consume checks; the successful route must still execute all six legal operations.

Our distance heuristic prefers 6 over 5 after reaching 3. It cannot see that 5 leads to 10, then 20, then 17 and 19. More width preserves that possibility. Reduce the move budget and width cannot compensate for missing depth. Try other targets: a successful demonstration on 19 is not evidence that three paths are optimal everywhere.

+Deep diveWhere MCTS differsFrontier

Monte Carlo Tree Search repeatedly selects a path, expands a state, obtains an evaluation (often through a rollout or value estimate), and backs that value up through its ancestors. It balances exploiting high-value paths with exploring less-visited ones. Beam search instead retains a bounded frontier by a score. Our lab is beam search; it has no visit counts, rollouts or MCTS backup. For language-model search, defining a state, a useful evaluator and a stopping rule remains consequential design work.

Several answers, one decision

A crowd can repeat one mistake

Another way to spend compute is to start over. Generate several full solutions, extract their final answers, and choose the most frequent answer. Different wordings that end in 21 count toward the same answer.

Self-consistency improved results on the paper's arithmetic and commonsense benchmarks by sampling multiple reasoning paths and aggregating their answers. Established Agreement is useful when correct answers accumulate more reliably than any particular wrong answer. It is not a truth test.

Imagine eight runs that all overlook “half moved outside” and answer 42. Eight votes can make one misunderstanding look certain. Even separately sampled completions share a model, prompt and training history, so they can share a systematic error without literally copying text.

If each independent candidate solves a fixed task with probability $p$, the chance that at least one succeeds is:

$$P(\text{at least one correct})=1-(1-p)^N.$$

At $p=0.25$, four samples give $1-0.75^4\approx68.4\%$. That is availability, not the probability that your selection rule returns the correct answer. If no trustworthy selector identifies the good candidate, it may be wasted. The equation also assumes independent, identical trials on this task; plugging an average accuracy across mixed-difficulty tasks into it generally gives the wrong aggregate result.

Who checks the checker

A verifier evaluates a candidate. A calculator can check arithmetic. Tests can check a program on specified cases. A learned reward model can score a solution. These are different kinds of evidence with different blind spots.

Best-of-N generates N candidates and chooses the one with the highest verifier score. Self-consistency chooses by answer frequency. Cobbe and colleagues introduced the GSM8K grade-school math benchmark and trained verifiers to judge model solutions; generating many candidates and keeping the one the verifier ranked highest significantly improved performance on it. Established An oracle that simply tells you which candidates are truly correct is a useful experimental upper bound on selection from that batch, not a tool most applications possess.

In this lab, the correct answer is deliberately known. Each tile shows a candidate's answer and its verifier score, and a star marks the verifier's pick. You can see both what each selector chooses and whether the batch contained a better option. Press Play at the starting seed and watch the commonest mistake win the vote. Then make the verifier less reliable or the candidates more repetitive.

Separate a generation failure (“none of the candidates was correct”) from a selection failure (“a correct answer was present but discarded”). They suggest different fixes. Interpretation More samples address the first only when the generator has a chance of success. Improving the verifier may address the second.

The lab's verifier errs independently on each candidate, so more candidates mostly help it. Real scorers also have systematic blind spots, and then more candidates give them more chances to find a wrong answer they over-rate. Gao and colleagues measured this with a proxy reward model and a held-out “gold” one: pushing best-of-N harder against the proxy eventually lowered the gold score. Established

A test suite is stronger evidence than confident prose, but it checks only the cases and specification it encodes. A model can exploit a weak checker. For open-ended explanation, research design or ambiguous real-world decisions, defining a reward is itself part of the problem.

Reward the destination or the route

Return to $6\times7\div2$. Both of these responses end in 21:

• Valid: 6 × 7 = 42; 42 ÷ 2 = 21.
• Lucky: 6 × 7 = 40; 40 ÷ 2 = 21.

An outcome checker that looks only at the final number rewards both. A step checker rejects the second. Outcome supervision supplies a label for the result; process supervision supplies labels or scores for intermediate steps.

Let's Verify Step by Step found process supervision more effective than outcome supervision in its MATH evaluation and released step-level feedback data. Established This is evidence from a particular setup, not proof that process supervision always wins. Step labels can be expensive and ambiguous, and a learned process checker can be wrong.

More detail is not necessarily a better process. A verbose response can repeat an arithmetic mistake four times. Rewarding length, self-confidence or the phrase “let me verify” rewards observable surface features; none is equivalent to checking the arithmetic.

Teach the generator

Practice with a checkable reward

At inference time, we can choose among attempts. At training time, we can change the probability of those attempts. The reinforcement-learning loop is familiar from Chapter 5: sample behavior, obtain reward, update the policy. Here the behavior is a token sequence.

Reinforcement learning with verifiable rewards uses an executable or rule-based check to produce training feedback—for example, agreement with a known mathematical answer or passing specified code tests. Established “Verifiable” describes the reward mechanism. It does not guarantee that the specification is complete, the checker cannot be exploited, or the model will generalize.

The following lab actually updates four policy logits. It cannot invent a fifth response, so it isolates one question: which existing behavior does this reward make more likely? Train with each reward and compare correct-answer probability to valid-derivation probability.

+Deep diveThe exact gradient in the labShould know

Let $\theta_i$ be the logit of response $i$, $p_i=\operatorname{softmax}(\theta)_i$ and $r_i$ its reward. The expected reward is $J=\sum_i p_i r_i$. Its gradient is

$$\frac{\partial J}{\partial\theta_i}=p_i(r_i-J).$$

The lab applies $\theta_i\leftarrow\theta_i+\eta p_i(r_i-J)$ with learning rate $\eta=1$. Initially all four responses have probability 0.25. Under outcome reward, $r=[1,1,0,0]$ and $J=0.5$, so the first two logits each increase by 0.125 and the others decrease by 0.125. The next distribution favors correct final answers while remaining unable to distinguish valid work from lucky work. This is exact expected-reward optimization in a tiny bandit; real LLM RL estimates gradients from sampled token sequences and often constrains updates.

Relative rewards and the training recipe

A reward of 1 is informative relative to something. If every sampled solution is correct—or every one is wrong—comparing that group supplies no within-group preference. A useful training problem must connect the model's current capabilities to a learnable signal.

DeepSeekMath introduced Group Relative Policy Optimization (GRPO), which uses group-relative rewards and avoids the separate critic used in common PPO implementations. Established Its group-normalized advantage can be written schematically as

$$A_i=\frac{r_i-\operatorname{mean}(r)}{\operatorname{std}(r)+\epsilon}.$$

This is the advantage signal, not the full GRPO objective. Probability ratios, clipping, token aggregation and regularization also matter. Implementations differ; a zero-variance group needs explicit handling. For rewards $[1,0,0,1]$, the population mean and standard deviation are 0.5, giving advantages close to $[1,-1,-1,1]$ when $\epsilon$ is tiny.

The January 2025 DeepSeek-R1 report (its first arXiv version) distinguished R1-Zero, which applied RL without preliminary SFT, from R1, which used a multi-stage recipe including cold-start data. It also described transferring generated training data to smaller models. Established These are versioned historical examples, checked October 6, 2026—not a description of every current system. “RL alone” in this context still starts from a pretrained model; it does not mean learning language and mathematics from nothing.

Put together, the pieces of this chapter form three loops that run at different times and change different things. Step through each one.

Train: RL with checkable rewardsDistill: turn attempts into dataServe: spend inference per request

Changes the model's weights. Runs offline, before a checkpoint ships.

1. 1Pick checkable problemsMath with known answers, code with tests. The problems must be hard enough that some attempts fail and easy enough that some succeed.
2. 2Sample a group
3. 3Check each one
4. 4Compare within the group
5. 5Update the weights

← BackNext stage →

A teaching map, not any one lab’s pipeline. Real recipes interleave stages: DeepSeek-R1, for example, alternated supervised fine-tuning and RL, and its distilled models were trained on data the RL model generated.

How much an RL recipe adds new generalizable problem-solving behavior, versus amplifying behavior already available under sampling, depends on the model, data, task and evaluation budget. Active research A larger benchmark score alone cannot resolve that distinction. Inspect the base model's candidate distribution, matched-compute baselines and held-out task families.

Turn successful attempts into data

A slow, expensive generator can produce many attempts. Check them, retain useful solutions, and fine-tune another model on the retained question–solution pairs. This creates synthetic reasoning data. Training a smaller student on those traces is one form of reasoning distillation; it need not match the teacher's token probabilities.

The teacher can even be the student itself. STaR ran a simple loop: generate rationales for many questions, retry failures with the correct answer given as a hint, fine-tune on the rationales that ended in correct answers, and repeat. It improved over a model fine-tuned to predict answers directly on the paper's datasets. Established

Supervised training on selected teacher outputs changes the student's weights; presenting the same outputs as prompt examples does not. Established The teacher need not be a single model call. It can be a model plus search, tools and filtering. The training record should say which system produced the targets.

The data-pipeline analogy is direct: generate → validate → filter → deduplicate → split → train → evaluate. Keep held-out questions and their near-duplicates out of both generation prompts and student training. A correct final answer attached to a flawed rationale can still be a bad teaching example. Filtering only by final answer inherits the checker problem from the previous lab.

Distillation moves some expense upstream: spend more generating training targets so the student may need less work on later requests. It does not guarantee the student's competence, faithfulness or efficiency on new tasks. Compare it with the unmodified student using the same evaluation and inference budget.

From prompt trick to product

Every mechanism in this chapter appeared in research papers before anyone sold a “reasoning model”. Worked examples and verifiers came first, then voting, search and step-level rewards, then group-relative RL.

OpenAI's September 2024 announcement of o1 described a model trained with large-scale reinforcement learning to use a chain of thought, and reported that its performance improved with both more RL training compute and more time spent thinking at test time. Established The announcement gave few training details; checked October 6, 2026.

Its AIME 2024 results put this chapter's selection ideas side by side. OpenAI reported that o1 solved 74% of problems with one sample per problem, 83% with a vote over 64 samples, and 93% when re-ranking 1,000 samples with a learned scoring function. Established One generator, three selection rules, three numbers, and the inference cost rises at each step. These are the company's own evaluations, not an independent measurement.

The same announcement said users would not see the raw chain of thought, only a model-generated summary of it. The text a user reads is then not even the intermediate sequence the model produced, which matters for the last section of this chapter. Four months later, DeepSeek-R1 published open weights and a written training account, so the recipe could be studied outside one company.

“Reasoning model” became a product category with these releases, but the ingredients are older: the 2024–2025 change was mainly training intermediate work with checkable rewards at scale and budgeting inference for it. Interpretation

Spend carefully

More compute needs a useful place to go

There are at least three inference budgets: depth within a solution, breadth across solutions, and verification to select or revise them. A token cap controls only part of that system. Parallel candidates can reduce wall-clock delay while consuming more total compute; a long sequential chain cannot gain that same parallel speedup.

Snell and colleagues found that effective test-time compute allocation depended on problem difficulty and the inference strategy in their experiments. Established Their result motivates choosing the allocation by task; it does not establish a universal curve or make inference compute interchangeable with training compute.

Our search lab shows a narrow version of this: width saves a discarded path, depth permits enough operations, and an exact check recognizes the target. Giving the wrong heuristic more budget can still disappoint. Likewise, a language model can spend tokens repeating a misconception.

For a rough planning ledger, if a model costs $C_{\text{train}}$ to train and serves $Q$ requests at average inference cost $C_{\text{infer}}$, then

$$C_{\text{total}}=C_{\text{train}}+Q\,C_{\text{infer}}.$$

The units must match—FLOPs, energy or money under a specified costing model. This is accounting, not a performance law. Quality, latency, memory and request difficulty constrain which allocations are useful. Spending 10 extra units per request adds 10 million units across a million requests; an upstream improvement can therefore be worth amortizing if it actually saves work at matched quality.

Evaluate a budget policy, not just a model name: which requests receive more work, when does the system stop, and what does it do when the budget runs out? Interpretation An answer that states what remains unresolved is preferable to treating exhausted tokens as evidence of success.

Read the evidence rather than the monologue

A long worked answer is an artifact you can inspect. It is not a guaranteed explanation of the computation that caused the answer. A system may expose a summary, hide intermediate tokens, or generate a plausible rationale after being influenced by something it never mentions.

Turpin and colleagues demonstrated settings where biasing information changed model answers without appearing in the chain-of-thought explanation. Established The conclusion is specific: a convincing explanation alone does not establish faithfulness. It does not imply that all intermediate reasoning is useless or always misleading.

Before trusting a reasoning result, make five questions concrete:

1. What changed? Prompt, weights, search, tools, verifier, or all of them?
2. What was the budget? Count generated tokens, repeated prompt processing, verifier work and latency.
3. What was selected? First answer, vote, best scored answer, or an oracle-picked success?
4. What was held out? Check contamination, task diversity and uncertainty across runs.
5. What was actually verified? A final number, intermediate steps, tests, or an open-ended human judgment?

Reasoning systems are useful when extra work buys better outcomes under a well-specified check and budget. The amount of text produced is not that outcome. Interpretation

You have now connected search from Chapter 1, autoregressive generation from Chapter 8, policy optimization from Chapter 10, serving costs from Chapter 11, and verification from Chapter 13. Chapter 15 extends the inputs and outputs beyond text. The same questions remain when the model reasons over an image, a recording or a physical action: what did it observe, what did it compute, and what evidence supports the answer?