Skip to content
Road to Intelligence

Concept · Chapter 14: Reasoning Models

Synthetic Reasoning Data and Distillation

Should knowKnow well10 minDifficulty

Reasoning distillation fine-tunes a model on solutions that a stronger or more expensive system generated and that passed a check, moving inference effort upstream into training data.

The problem

A strong reasoning system is too slow or expensive to serve everywhere, and hand-written step-by-step solutions are scarce.

The solution

Run the expensive system on many questions, keep the solutions that verify, and use them as supervised training data for a cheaper model, or for the same model, repeatedly.

The consequence

Smaller models gain much of the teacher's behaviour on similar tasks, and the filter, the deduplication and the evaluation split decide what exactly they learn.

You should understand first

  1. Probability and Distributions
  2. Entropy
  3. Softmax
  4. Loss Functions
  5. Cross-Entropy Loss
  6. KL Divergence
  7. Knowledge Distillation
  8. Text as Data
  9. Conditional Probability and Bayes' Theorem
  10. Probability of Sequences
  11. Language Modeling
  12. Vectors
  13. Dot Product
  14. Embeddings
  15. Attention
  16. Self-Attention
  17. Multi-Head Attention
  18. Causal Masking
  19. Positional Encoding
  20. Residual Connections
  21. Layer Normalization
  22. Feed-Forward Sublayer (MLP)
  23. The Transformer Block
  24. Encoder, Decoder & Encoder–Decoder
  25. Pretrain, Then Fine-Tune
  26. Supervised Fine-Tuning
  27. The Turing Test
  28. Symbolic AI
  29. Logic and Rules
  30. Expert Systems
  31. Knowledge Representation
  32. The Knowledge-Acquisition Bottleneck
  33. From Rules to Learning
  34. Supervised, Unsupervised and Self-Supervised Learning
  35. Autoregressive Next-Token Prediction
  36. Decoding: Greedy, Temperature, Top-k, Top-p
  37. Learning from Comparisons
  38. Features, Labels and Tasks
  39. Derivatives and Gradients
  40. Gradient Descent
  41. Linear Regression
  42. Logistic Regression
  43. Reward Models
  44. Pretraining at Scale
  45. GPT-1 → GPT-2 → GPT-3
  46. In-Context Learning
  47. Chain of Thought
  48. Expected Value and Variance
  49. Sampling and Uncertainty
  50. Self-Consistency
  51. Verifiers and Best-of-N
  52. Synthetic Reasoning Data and Distillation

Spend once, teach many times

A teacher system can be slow: a large model, sampling 64 attempts, searching, running tests. Run it once over a large question set, keep what checks out, and you have training data. A student fine-tuned on that data may then answer similar questions with one cheap attempt. The expensive work has moved from every request to one dataset build.

This is knowledge distillation in a broad sense. Classic distillation trains a student to match a teacher's output probabilities. Established Reasoning distillation usually trains only on the teacher's sampled text, with ordinary supervised fine-tuning; the student never sees the teacher's probabilities.

A model can be its own teacher

STaR (the Self-Taught Reasoner) generated rationales for many questions using a few examples, retried failures with the correct answer given as a hint, fine-tuned on the rationales that ended in correct answers, and repeated; it improved over a model fine-tuned to predict answers directly on its datasets. Established

The retry with a hint, which the paper calls rationalization, matters: without it the model only learns from problems it could already solve.

The DeepSeek-R1 report described fine-tuning smaller open models on reasoning data generated with R1, and reported strong reasoning-benchmark results for these distilled models. Established

The pipeline, as a data engineer would build it

  1. Generate. Record the teacher checkpoint, prompt, sampling settings, tools and seed for every example.
  2. Verify. Check answers, run tests. Store the verdict, not just the survivors.
  3. Filter. Decide what passes. Final-answer filters keep lucky derivations (see the reward lab); step checks or a process scorer are stricter but costlier.
  4. Deduplicate. Near-duplicate problems inflate apparent learning.
  5. Split. Keep evaluation questions, and paraphrases of them, out of generation prompts and training data. This is contamination control.
  6. Fine-tune the student.
  7. Evaluate the student against its own pre-training baseline, at the same inference budget, on held-out task families.

Tiny example

The teacher answers 10,000 problems with 8 samples each. 62% of problems have at least one sample that passes the answer check; keeping one passing solution per problem gives 6,200 examples. If a step checker would reject 10% of those as lucky, the strict dataset has 5,580. Is the cleaner, smaller set better? Only an evaluation on held-out problems can say. (Illustrative numbers.)

Limits

  • The student learns what is in the data. Problems the teacher never solved contribute nothing.
  • Imitating long, confident text is not proof of the underlying skill; a student can copy the style of self-checking without checking.
  • A distilled student can still need long outputs, so it is not automatically cheap to serve.
  • Teacher accuracy, student accuracy, dataset cost and serving cost are four different numbers. Report all four.

Mini experiment

Take ten problems you can check. For each, write two candidate solutions: one valid and one that reaches the right answer by a flaw. Apply an answer-only filter and a step-by-step filter. How many flawed solutions would a student see under each, and what would you expect it to learn from them?

What to remember

  • Pipeline: generate → verify → filter → deduplicate → split → fine-tune → evaluate.
  • The teacher can be a model plus search, tools and many samples; record which system produced each example.
  • STaR: keep your own rationales that reach correct answers, fine-tune, repeat.
  • Filtering on final answers keeps lucky derivations too.
  • Compare the student before and after at the same inference budget, on held-out tasks.

Key papers

Important

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI et al. · 2025

An openly released reasoning model, with a detailed account of training long chains of reasoning mainly through reinforcement learning on verifiable problems.

How to read it: Read the R1-Zero and R1 sections separately: the first is RL straight from a base model, the second a multi-stage recipe with cold-start data, and the distillation results are a third story.

~1 h readarXiv:2501.12948✓ verified 2026-09-26
Essential

Distilling the Knowledge in a Neural Network

Geoffrey Hinton, Oriol Vinyals, Jeff Dean · 2015

Defined knowledge distillation: train a small student on a large teacher's softened output probabilities, which carry more information than hard labels.

How to read it: Section 2 explains temperature and soft targets in a page.

~25 min readarXiv:1503.02531✓ verified 2026-10-05
Important

STaR: Bootstrapping Reasoning With Reasoning

Eric Zelikman, Yuhuai Wu et al. · 2022 · NeurIPS 2022

An early, clear version of the loop behind synthetic reasoning data: a model generates rationales, keeps the ones that reach correct answers, and trains on them.

How to read it: Notice the filter is the final answer only, so a rationale that reaches the right answer by a flawed route is kept; compare Chapter 14's lucky answer.

~35 min readarXiv:2203.14465✓ verified 2026-10-06