Concept · Chapter 8: The Rise of Large Language Models
Scaling Laws (Preview)
A language model's loss falls smoothly and predictably, as a power law, as its parameters, training data and compute grow, which turned 'make it bigger' into a plannable engineering decision.
The problem
Training a frontier model costs months and vast sums; teams needed to know in advance whether scaling up would pay off, and how to split a budget between model size and data.
The solution
Train many smaller models, fit how loss depends on parameters N, data D and compute C, and extrapolate.
The consequence
Labs could predict big-model loss from small runs and justify ever-larger investments; later work (Chinchilla) corrected the recommended balance toward much more data.
You should understand first
- Text as Data
- Vectors
- One-Hot Encoding
- Tokenization
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Dot Product
- Embeddings
- Attention
- Softmax
- Self-Attention
- Causal Masking
- Entropy
- Loss Functions
- Cross-Entropy Loss
- Autoregressive Next-Token Prediction
- Pretraining at Scale
- Parameters, Tokens and Context Windows
- Scaling Laws (Preview)
What was found
Kaplan and colleagues (2020) found that language-model cross-entropy loss falls as a power law in model size, dataset size and training compute, with some trends spanning more than seven orders of magnitude, while details such as width versus depth matter little within a wide range Established. A power law looks like
with a small exponent (below 0.1 for parameters in their fits). Take logarithms and it becomes a straight line: every tenfold increase in parameters cuts the loss by the same fraction.
Tiny example. With , multiplying parameters by 10 multiplies the loss by : a 16% reduction, again and again, for each tenfold step. Steady, but each step costs ten times more.
Why it mattered
Predictability turned scale into planning. OpenAI reported predicting aspects of GPT-4's performance from models trained with no more than 1/1,000th of its compute Established. The training compute of a dense model is roughly floating-point operations, so a budget can be split between parameters N and tokens D in advance.
The Chinchilla study (2022) revised the recommended split: for a fixed compute budget, parameters and training tokens should grow in equal proportion, and a 70B model trained on 1.4 trillion tokens outperformed the 280B Gopher Established. That is about 20 tokens per parameter, a ratio now often quoted as a rule of thumb. Chapter 9 works through the numbers.
What to remember
- Loss ≈ power law in N, D and C, with trends over more than seven orders of magnitude (Kaplan et al. 2020).
- On a log–log plot, a power law is a straight line: steady returns for each multiplication of resources.
- Chinchilla (2022): scale parameters and tokens together; about 20 tokens per parameter in its own case.
- Scaling laws predict loss, not specific abilities, and they describe trends, not guarantees.
Key papers
Scaling Laws for Neural Language Models
Jared Kaplan, Sam McCandlish et al. · 2020
Found that language-model loss falls as a smooth power law in parameters, data and compute — making model scale something you could plan.
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud et al. · 2022 · NeurIPS 2022
Showed that many large models were undertrained: for a fixed compute budget, parameters and training tokens should grow roughly in proportion.
GPT-4 Technical Report
OpenAI et al. · 2023
Documented a large jump in capability, including human-level scores on many professional and academic exams, and marked the point where frontier labs stopped disclosing model size, data and training details.
How to read it: Note what the report does not contain: architecture, parameter count, data and compute are all withheld.