Concept · Chapter 8: The Rise of Large Language Models
Emergent Abilities and the Debate
Some abilities appear to jump from near chance to competent as models grow; whether these jumps are real phase changes or artefacts of all-or-nothing metrics is debated.
The problem
If loss improves smoothly, why do some benchmark scores seem to switch on suddenly at a certain model size?
The solution
Compare the same models under different metrics: exact-match accuracy versus per-token or partial-credit measures.
The consequence
Many apparent jumps smooth out under continuous metrics, which matters for forecasting: capabilities may be more predictable than headlines suggest, but not every jump has been explained away.
You should understand first
- Text as Data
- Vectors
- One-Hot Encoding
- Tokenization
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Dot Product
- Embeddings
- Attention
- Softmax
- Self-Attention
- Causal Masking
- Entropy
- Loss Functions
- Cross-Entropy Loss
- Autoregressive Next-Token Prediction
- Pretraining at Scale
- Parameters, Tokens and Context Windows
- Scaling Laws (Preview)
- Emergent Abilities and the Debate
The claim
Wei and colleagues (2022) catalogued tasks on which small models score near chance and larger ones score well, and defined an ability as emergent if it is not present in smaller models but is present in larger ones Established. Multi-digit arithmetic and some multi-step reasoning benchmarks were examples. If true in general, emergence means you cannot forecast what the next, bigger model will be able to do.
The counter-argument
Schaeffer, Miranda and Koyejo (2023) argued that many such jumps come from the choice of metric: nonlinear or discontinuous metrics produce apparent emergence, while linear or continuous metrics applied to the same model outputs show smooth, predictable improvement Established.
Tiny example. Suppose each digit of a 5-digit answer is right with probability , and improves smoothly with scale: 0.5, 0.7, 0.9, 0.99. Exact match needs all five digits: = 0.03, 0.17, 0.59, 0.95. The per-digit score rises steadily; the exact-match score sits near zero and then shoots up. Same models, same outputs, different story.
Where it stands
Whether some capabilities genuinely appear abruptly with scale, and how predictable new abilities are, remains debated Active research. The practical lesson is about measurement: when a result looks like a sudden jump, check the metric, the number of test items (Chapter 2's sampling uncertainty) and whether a partial-credit measure tells a smoother story.
What to remember
- Wei et al. 2022: an ability is emergent if absent in smaller models and present in larger ones.
- Schaeffer et al. 2023: nonlinear or discontinuous metrics (exact match) can create apparent jumps; continuous metrics show smooth gains.
- A 5-digit answer scored all-or-nothing needs every digit right: smooth per-digit gains look like a sudden jump.
- Status: actively debated. Both 'it's all the metric' and 'there are real thresholds' overstate what is known.
Key papers
Emergent Abilities of Large Language Models
Jason Wei, Yi Tay et al. · 2022 · TMLR
Catalogued tasks where performance stays near chance for small models and then jumps at larger scale, and named them 'emergent abilities'.
Are Emergent Abilities of Large Language Models a Mirage?
Rylan Schaeffer, Brando Miranda, Sanmi Koyejo · 2023 · NeurIPS 2023
The counter-argument: many apparent jumps come from all-or-nothing metrics such as exact match. Scored with continuous metrics, the same models improve smoothly.