Concept · Chapter 8: The Rise of Large Language Models
Why Next-Token Prediction Goes So Far
To predict the next token of diverse human text well, a model is pushed to capture whatever regularities produced that text, which is why a simple objective yields broad abilities, and also why those abilities have characteristic limits.
The problem
It seems implausible that 'guess the next word' could produce translation, coding and reasoning abilities.
The solution
View prediction as compression: every regularity a model captures (grammar, facts, arithmetic, styles of argument) lowers its loss on text that depends on it.
The consequence
Capability can come from scale and data rather than task-specific design, but the model learns what helps prediction on its data, not truth or goals, so errors and gaps follow the data.
You should understand first
- Text as Data
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Vectors
- Dot Product
- Embeddings
- Attention
- Softmax
- Self-Attention
- Causal Masking
- Entropy
- Loss Functions
- Cross-Entropy Loss
- Autoregressive Next-Token Prediction
- Why Next-Token Prediction Goes So Far
The argument
Imagine predicting the next word of a physics textbook, a chess commentary, a Python file and a legal contract. Surface statistics get you only so far. To predict that a function returns a + b, it helps to have learned what the code does. To predict the next move in a game record, it helps to track the board. To predict the end of "the capital of Nepal is", it helps to know the fact. Every such regularity lowers the loss, so a large enough model trained on enough varied text is pushed toward representing a great many of them Interpretation.
Prediction is compression
This has a precise form. Shannon's information theory implies that a model assigning probability to the actual next symbol can encode it in about bits, so lower cross-entropy means better compression Established. A model with loss of 1 bit per token compresses text to 1 bit per token.
Tiny example. If the model gives the true next token probability 1/2, that costs 1 bit. With probability 1/1,024 it costs 10 bits. A model that "understands" more of the text is surprised less often, and its compressed file is smaller.
Delétang and colleagues (2023) used language models as compressors: Chinchilla 70B, trained mainly on text, compressed ImageNet image patches to 43.4% of their size and LibriSpeech audio to 16.4%, better than PNG (58.5%) and FLAC (30.3%) Established.
Where the argument stops
Whether good prediction amounts to understanding, or to building an internal model of the world, is contested Interpretation. Some probing studies find structured internal representations; critics stress that models can produce fluent text without reliable grounding. What is clear is what the objective does not reward:
- Truth over likelihood. A common misconception written thousands of times is easy to predict. The objective rewards matching the data, not being right.
- Helpfulness. A base model continues text; it does not try to help. That comes later (Chapter 10).
- Rare knowledge. Facts that appear rarely are learned weakly and reproduced unreliably, which is one source of confident errors ("hallucinations").
Why should I care?
As a researcher
Whether prediction objectives produce internal world models, and how far they can go without other training signals, is one of the central open questions in the field.
As an engineer
It sets expectations: fluent output reflects what is likely in the training distribution, which usually but not always coincides with what is correct.
Modern systems that depend on it
- scaling laws
- the emergence debate
- why post-training is needed (Chapter 10)
Historical context
Before
Language models were seen as tools for speech recognition and autocomplete, not as general problem solvers.
After
Next-token prediction at scale became the main route to general-purpose AI systems.
Used today
It is the pretraining objective of essentially every current LLM; post-training builds on what it produces.
What to remember
- Lower prediction loss means shorter compressed text: a good predictor is a good compressor (Shannon).
- Anything that helps predict text (a fact, a grammar rule, how a proof usually proceeds) is worth learning for the loss.
- Established: LLMs compress even images and audio well (Delétang et al. 2023). Debated: whether this amounts to 'understanding'.
- The model learns what is likely, not what is true: popular misconceptions are also predictable.
Key papers
A Mathematical Theory of Communication
C. E. Shannon · 1948 · Bell System Technical Journal
Founded information theory: it defined entropy as a measure of uncertainty and showed how much any message can be compressed. Cross-entropy loss and perplexity come straight from here.
How to read it: Don't read it cover to cover. Part I (sections 1–7) contains entropy and the famous 'series of approximations to English' — a 1948 language model.
Language Models are Unsupervised Multitask Learners
Alec Radford, Jeffrey Wu et al. · 2019 · OpenAI technical report
GPT-2: a 1.5-billion-parameter model trained on 40 GB of web text that performed tasks with no fine-tuning at all (zero-shot), just from how the prompt was phrased.
Language Modeling Is Compression
Grégoire Delétang, Anian Ruoss et al. · 2023 · ICLR 2024
Made the prediction–compression link concrete: a good predictor is a good compressor. Chinchilla 70B, trained mostly on text, compressed image patches and audio better than PNG and FLAC.