Concept · Chapter 8: The Rise of Large Language Models
Parameters, Tokens and Context Windows
Three numbers describe a language model's scale: how many learned parameters it has, how many tokens it was trained on, and how many tokens it can attend to at once (its context window).
The problem
Model announcements quote sizes like '7B' or '128k context' that are easy to confuse and to over-interpret.
The solution
Separate the three: parameters are capacity, training tokens are experience, and the context window is working memory at inference time.
The consequence
You can read a model card sensibly, estimate memory and cost, and see why more parameters alone do not make a better model.
You should understand first
- Text as Data
- Vectors
- One-Hot Encoding
- Tokenization
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Dot Product
- Embeddings
- Attention
- Softmax
- Self-Attention
- Causal Masking
- Entropy
- Loss Functions
- Cross-Entropy Loss
- Autoregressive Next-Token Prediction
- Pretraining at Scale
- Parameters, Tokens and Context Windows
Three different numbers
- Parameters are the model's learned weights: the embedding table, attention and feed-forward matrices in every layer. They are its capacity. In 16-bit precision each takes 2 bytes, so a 7-billion-parameter model's weights alone take about 14 GB.
- Training tokens measure how much text the model learned from. The same 175B-parameter architecture trained on 30 billion or 300 billion tokens would be a very different model.
- Context window is how many tokens the model can process in one go: the prompt, any documents pasted in, and the answer it writes. It is working memory at inference time, not knowledge: text outside the window is simply not seen.
Bigger is not automatically better
In 2022 the Chinchilla study found that many large models of the time were undertrained for their size, and that a 70B model trained on about four times more data outperformed the 280B Gopher Established. In 2023 LLaMA-13B, trained on over a trillion tokens of public data, outperformed the 175B GPT-3 on most benchmarks its authors reported Established. Parameters, data and compute have to be balanced; scaling laws describe how.
What to remember
- Parameters: learned weights. '7B' = 7 billion. Roughly 2 bytes each in 16-bit precision, so a 7B model needs about 14 GB just for weights.
- Training tokens: how much text the model learned from. GPT-3: 300B; LLaMA (2023): trillions.
- Context window: the maximum tokens per request (prompt + output). GPT-2: 1,024; GPT-3: 2,048; recent models: far more.
- A smaller model trained on more tokens can beat a larger one: LLaMA-13B beat GPT-3 175B on most reported benchmarks.
Key papers
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann et al. · 2020 · NeurIPS 2020
GPT-3 (175B parameters) showed that a large enough language model can perform new tasks from a few examples in its prompt, without any gradient updates.
How to read it: 75 pages. Sections 1–2 and Figure 1.2 carry the core idea; Section 6 on broader impacts is worth reading too.
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud et al. · 2022 · NeurIPS 2022
Showed that many large models were undertrained: for a fixed compute budget, parameters and training tokens should grow roughly in proportion.
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril et al. · 2023
Showed that smaller models trained on more tokens, using only publicly available data, can rival much larger ones, and released weights to researchers, starting the open-weight wave.