Concept · Chapter 8: The Rise of Large Language Models
GPT-1 → GPT-2 → GPT-3
Between 2018 and 2020 OpenAI kept the same recipe (a decoder-only Transformer trained on next-token prediction) and scaled it about a thousandfold, and the way the models were used changed from fine-tuning to zero-shot to few-shot prompting.
The problem
Was next-token prediction just a pretraining trick before fine-tuning, or could the model itself do tasks?
The solution
Scale model size, data and context, and evaluate by describing tasks in the prompt instead of fine-tuning.
The consequence
GPT-3 showed that a single large model could perform many tasks from a few examples in its prompt, which redirected the field toward scale and prompting.
You should understand first
- Text as Data
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Vectors
- Dot Product
- Embeddings
- Attention
- Softmax
- Self-Attention
- Causal Masking
- Entropy
- Loss Functions
- Cross-Entropy Loss
- Autoregressive Next-Token Prediction
- Pretraining at Scale
- GPT-1 → GPT-2 → GPT-3
Three models, one recipe
| GPT-1 (2018) | GPT-2 (2019) | GPT-3 (2020) | |
|---|---|---|---|
| Parameters | about 117M (12 layers) | up to 1.5B (48 layers) | 175B |
| Data | 7,000+ unpublished books | WebText, about 40 GB | 300B training tokens from filtered web, books, Wikipedia |
| Context | 512 tokens | 1,024 tokens | 2,048 tokens |
| How it was used | pretrain, then fine-tune per task | zero-shot: describe the task in the prompt | few-shot: show examples in the prompt |
The architecture barely changed: a stack of decoder blocks with a causal mask, trained on next-token prediction. What changed was scale, roughly a thousandfold in parameters, and the question researchers asked of the model.
The shift in how models were used
- GPT-1 still fine-tuned on each task and beat the state of the art on 9 of the 12 tasks studied Established.
- GPT-2 reached state-of-the-art results on 7 of 8 language-modelling benchmarks with no fine-tuning (zero-shot) Established. It also did rough versions of translation, summarisation and question answering when the prompt was phrased the right way, for example ending an article with "TL;DR:" to get a summary. OpenAI initially released only smaller versions, citing concerns about misuse Established, a first public argument about releasing a language model.
- GPT-3 performed many tasks from a handful of examples placed in the prompt, with no gradient updates, sometimes matching fine-tuned systems Established. This is in-context learning.
What to remember
- GPT-1 (2018): 12 layers, books corpus; pretrain then fine-tune; better on 9 of 12 tasks.
- GPT-2 (2019): 1.5B parameters, 40 GB of web text, 1,024-token context; zero-shot tasks from the prompt.
- GPT-3 (2020): 175B parameters, 300B training tokens, 2,048-token context; few-shot in-context learning.
- Same architecture family throughout; what changed was scale and how the model was used.
Key papers
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann et al. · 2020 · NeurIPS 2020
GPT-3 (175B parameters) showed that a large enough language model can perform new tasks from a few examples in its prompt, without any gradient updates.
How to read it: 75 pages. Sections 1–2 and Figure 1.2 carry the core idea; Section 6 on broader impacts is worth reading too.
Improving Language Understanding by Generative Pre-Training
Alec Radford, Karthik Narasimhan et al. · 2018 · OpenAI technical report
GPT-1: a 12-layer decoder-only Transformer pretrained to predict the next token on over 7,000 unpublished books, then fine-tuned. It improved the state of the art on 9 of 12 tasks and set the template for every GPT since.
Language Models are Unsupervised Multitask Learners
Alec Radford, Jeffrey Wu et al. · 2019 · OpenAI technical report
GPT-2: a 1.5-billion-parameter model trained on 40 GB of web text that performed tasks with no fine-tuning at all (zero-shot), just from how the prompt was phrased.
Watch
Andrej Karpathy
Let's reproduce GPT-2 (124M)
Build and train the smallest GPT-2 from scratch in PyTorch, then optimise it. For when you want to implement what this chapter describes.