Concept · Chapter 8: The Rise of Large Language Models
Pretraining at Scale
Pretraining runs next-token prediction over a very large, diverse text corpus, producing a base model that has absorbed patterns of language, facts and reasoning styles from its data.
The problem
Labelled data is scarce, but capable language use seems to require exposure to an enormous range of text.
The solution
Collect hundreds of billions to trillions of tokens (web pages, books, code, papers), filter and deduplicate them, and train a large causal Transformer on next-token prediction.
The consequence
The resulting base model is general-purpose but raw: it continues text rather than following instructions, and it inherits its data's knowledge, gaps and biases.
You should understand first
What goes in
Pretraining data is text at the largest scale a team can gather and clean: filtered web crawls, books, encyclopedias, scientific papers, and code. GPT-2 was trained on WebText, about 40 GB of text from over 8 million web pages Established; every GPT-3 model was trained on 300 billion tokens Established. Later models train on many trillions. Collecting, filtering, deduplicating and mixing this data is a large part of the work (Chapter 9).
What comes out
A base model. Give it the start of a document and it writes a plausible continuation. Ask it a question and it may answer, or it may continue with more questions, as if it were completing a quiz page. It has no built-in notion of being helpful; it models text.
To predict text well across so many sources, the model must pick up grammar, facts, the style of different genres, common reasoning patterns and much more Interpretation. Why prediction goes so far looks at that argument and its limits.
In 2021 a Stanford report proposed the term "foundation model" for models trained on broad data that can be adapted to many downstream tasks Established. The phrase stuck because it captures the new economics: pretrain once at great cost, adapt many times cheaply.
What to remember
- Objective: next-token prediction. Data: as much diverse text as possible, filtered and deduplicated.
- GPT-2: about 40 GB of web text. GPT-3: 300 billion training tokens. Recent models: many trillions.
- The output is a base model: it continues documents; it is not yet an assistant.
- A model can only know what its data contains, up to its training cutoff.
- 'Foundation model' (2021): one broadly trained model adapted to many tasks.
Key papers
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann et al. · 2020 · NeurIPS 2020
GPT-3 (175B parameters) showed that a large enough language model can perform new tasks from a few examples in its prompt, without any gradient updates.
How to read it: 75 pages. Sections 1–2 and Figure 1.2 carry the core idea; Section 6 on broader impacts is worth reading too.
Improving Language Understanding by Generative Pre-Training
Alec Radford, Karthik Narasimhan et al. · 2018 · OpenAI technical report
GPT-1: a 12-layer decoder-only Transformer pretrained to predict the next token on over 7,000 unpublished books, then fine-tuned. It improved the state of the art on 9 of 12 tasks and set the template for every GPT since.
Language Models are Unsupervised Multitask Learners
Alec Radford, Jeffrey Wu et al. · 2019 · OpenAI technical report
GPT-2: a 1.5-billion-parameter model trained on 40 GB of web text that performed tasks with no fine-tuning at all (zero-shot), just from how the prompt was phrased.
On the Opportunities and Risks of Foundation Models
Rishi Bommasani, Drew A. Hudson et al. · 2021
Named the shift: a single model trained on broad data and adapted to many tasks, a 'foundation model'. A long survey of capabilities, applications and risks.
How to read it: Read the introduction (section 1) only; the rest is a reference to dip into.
Watch
Andrej Karpathy
Deep Dive into LLMs like ChatGPT
A long, general-audience walk through the whole pipeline behind a chat model, from internet text to tokens to pretraining to post-training.
Andrej Karpathy
Let's reproduce GPT-2 (124M)
Build and train the smallest GPT-2 from scratch in PyTorch, then optimise it. For when you want to implement what this chapter describes.