Concept · Chapter 8: The Rise of Large Language Models
From Base Model to ChatGPT
A pretrained base model continues text; instruction tuning and reinforcement learning from human feedback turn it into an assistant that answers questions and follows requests, which is what ChatGPT made public in 2022.
The problem
Base models are powerful but awkward: asked a question they may continue with more questions, ramble, or produce harmful text, and they must be coaxed with carefully built prompts.
The solution
Fine-tune on many tasks phrased as instructions with good responses (instruction tuning), then optimise for responses people prefer (RLHF).
The consequence
A much smaller tuned model can be preferred over a far larger base model, and a conversational interface put LLMs in front of the public.
You should understand first
- Text as Data
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Vectors
- Dot Product
- Embeddings
- Attention
- Softmax
- Self-Attention
- Causal Masking
- Entropy
- Loss Functions
- Cross-Entropy Loss
- Autoregressive Next-Token Prediction
- Pretraining at Scale
- GPT-1 → GPT-2 → GPT-3
- In-Context Learning
- From Base Model to ChatGPT
Same question, two models
Give a base model the prompt "What is the capital of France?" and it might reply "What is the capital of Germany? What is the capital of Italy?", because lists of quiz questions are common on the web. A chat model replies "Paris." Both are next-token predictors; they differ in what they were tuned to treat as a good continuation.
Two tuning steps (a preview of Chapter 10)
Instruction tuning
Fine-tune on many examples of instructions paired with good responses. FLAN (2021) instruction-tuned a 137B model on over 60 datasets rewritten as instructions, and it beat zero-shot GPT-3 on 20 of 25 unseen tasks evaluated Established.Learning from preferences (RLHF)
Collect human rankings of several responses, train a reward model to predict them, and optimise the language model against that reward with a policy-gradient method. In InstructGPT (2022), human evaluators preferred outputs of a 1.3B-parameter tuned model to those of the 175B GPT-3 Established.
A common summary: pretraining supplies most of the knowledge and skill, and post-training mainly shapes behaviour, format and tone Interpretation. Chapter 10 tests that summary.
The ChatGPT moment
ChatGPT was released on 30 November 2022 as a chat interface to an RLHF-tuned model, and reached a mass audience within weeks Established. GPT-4 followed in March 2023, reporting human-level performance on many professional and academic exams while withholding details of its size, data and training Established. Open-weight models such as LLaMA arrived the same year. The field's centre of gravity moved from research benchmarks to deployed products, and from open papers to partially closed systems.
What to remember
- Base model: continues documents. Chat model: pretrained base + instruction tuning + preference tuning (RLHF).
- FLAN (2021): instruction tuning on 60+ tasks improved zero-shot instruction following on unseen tasks.
- InstructGPT (2022): people preferred the 1.3B tuned model's outputs to those of the 175B GPT-3 base.
- ChatGPT (November 2022) wrapped this in a chat interface and reached a mass audience within weeks.
- Post-training changes behaviour more than knowledge: capability mostly comes from pretraining (Chapter 10).
Key papers
Deep reinforcement learning from human preferences
Paul Christiano, Jan Leike et al. · 2017 · NeurIPS 2017
Showed that agents can be trained from human comparisons between behaviours rather than a hand-written reward — the foundation of RLHF.
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu et al. · 2022 · NeurIPS 2022
InstructGPT: the supervised fine-tuning + reward model + RL recipe that turned GPT-3 into an instruction-following assistant, and the template for ChatGPT.
How to read it: Figure 2 is the three-step RLHF pipeline you'll meet in Chapter 10.
Finetuned Language Models Are Zero-Shot Learners
Jason Wei, Maarten Bosma et al. · 2021 · ICLR 2022
Instruction tuning: fine-tune on many tasks phrased as instructions and the model follows instructions for new tasks too. The 137B FLAN beat zero-shot GPT-3 on 20 of 25 tasks.
GPT-4 Technical Report
OpenAI et al. · 2023
Documented a large jump in capability, including human-level scores on many professional and academic exams, and marked the point where frontier labs stopped disclosing model size, data and training details.
How to read it: Note what the report does not contain: architecture, parameter count, data and compute are all withheld.
Watch
Andrej Karpathy
Deep Dive into LLMs like ChatGPT
A long, general-audience walk through the whole pipeline behind a chat model, from internet text to tokens to pretraining to post-training.