The Rise of Large Language Models

The problem

One model for every task

By 2017 the Transformer existed, but language AI was still a collection of specialists. A sentiment classifier, a question answerer and a translation system were separate models, each trained on its own labelled dataset. Labels are slow and expensive to collect. Unlabelled text, on the other hand, was nearly unlimited.

Vision had already found a way around this: pretrain a network on ImageNet, then fine-tune it for a new task with a small dataset (Chapter 5). In 2018 language caught up. The idea was to pretrain a network on raw text with a task the text supplies itself, then fine-tune it on a little labelled data. ELMo, ULMFiT, OpenAI's GPT and Google's BERT all appeared that year Established. BERT, an encoder trained to fill in masked words, topped the benchmarks. GPT, a decoder trained to predict the next token, looked like the weaker cousin.

This chapter follows the weaker cousin. With the recipe barely changed, it was scaled about a thousandfold, and it turned out that one model could do many tasks with no fine-tuning at all.

The input

Text becomes tokens

A language model reads numbers, so text must be cut into units first. Whole words make the vocabulary open-ended, since every new name or typo is unknown. Single characters make sequences very long. Modern models use subword tokens: common words are single tokens, and rare words are spelled from a few pieces. Each token has an integer ID, and each ID selects one row of the model's embedding table.

The pieces are learned with byte-pair encoding (BPE). Start from single characters, count every adjacent pair in the corpus, merge the most frequent pair into a new symbol, and repeat. Train one below, on the tiny example from the paper that introduced BPE for translation. After five merges, the word lowest, which never appears in the corpus, is spelled from learned pieces as low + est_.

Real tokenizers do exactly this on bytes, over huge corpora, until the vocabulary holds tens of thousands of pieces. GPT-2 used a byte-level BPE vocabulary of 50,257 tokens Established. Here are the real GPT-2 and GPT-4 tokenizers running in your browser. Try the Devanagari example, then numbers, then ask how many r's are in strawberry.

Several well-known LLM quirks come straight from this step. The model never sees letters, so spelling questions are about something it never observes directly. Long numbers split into irregular chunks, so digit-by-digit arithmetic is awkward. And text in languages the vocabulary serves poorly costs more tokens, so it is more expensive and fills the context window faster.

The objective

Predict the next token

A GPT model is Chapter 7's decoder-only Transformer with one job. Read the tokens so far, and output a probability for every token in the vocabulary as the next one. Training compares that prediction with the token that actually came next, using cross-entropy. Thanks to the causal mask, a single pass over a 2,000-token document gives 2,000 predictions to learn from. No labels are needed, because the text provides its own answers. This is self-supervised learning, and it is why the training data can be as large as the web.

$$P(x_1,\ldots,x_T)=\prod_{t=1}^{T}P_\theta(x_t\mid x_1,\ldots,x_{t-1})$$

This is the same chain rule as the n-gram models of Chapter 6. The difference is that the context is now thousands of tokens handled by attention, rather than two words looked up in a count table.

Generating text runs the model in a loop: predict a distribution, choose a token, append it, and predict again. That is why chat replies appear word by word.

Train this objective over hundreds of billions of tokens of books, web pages and code, and you get a base model. It is a general-purpose text continuer that has absorbed a great deal of language, knowledge and style, along with its data's gaps and biases. A 2021 Stanford report called such broadly trained, widely adaptable models "foundation models" Established.

The scale-up

GPT-1 to GPT-3

The architecture barely changed. What changed was scale, and with it the way people used the model. GPT-1 was fine-tuned for each task and improved the state of the art on 9 of 12 Established. GPT-2 needed no fine-tuning to reach state-of-the-art results on 7 of 8 language-modelling benchmarks Established. It also produced rough summaries when an article was followed by "TL;DR:". GPT-3 could perform many tasks from just a few examples placed in its prompt.

These three numbers are easy to confuse. Parameters are capacity. Training tokens are experience. The context window is working memory during a single request. More of one does not make up for too little of another.

A new way to use a model

Learning from the prompt

GPT-3's paper was titled Language Models are Few-Shot Learners. Instead of fine-tuning, you write a prompt whose most natural continuation is the answer:

English: sea otter → French: loutre de mer
English: cheese → French: fromage
English: peppermint → French:

The model's weights do not change. It "learns" the task inside a single forward pass, as attention reads the examples. This is in-context learning. It grew much stronger with model size Established, and it turned programming a model into writing text. The examples may not teach what you would expect, though. Replacing the labels in few-shot examples with random ones often barely hurts accuracy Established, which suggests the examples mostly show the format and the kind of task. How transformers learn in context, and whether specific attention circuits ("induction heads") are the main mechanism, is still being studied Active research.

The puzzle

Why prediction goes so far

How can "guess the next word" produce translation, code and step-by-step explanations? One answer is that to predict human text well, you have to capture whatever produced it. Predicting the end of a proof is easier if you follow the proof. Predicting the next line of a program is easier if you know what the code does. Predicting "the capital of Nepal is…" is easier if you know the fact. Each regularity a model captures lowers its loss, so scale and diverse data push it to capture a great many Interpretation.

This has a precise form: prediction is compression. A model that gives the true next token probability p can encode it in about −log₂ p bits. Language models used as compressors beat standard compressors, even on images and audio Established.

The same view shows the limits. The objective rewards what is likely in the data, not what is true. A popular misconception is easy to predict, and a rare fact is learned weakly. Whether this kind of prediction amounts to understanding is still argued about Interpretation.

Scale as a plan

Bigger, predictably better

In 2020, Kaplan and colleagues found that language-model loss falls as a smooth power law in parameters, data and compute, across more than seven orders of magnitude Established. On a log–log plot, those trends are straight lines, so a lab could predict a large model's loss from small experiments and budget accordingly. Chinchilla (2022) corrected the recommended balance toward far more data per parameter Established. Chapter 9 works through the numbers.

Smooth loss curves raised a puzzle, because some benchmark scores seemed to switch on suddenly at a certain size. Are these emergent abilities real thresholds? Or do they appear because an all-or-nothing metric such as exact match turns steady gains into a jump? The answer is still debated Active research, and the concept card shows how much a metric can change the picture.

The last step

Choosing the next token

The model's output is a probability distribution, not a word. Something has to choose. Always taking the top token (greedy decoding) is predictable, but it gives the same bland answer every time. Sampling in proportion to the probabilities adds variety, but occasionally it picks something absurd from the long tail of unlikely tokens.

Three dials manage this. Temperature sharpens or flattens the distribution. Top-k keeps only the k likeliest tokens. Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to p. Try them below on a toy model. Raise the temperature to 1.5 and watch unlikely words creep into the sentences, then set top-p to 0.9 and watch them disappear. Push the temperature higher still and a few get through again, because the flattened tail now holds enough probability to survive the cut.

From model to product

From base model to ChatGPT

Ask a base model "What is the capital of France?" and it may continue with "What is the capital of Germany?", as a web page of quiz questions would. It models documents. It does not answer.

Two more training stages close the gap, and Chapter 10 covers them in detail. Instruction tuning fine-tunes on many tasks written as instructions with good responses. RLHF then optimises the model for responses people prefer, using the policy-gradient methods from Chapter 5. In InstructGPT, people preferred a tuned 1.3-billion-parameter model's answers to those of the 175-billion-parameter GPT-3 base Established. ChatGPT, released on 30 November 2022, put this behind a chat box and reached a mass audience within weeks Established.

Why it matters

Why it matters

This chapter holds the central fact of modern AI. One simple, self-supervised objective, applied at scale to a general architecture, produced models that could do a wide range of tasks. It did so without task-specific design, and without anyone fully predicting in advance what they would do.

• For an engineer, the practical consequences are tokens (cost, limits, quirks), prompts (in-context learning), decoding settings (temperature and top-p) and the difference between base and chat models.
• For a researcher, the open questions start here. Why does prediction produce capability? How does in-context learning work inside the network? Are new abilities predictable? How much of an assistant's behaviour comes from pretraining, and how much from tuning?

What comes next

Inside the machine

We have treated training as "predict the next token at scale". Chapter 9 opens that box: where the trillions of tokens come from and how they are cleaned, how tokenizers are built for real, what the loss curve and learning-rate schedule look like, how a model too big for one GPU is split across thousands, and what the scaling laws actually recommend. Chapter 10 then turns a base model into an assistant.