Skip to content
Road to Intelligence

Concept · Chapter 8: The Rise of Large Language Models

Pretrain, Then Fine-Tune

Must knowUnderstand10 minDifficulty

Train one model on a huge amount of unlabelled text first, then adapt it to each task with a small labelled dataset; around 2018 this replaced training every NLP model from scratch.

The problem

Every language task (sentiment, question answering, entailment) needed its own labelled dataset and its own model, and labels are expensive.

The solution

Learn general knowledge of language from raw text with a self-supervised objective, then fine-tune the whole pretrained network on the task.

The consequence

Small labelled datasets became enough, benchmark scores jumped across NLP, and the pretrained model became the valuable asset.

You should understand first

  1. Text as Data
  2. Probability and Distributions
  3. Conditional Probability and Bayes' Theorem
  4. Probability of Sequences
  5. Language Modeling
  6. Vectors
  7. Dot Product
  8. Embeddings
  9. Attention
  10. Softmax
  11. Self-Attention
  12. Multi-Head Attention
  13. Causal Masking
  14. Positional Encoding
  15. Residual Connections
  16. Layer Normalization
  17. Feed-Forward Sublayer (MLP)
  18. The Transformer Block
  19. Encoder, Decoder & Encoder–Decoder
  20. Pretrain, Then Fine-Tune

The idea

Before 2018, a sentiment classifier and a question-answering system were separate models, each trained from (nearly) scratch on its own labelled data. Word vectors from word2vec helped a little, but everything above the first layer was learned anew every time.

The new recipe borrowed from vision's ImageNet pretraining:

  1. Pretrain

    Train a large network on raw text with a task the text supplies itself: predict the next word, or fill in hidden words. No labels needed, so the data can be enormous.

  2. Fine-tune

    Add a small output layer and keep training the whole network on a few thousand labelled examples of the actual task.

2018, the breakthrough year

  • ELMo took contextual word vectors from a pretrained bidirectional LSTM language model Established, so "bank" got a different vector by a river than in finance.
  • ULMFiT showed that fine-tuning an entire pretrained LSTM language model works well for text classification Established.
  • GPT-1 pretrained a 12-layer Transformer decoder on next-token prediction over more than 7,000 unpublished books, then fine-tuned it, improving the state of the art on 9 of 12 tasks Established.
  • BERT pretrained a Transformer encoder by predicting randomly masked words (15% of tokens) and set new results on eleven NLP tasks Established.

BERT's encoder dominated benchmarks for the next two years. The decoder route, next-token prediction, looked like the weaker cousin. The rest of this chapter is the story of how that changed.

What to remember

  • Stage 1, pretrain on unlabelled text (cheap data, expensive compute). Stage 2, fine-tune on a small labelled set.
  • 2018: ELMo (contextual vectors), ULMFiT (fine-tune a language model), GPT-1 (Transformer decoder), BERT (Transformer encoder).
  • BERT pretrains by filling in masked words; GPT by predicting the next token.
  • Vision had done this since ImageNet; 2018 was NLP's version.

Key papers

Optional

Deep contextualized word representations

Matthew E. Peters, Mark Neumann et al. · 2018 · NAACL 2018

ELMo: word vectors that change with the sentence, taken from a pretrained bidirectional LSTM language model. 'Bank' by a river and 'bank' with an account finally got different vectors.

~30 min readarXiv:1802.05365✓ verified 2026-09-26
Optional

Universal Language Model Fine-tuning for Text Classification

Jeremy Howard, Sebastian Ruder · 2018 · ACL 2018

ULMFiT showed that 'pretrain a language model, then fine-tune the whole thing' works for NLP the way ImageNet pretraining worked for vision.

~30 min readarXiv:1801.06146✓ verified 2026-09-26
Essential

Improving Language Understanding by Generative Pre-Training

Alec Radford, Karthik Narasimhan et al. · 2018 · OpenAI technical report

GPT-1: a 12-layer decoder-only Transformer pretrained to predict the next token on over 7,000 unpublished books, then fine-tuned. It improved the state of the art on 9 of 12 tasks and set the template for every GPT since.

~30 min read✓ verified 2026-09-26