Skip to content
Road to Intelligence

Concept · Chapter 8: The Rise of Large Language Models

In-Context Learning

Must knowKnow well15 minDifficulty

In-context learning is a model performing a new task from instructions or a few examples placed in its prompt, with no change to its weights.

The problem

Fine-tuning for every task needs labelled data, training runs and a separate copy of the model.

The solution

Describe the task in the prompt (zero-shot) or show a few input–output examples (few-shot), and let the pretrained model continue the pattern.

The consequence

One frozen model serves countless tasks, and 'programming' it becomes writing prompts, though results depend on wording and remain less predictable than training.

You should understand first

  1. Text as Data
  2. Probability and Distributions
  3. Conditional Probability and Bayes' Theorem
  4. Probability of Sequences
  5. Language Modeling
  6. Vectors
  7. Dot Product
  8. Embeddings
  9. Attention
  10. Softmax
  11. Self-Attention
  12. Causal Masking
  13. Entropy
  14. Loss Functions
  15. Cross-Entropy Loss
  16. Autoregressive Next-Token Prediction
  17. Pretraining at Scale
  18. GPT-1 → GPT-2 → GPT-3
  19. In-Context Learning

Intuition: continue the pattern

A pretrained model continues documents. So write a document whose natural continuation is the answer you want:

English: sea otter → French: loutre de mer
English: cheese → French: fromage
English: peppermint → French:

The most likely next tokens are a French word. No one trained the model on this task; the prompt set it up. That is few-shot prompting. Drop the examples and just write "Translate English to French: peppermint →" and it is zero-shot.

What the examples actually do

GPT-3's authors reported that few-shot performance improved steeply with model size, and on some tasks approached fine-tuned systems Established. But the examples may not teach what you think. Min and colleagues (2022) found that replacing the labels in few-shot examples with random ones barely hurt accuracy across 12 models; what mattered more was showing the label set, the kind of input and the format Established. Much of in-context learning seems to be locating a task the model already learned in pretraining, rather than learning a new one from scratch Interpretation, though larger models can also pick up genuinely new mappings from examples.

How can a frozen network learn?

The weights do not change; everything happens in one forward pass through self-attention, where later tokens can read earlier ones. Olsson and colleagues proposed that "induction heads" (attention heads that complete a pattern [A][B] … [A] → [B]) may be a major mechanism behind in-context learning; the evidence is causal in small models and correlational in large ones Active research.

Why should I care?

As a researcher

How a fixed network 'learns' from its input is an open scientific question, with mechanistic (induction heads) and theoretical accounts under active study.

As an engineer

Few-shot examples, format demonstrations and instructions are the cheapest way to adapt a model, and knowing their limits tells you when to fine-tune or retrieve instead.

Modern systems that depend on it

  • prompt engineering
  • retrieval-augmented generation (Chapter 12)
  • agents (Chapter 13)

Historical context

Before

Every new task needed a labelled dataset and a fine-tuning run.

After

A task can be specified in plain text at inference time; the weights never change.

Used today

Every prompt with instructions or examples relies on it, from classification templates to tool-use formats.

What to remember

  • Zero-shot: instructions only. One-shot: one example. Few-shot: several examples, then the new input.
  • No gradient updates: everything happens in the forward pass, through attention over the prompt.
  • Examples mostly teach the format, label set and kind of input; even random labels often barely hurt (Min et al. 2022).
  • It grew markedly stronger with scale (GPT-3), but is sensitive to wording and example order.
  • Induction heads are one proposed mechanism (active research).

Key papers

Essential

Language Models are Few-Shot Learners

Tom B. Brown, Benjamin Mann et al. · 2020 · NeurIPS 2020

GPT-3 (175B parameters) showed that a large enough language model can perform new tasks from a few examples in its prompt, without any gradient updates.

How to read it: 75 pages. Sections 1–2 and Figure 1.2 carry the core idea; Section 6 on broader impacts is worth reading too.

~1 h 30 min readarXiv:2005.14165✓ verified 2026-09-26
Optional

In-context Learning and Induction Heads

Catherine Olsson, Nelson Elhage et al. · 2022

Proposed a concrete mechanism: 'induction heads', attention heads that complete [A][B] … [A] → [B]. Their appearance during training coincides with a jump in in-context learning.

~1 h readarXiv:2209.11895✓ verified 2026-09-26

Watch