A curated set, not a bibliography. Each paper is listed with the problem it solved and what it changed, and every title, author list and link was checked against arXiv. Read the essentials first, in chronological order — they tell the story on their own.See how they connect: the family tree →
Replaced the unanswerable question 'Can machines think?' with a testable one — the imitation game — and anticipated learning machines decades early.
Problem
'Thinking' has no agreed definition, so the question of machine intelligence seemed impossible to discuss scientifically.
What was new
An operational test based on conversation, a systematic rebuttal of objections to machine intelligence, and a proposal to build 'child machines' that learn rather than being fully programmed.
Yann LeCun, Léon Bottou et al. · 1998 · Proceedings of the IEEE
The LeNet paper: convolutional networks trained end-to-end with gradient descent for handwriting recognition, deployed commercially for reading cheques.
Problem
Handwriting recognition relied on hand-designed feature extractors plus a trainable classifier.
What was new
Learn the features too: convolutional networks trained end to end, plus whole systems trained with gradients.
A short, practical essay on the lessons ML practitioners learn the hard way: generalization is what counts, data beats cleverness, and intuition fails in high dimensions.
Problem
The folk knowledge that separates successful ML projects from failed ones was rarely written down.
What was new
Twelve compact lessons — overfitting, the curse of dimensionality, feature engineering, more data vs smarter algorithms, and more.
How to read it: The best single reading for Chapter 3. Read it after the chapter; much of it will click.
~30 min readdoi:10.1145/2347736.2347755✓ verified 2026-09-26
Tomas Mikolov, Kai Chen et al. · 2013 · ICLR 2013 (workshop)
Showed that simple, fast models trained on billions of words produce word vectors whose geometry captures meaning — the idea behind every embedding you use today.
Problem
Neural language models learned good word representations but were too slow to train on very large corpora.
What was new
Two stripped-down architectures (CBOW and skip-gram) that drop the expensive hidden layer, making it practical to learn embeddings from huge datasets.
Ilya Sutskever, Oriol Vinyals, Quoc V. Le · 2014 · NeurIPS 2014
Established the encoder–decoder pattern: read an input sequence into a vector, then generate an output sequence from it. Its central weakness motivated attention.
Problem
Standard neural networks need fixed-size inputs and outputs, but translation maps sequences to sequences of different lengths.
What was new
A deep LSTM encoder compresses the source sentence into one vector; a second LSTM decodes the translation from it, trained end to end.
Introduced attention in neural networks for language: instead of squeezing a sentence into one vector, the decoder looks back at every input word and decides which ones matter right now.
Problem
Encoder–decoder models squeezed the whole source sentence into a single fixed-length vector, and translation quality fell sharply on long sentences.
What was new
A learned alignment: at each output step the model scores every encoder state, normalizes the scores with softmax, and uses the weighted average as context.
Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.
Problem
Recurrent models process tokens one after another, which limits parallel training and forces distant words to interact through many sequential steps.
What was new
Dropped recurrence entirely: stacks of multi-head self-attention and feed-forward layers, with positional encodings, residual connections and layer normalization.
How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.
~1 h 15 min readarXiv:1706.03762✓ verified 2026-09-26
PPO: a simple, robust actor–critic policy-gradient method. It became the default RL algorithm in many labs and was the optimiser in InstructGPT-style RLHF.
Problem
Policy-gradient updates that are too large can wreck a policy in a single step; the principled fixes were complicated.
What was new
A clipped objective that removes the incentive to move the policy too far from the one that collected the data, so the same batch can be reused for several updates.
Richard S. Sutton, Andrew G. Barto · 2018 · MIT Press
The standard textbook, free online from the authors. Everything in this chapter's RL half (MDPs, value functions, Q-learning, exploration, policy gradients, actor–critic) is developed carefully there.
Problem
Reinforcement learning ideas were scattered across control theory, psychology and AI.
What was new
A unified treatment built around value functions, temporal-difference learning and policy methods.
How to read it: Chapters 1, 3 and 6 cover the core: the problem, MDPs, and temporal-difference learning including Q-learning. Chapter 13 is policy gradients.
GPT-1: a 12-layer decoder-only Transformer pretrained to predict the next token on over 7,000 unpublished books, then fine-tuned. It improved the state of the art on 9 of 12 tasks and set the template for every GPT since.
Problem
Labelled data for each language task is scarce, while unlabelled text is plentiful.
What was new
Generative pretraining of a Transformer decoder on long, contiguous text, followed by supervised fine-tuning with minimal task-specific changes to the architecture.
GPT-2: a 1.5-billion-parameter model trained on 40 GB of web text that performed tasks with no fine-tuning at all (zero-shot), just from how the prompt was phrased.
Problem
Even pretrained models needed a labelled dataset and fine-tuning for every task.
What was new
Scale the same next-token objective to a bigger model and a large, diverse web dataset (WebText), with byte-level BPE, and evaluate zero-shot.
Tom B. Brown, Benjamin Mann et al. · 2020 · NeurIPS 2020
GPT-3 (175B parameters) showed that a large enough language model can perform new tasks from a few examples in its prompt, without any gradient updates.
Problem
Each new NLP task needed its own labelled dataset and fine-tuning run.
What was new
Scale a decoder-only Transformer by ~100× and evaluate zero-, one- and few-shot 'in-context learning' across many tasks.
Alexey Dosovitskiy, Lucas Beyer et al. · 2020 · ICLR 2021
Showed that a nearly unmodified Transformer, reading an image as a sequence of patches, can match strong CNNs when pretrained on enough data. Vision and language began to share one architecture.
Problem
Transformers dominated language, but vision still relied on convolutions' built-in locality.
What was new
Cut the image into 16×16 patches, embed each patch like a token, add position embeddings and run a standard Transformer encoder.
InstructGPT: the supervised fine-tuning + reward model + RL recipe that turned GPT-3 into an instruction-following assistant, and the template for ChatGPT.
Problem
Pretrained language models continue text; they don't reliably follow instructions or behave helpfully.
What was new
Fine-tune on human demonstrations, train a reward model on human rankings, then optimize the model against it with PPO.
C. E. Shannon · 1948 · Bell System Technical Journal
Founded information theory: it defined entropy as a measure of uncertainty and showed how much any message can be compressed. Cross-entropy loss and perplexity come straight from here.
Problem
There was no precise way to measure information, or to say how efficiently it could be encoded and transmitted over a noisy channel.
What was new
Entropy H = −Σ p log p as the average information of a source, plus limits on compression and reliable communication. It even includes early statistical models of English text built from n-gram frequencies.
How to read it: Don't read it cover to cover. Part I (sections 1–7) contains entropy and the famous 'series of approximations to English' — a 1948 language model.
~2 h readdoi:10.1002/j.1538-7305.1948.tb01338.x✓ verified 2026-09-26
John McCarthy, Marvin L. Minsky et al. · 1955 · Reprinted in AI Magazine 27(4), 2006
The document that named the field 'artificial intelligence' and set out its founding ambition.
Problem
Scattered work on thinking machines had no shared name, agenda or community.
What was new
A two-month study proposed on the conjecture that every aspect of learning or intelligence can in principle be described precisely enough for a machine to simulate it.
Kurt Hornik, Maxwell Stinchcombe, Halbert White · 1989 · Neural Networks
Proved that a network with a single hidden layer can approximate essentially any continuous function, given enough units — the 'universal approximation' result.
Problem
Were multi-layer networks fundamentally limited, as single-layer perceptrons had been shown to be?
What was new
A proof that standard feedforward networks with one hidden layer are universal approximators.
Christopher J. C. H. Watkins, Peter Dayan · 1992 · Machine Learning
Proved that Q-learning, which Watkins introduced in his 1989 thesis, converges to the optimal action values under stated conditions: learn the best behaviour while behaving otherwise.
Problem
How can an agent learn optimal behaviour without a model of its environment, from its own trial and error?
What was new
A convergence proof for the Q-learning update: every state–action pair is tried infinitely often and learning rates shrink appropriately.
REINFORCE: the basic policy-gradient estimator. Make the actions that led to high reward more probable. The same estimator sits inside modern RL fine-tuning of language models.
Problem
How do you follow the gradient of expected reward when the reward depends on random actions and is not differentiable?
What was new
An unbiased gradient estimate: reward (minus a baseline) times the gradient of the log-probability of the action taken.
Random forests — many decision trees trained on random subsets of data and features, then averaged — remain one of the strongest methods for tabular data.
Problem
Single decision trees overfit easily and are unstable: small data changes produce very different trees.
What was new
Average many decorrelated trees, each grown on a bootstrap sample with a random subset of features at each split.
~50 min readdoi:10.1023/A:1010933404324✓ verified 2026-09-26
Yoshua Bengio, Réjean Ducharme et al. · 2003 · Journal of Machine Learning Research
Learned word representations and next-word probabilities jointly, so similar words could help the model generalize to word sequences it had never counted.
Problem
An n-gram table sees most possible word sequences zero times, and counting alone cannot share evidence between similar words.
What was new
A neural probability model that looks up a distributed vector for each context word and learns those vectors with the prediction task.
A dataset, not a model: millions of labelled images organised by WordNet categories. Its yearly challenge became the benchmark on which deep CNNs proved themselves in 2012.
Problem
Vision datasets were too small to train or fairly compare models that learn many parameters.
What was new
Collect and label images at a much larger scale using crowdsourcing, organised into a hierarchy of categories.
Shachar Kaufman, Saharon Rosset et al. · 2012 · ACM Transactions on Knowledge Discovery from Data
Named and systematized data leakage — information in training data that won't exist at prediction time — one of the most common ways ML results turn out to be fake.
Problem
Models that looked excellent in evaluation failed in deployment because their training data contained hints about the answer.
What was new
A formal definition of leakage, a catalogue of real examples, and methods to detect and avoid it.
~45 min readdoi:10.1145/2382577.2382579✓ verified 2026-09-26
Colin Raffel, Noam Shazeer et al. · 2019 · JMLR 2020
Framed every NLP task as text in, text out, using an encoder–decoder Transformer — and ran a huge, careful set of ablations that is still a model of empirical method.
Problem
Transfer-learning results were hard to compare because every paper changed many things at once.
What was new
One text-to-text format for all tasks, plus a systematic study of objectives, architectures and data.
Explained why maximising likelihood at generation time gives 'bland and strangely repetitive' text, and introduced nucleus (top-p) sampling, now a default setting in LLM APIs.
Problem
Greedy and beam search produce degenerate, repetitive text even from a good model, while pure sampling wanders into the unreliable tail.
What was new
Sample from the smallest set of tokens whose probabilities add up to p: the 'nucleus', whose size adapts to how confident the model is.
Jason Wei, Maarten Bosma et al. · 2021 · ICLR 2022
Instruction tuning: fine-tune on many tasks phrased as instructions and the model follows instructions for new tasks too. The 137B FLAN beat zero-shot GPT-3 on 20 of 25 tasks.
Problem
Base models were good at few-shot prompting but weak at simply following an instruction with no examples.
What was new
Fine-tune a pretrained model on over 60 datasets rewritten as natural-language instructions, then test on unseen task types.
Whisper: an encoder–decoder Transformer trained on 680,000 hours of audio paired with transcripts gathered from the internet. Speech recognition became one more sequence-to-sequence problem solved by scale.
Problem
Speech recognisers trained on curated datasets were brittle on new accents, noise and domains.
What was new
Train one model on a large, noisy, multilingual weakly supervised dataset; it transcribes, translates and identifies language, and generalises well without fine-tuning.
The counter-argument: many apparent jumps come from all-or-nothing metrics such as exact match. Scored with continuous metrics, the same models improve smoothly.
Problem
Were emergent abilities a property of the models or of how researchers measured them?
What was new
A simple mathematical model plus re-analyses showing that nonlinear or discontinuous metrics can manufacture sharp transitions.
Showed that smaller models trained on more tokens, using only publicly available data, can rival much larger ones, and released weights to researchers, starting the open-weight wave.
Problem
The strongest language models were closed, and very large.
What was new
Models from 7B to 65B parameters trained on trillions of tokens of public data; the 13B model outperformed GPT-3 (175B) on most benchmarks reported.
Documented a large jump in capability, including human-level scores on many professional and academic exams, and marked the point where frontier labs stopped disclosing model size, data and training details.
Problem
How to report a frontier model's capabilities and risks when training details are withheld?
What was new
A multimodal (image and text input) model, predictable scaling of loss from much smaller runs, and a system card on safety work.
An openly released reasoning model, with a detailed account of training long chains of reasoning mainly through reinforcement learning on verifiable problems.
Problem
How reasoning models were trained was largely undisclosed.
What was new
Large-scale RL with rule-based rewards on math and code, plus distillation of the resulting reasoning into smaller open models.
D. H. Hubel, T. N. Wiesel · 1962 · The Journal of Physiology
Neuroscience, not machine learning, but it described visual neurons that respond to oriented edges in a small patch of the visual field: the idea behind local, edge-detecting filters.
Problem
How does the visual cortex turn light falling on the retina into responses to shapes?
What was new
Recordings from single neurons showing small receptive fields, orientation selectivity, and 'simple' and 'complex' cells arranged in a hierarchy.
An early layered network with local feature detectors and pooling-like stages, built to recognise a pattern wherever it appears: the architectural ancestor of the CNN.
Problem
A pattern recogniser should not have to relearn a shape for every position it might appear in.
What was new
Alternating layers of local feature-extracting cells and cells that tolerate small shifts, inspired by Hubel and Wiesel's simple and complex cells.
Geoffrey Hinton, Li Deng et al. · 2012 · IEEE Signal Processing Magazine
Four research groups reported that deep networks beat the Gaussian mixture models that had powered speech recognisers for decades. Speech fell to deep learning in the same years as vision.
Problem
Acoustic models based on Gaussian mixtures had plateaued.
What was new
Replace the Gaussian mixture acoustic model with a deep feed-forward network that predicts speech-state probabilities from a window of frames.
~40 min readdoi:10.1109/MSP.2012.2205597✓ verified 2026-09-26
Olaf Ronneberger, Philipp Fischer, Thomas Brox · 2015 · MICCAI 2015
A labelling for every pixel: an encoder that shrinks the image and a decoder that grows it back, with skip connections carrying fine detail across. The U shape later became the backbone of many diffusion image generators.
Problem
Segmentation needs both context (what is this?) and precise location (exactly which pixels?), and biomedical training sets are small.
What was new
A symmetric contracting and expanding network whose skip connections copy high-resolution features to the matching decoder stage.
Matthew E. Peters, Mark Neumann et al. · 2018 · NAACL 2018
ELMo: word vectors that change with the sentence, taken from a pretrained bidirectional LSTM language model. 'Bank' by a river and 'bank' with an account finally got different vectors.
Problem
Word2vec and GloVe give each word one vector regardless of context.
What was new
Use the internal states of a language model pretrained on a large corpus as contextual features for downstream models.
Named the shift: a single model trained on broad data and adapted to many tasks, a 'foundation model'. A long survey of capabilities, applications and risks.
Problem
The field lacked shared vocabulary for models like BERT and GPT-3 that serve as a base for many downstream systems.
What was new
The term 'foundation model' and a broad analysis of homogenisation, emergence and societal impact, from over a hundred authors at Stanford.
A surprising result: replacing the labels in few-shot examples with random ones barely hurt performance. The examples mainly showed the format, the label space and the kind of input.
Problem
Nobody knew which parts of few-shot demonstrations actually drive in-context learning.
What was new
Controlled experiments across 12 models that vary labels, inputs and format independently.
Proposed a concrete mechanism: 'induction heads', attention heads that complete [A][B] … [A] → [B]. Their appearance during training coincides with a jump in in-context learning.
Problem
In-context learning was observed but its mechanism inside the network was unknown.
What was new
Six lines of evidence linking induction heads to in-context learning: causal in small attention-only models, correlational in larger ones.
Made the prediction–compression link concrete: a good predictor is a good compressor. Chinchilla 70B, trained mostly on text, compressed image patches and audio better than PNG and FLAC.
Problem
Is 'next-token prediction is compression' just a slogan, or measurable?
What was new
Use large language models with arithmetic coding as lossless compressors and compare them with standard compressors across text, images and audio.