Skip to content
Road to Intelligence

Concept · Chapter 8: The Rise of Large Language Models

Autoregressive Next-Token Prediction

Must knowKnow well15 minDifficulty

A GPT-style model reads the tokens so far and outputs a probability for every possible next token; generating text means picking one, appending it, and repeating.

The problem

We want one training objective that uses raw text with no labels and one procedure that can produce any kind of text.

The solution

Train a causal Transformer to predict each next token with cross-entropy loss; at generation time, sample a token from its output distribution, append it to the input and run again.

The consequence

Every position in every document becomes a training example, and one model can continue any text, which later turns out to include answering questions and following instructions.

You should understand first

  1. Text as Data
  2. Probability and Distributions
  3. Conditional Probability and Bayes' Theorem
  4. Probability of Sequences
  5. Language Modeling
  6. Vectors
  7. Dot Product
  8. Embeddings
  9. Attention
  10. Softmax
  11. Self-Attention
  12. Causal Masking
  13. Entropy
  14. Loss Functions
  15. Cross-Entropy Loss
  16. Autoregressive Next-Token Prediction

Intuition: autocomplete, all the way down

Your phone's keyboard suggests the next word. A GPT model does the same thing with a much longer memory and a much better model of text: given everything so far, it assigns a probability to each of its tens of thousands of tokens. To write a paragraph, it predicts, commits to one token, and predicts again with that token included.

Training: every position is a lesson

Take the sentence "the cat sat on the mat", tokenized. With a causal mask, one forward pass produces a prediction at every position at once:

SeesShould predict
thecat
the catsat
the cat saton
……

The loss is the average cross-entropy: −log⁡P(actual next token)-\log P(\text{actual next token}) at each position. If the model gave "mat" probability 0.4, that position costs −ln⁡0.4≈0.92-\ln 0.4 \approx 0.92 nats. A document of 2,000 tokens provides 2,000 of these lessons, with labels that came free with the text. This is self-supervised learning.

L=−1T∑t=1Tlog⁡Pθ(xt∣x1,…,xt−1)\mathcal{L} = -\frac{1}{T}\sum_{t=1}^{T}\log P_\theta(x_t \mid x_1,\ldots,x_{t-1})

Generation: one token at a time

  1. Predict

    Run the model on the tokens so far; take the final position's logits and apply softmax.

  2. Choose

    Pick a token: the most likely one (greedy) or a random draw shaped by temperature, top-k or top-p (decoding).

  3. Append and repeat

    Add the chosen token to the input. Stop at an end-of-text token or a length limit.

Try it · toy model

Choose the Next Word

Turn the temperature, top-k and top-p dials on a model's next-word probabilities, build a sentence word by word, and compare whole generations.

Know well9 min

Why should I care?

As a researcher

It is the objective behind every GPT-style model; understanding it is the starting point for scaling, post-training and interpretability.

As an engineer

It explains why output streams token by token, why longer answers cost more and take longer, and why the same prompt can give different answers.

Modern systems that depend on it

  • decoding strategies
  • in-context learning
  • chat assistants
  • the KV cache (Chapter 11)

Historical context

Before

N-gram and recurrent language models predicted the next word too, but with short contexts or sequential computation.

After

A causal Transformer predicts every next token of a long sequence in parallel during training.

Used today

Every GPT-style chat model generates this way, one token per forward pass.

What to remember

  • Output: one logit per vocabulary token → softmax → P(next token | everything so far).
  • Training: at every position, cross-entropy against the token that actually came next. One document = thousands of examples.
  • Causal masking lets training score all positions in one pass; generation still goes one token at a time.
  • Generation loop: predict → choose (decode) → append → repeat, until an end token or a length limit.

Key papers

Essential

Attention Is All You Need

Ashish Vaswani, Noam Shazeer et al. · 2017 · NeurIPS 2017

Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.

How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.

~1 h 15 min readarXiv:1706.03762✓ verified 2026-09-26
Essential

Improving Language Understanding by Generative Pre-Training

Alec Radford, Karthik Narasimhan et al. · 2018 · OpenAI technical report

GPT-1: a 12-layer decoder-only Transformer pretrained to predict the next token on over 7,000 unpublished books, then fine-tuned. It improved the state of the art on 9 of 12 tasks and set the template for every GPT since.

~30 min read✓ verified 2026-09-26

Watch