Concept · Chapter 8: The Rise of Large Language Models
Decoding: Greedy, Temperature, Top-k, Top-p
A language model outputs a probability for every next token; a decoding strategy decides which one to use, trading predictability against variety.
The problem
Always taking the most likely token gives bland, repetitive text, while sampling from the full distribution occasionally picks tokens from the unreliable tail and derails.
The solution
Reshape the distribution with a temperature, cut the tail with top-k or top-p, then sample; or take the top token when determinism matters.
The consequence
The same model can be a careful answerer or a varied writer, and 'why did it say something different this time?' has a concrete answer.
You should understand first
- Text as Data
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Vectors
- Dot Product
- Embeddings
- Attention
- Softmax
- Self-Attention
- Causal Masking
- Entropy
- Loss Functions
- Cross-Entropy Loss
- Autoregressive Next-Token Prediction
- Decoding: Greedy, Temperature, Top-k, Top-p
Intuition
At each step the model hands over a ranked list: "mat" 40%, "floor" 20%, "sofa" 10% … and thousands of tokens with tiny probabilities. Decoding is the policy for choosing. Taking the top item every time is safe but dull. Rolling a die weighted by the probabilities is lively, but every so often the die lands on something absurd from the long tail.
Tiny numeric example
Three candidate tokens with logits 2, 1 and 0.
Temperature 1
Softmax gives 66.5%, 24.5%, 9.0%.Temperature 0.5
Divide logits by 0.5 → 4, 2, 0 → 86.7%, 11.7%, 1.6%. Sharper: the favourite dominates.Temperature 2
Divide by 2 → 1, 0.5, 0 → 50.6%, 30.7%, 18.6%. Flatter: more variety.Top-p = 0.9 (at T = 1)
Keep tokens in order until their total reaches 90%: 66.5% + 24.5% = 91% ≥ 90%, so the third token is dropped. Renormalise: 73.1% and 26.9%.
Try it
Try it · toy model
Turn the temperature, top-k and top-p dials on a model's next-word probabilities, build a sentence word by word, and compare whole generations.
Why truncation matters
Holtzman and colleagues (2019) showed that maximising likelihood during generation (greedy or beam search) produces "bland and strangely repetitive" text even from a strong model, while unrestricted sampling draws from an unreliable tail; they proposed nucleus (top-p) sampling Established. Top-k sampling had been used in story generation shortly before (Fan et al. 2018) Established. Top-p adapts: when the model is confident the nucleus may be one or two tokens; when many continuations are plausible it widens.
Why should I care?
As a researcher
Decoding interacts with evaluation: the same model scores differently under different strategies, and many generation failures are decoding failures.
As an engineer
Temperature and top-p are the knobs in every LLM API; setting them sensibly (low for extraction, higher for brainstorming) is basic practice.
Modern systems that depend on it
- chat assistants
- code generation
- self-consistency and other sampling-based methods (Chapter 14)
Historical context
Before
Greedy and beam search, borrowed from machine translation, where one best output was the goal.
After
Truncated sampling (top-k, top-p) with a temperature became the default for open-ended generation.
Used today
Exposed as temperature, top_p and top_k parameters in LLM APIs and local runtimes.
What to remember
- Greedy: always the top token. Deterministic, but prone to bland, repetitive text.
- Temperature T divides the logits: T < 1 sharpens, T > 1 flattens, T → 0 becomes greedy.
- Top-k: sample only among the k likeliest tokens.
- Top-p (nucleus): sample among the smallest set whose probabilities add up to p; its size adapts to the model's confidence.
- Temperature reshapes; top-k and top-p truncate. They are usually combined.
Key papers
Hierarchical Neural Story Generation
Angela Fan, Mike Lewis, Yann Dauphin · 2018 · ACL 2018
Popularised top-k sampling: pick only among the k most likely next words, which avoids both bland beam-search text and nonsense from the tail.
The Curious Case of Neural Text Degeneration
Ari Holtzman, Jan Buys et al. · 2019 · ICLR 2020
Explained why maximising likelihood at generation time gives 'bland and strangely repetitive' text, and introduced nucleus (top-p) sampling, now a default setting in LLM APIs.