Skip to content
Road to Intelligence

Concept · Chapter 8: The Rise of Large Language Models

Tokenization

Must knowKnow well16 minDifficulty

A tokenizer splits text into subword pieces from a fixed vocabulary and maps each piece to an integer ID; the model only ever sees those IDs.

The problem

Whole words make an open-ended vocabulary (every rare name and typo is unknown), while single characters make sequences very long and each unit nearly meaningless.

The solution

Learn a vocabulary of frequent pieces with byte-pair encoding: start from bytes, repeatedly merge the most frequent adjacent pair, and stop at a chosen vocabulary size.

The consequence

Any text can be encoded, common words cost one token and rare ones a few, and quirks of the split explain several odd LLM behaviours, from arithmetic slips to letter-counting mistakes.

You should understand first

  1. Text as Data
  2. Vectors
  3. One-Hot Encoding
  4. Tokenization

Intuition: a vocabulary of useful chunks

Words are too many and too open-ended; characters are too small. A tokenizer picks a middle ground: frequent words get a single token, and rare words are spelled from a few common pieces. "the" is one token; "flibbertigibbet" is several.

How byte-pair encoding learns the pieces

Start with every word written as single characters (or bytes), plus an end-of-word marker. Count every adjacent pair across the corpus, weighted by word frequency. Merge the most frequent pair into a new symbol everywhere. Repeat until the vocabulary reaches the size you want.

Tiny example (the toy vocabulary from Sennrich and colleagues' paper): low ×5, lower ×2, newest ×6, widest ×3.

  1. Merge 1

    e+s occurs 9 times (6 in newest, 3 in widest), tied for the lead: merge to es.

  2. Merges 2–3

    es+t → est, then est+_ → est_ (a word-final "est").

  3. Merges 4–5

    l+o → lo, lo+w → low (7 times each).

  4. Payoff

    The word lowest was never in the corpus, yet it is now just two tokens: low + est_.

To tokenize new text, replay the learned merges in the order they were learned. Nothing is ever out of vocabulary: at worst a word falls back to single characters, or, in byte-level BPE, to raw bytes.

Try it · toy model

Train a Tokenizer

Learn byte-pair encoding by hand: merge the most frequent pair, again and again, and watch an unseen word get spelled from learned pieces.

Know well7 min

What the model actually receives

Each token maps to an integer ID, and each ID selects one row of the model's embedding table: a learned vector. GPT-2 used byte-level BPE with a vocabulary of 50,257 tokens; its smallest version has 768 numbers per embedding row Established. At the output end, the model produces one score (logit) per vocabulary entry for the next token.

Try it

See What the Model Sees

Type anything and watch the real GPT-2 and GPT-4 tokenizers split it into tokens and IDs: common words, rare words, numbers, code and other scripts.

Know well6 min

Quirks that come from tokens

  • Spelling. "strawberry" may be one or two tokens, not ten letters, so "how many r's?" asks about something the model never sees directly.
  • Numbers. Long numbers split into irregular chunks (123, 45, 67), which makes digit-by-digit arithmetic harder.
  • Languages. A vocabulary learned mostly from English splits other scripts into many more tokens. The same sentence can cost several times more, and fill the context window faster.

Why should I care?

As a researcher

Tokenization choices affect compression, multilingual fairness, arithmetic and what a model can easily 'see'; it is a live design question, not solved plumbing.

As an engineer

Prices, rate limits and context windows are counted in tokens, and the same text can cost several times more in some languages than in English.

Modern systems that depend on it

  • embedding lookup
  • context windows
  • API pricing
  • the language-model head

Historical context

Before

Word-level vocabularies with an 'unknown word' token, or character-level models with very long sequences.

After

Subword vocabularies of tens of thousands of pieces that can encode any string, including code and emoji.

Used today

Every LLM has one. GPT-2 and the original GPT-3 used a 50,257-token byte-level BPE; GPT-3.5 and GPT-4 use a vocabulary of about 100,000; newer models larger still.

What to remember

  • Text → tokens → integer IDs → rows of an embedding table. The model never sees letters.
  • BPE: merge the most frequent adjacent pair, repeat; the merges learned are the tokenizer.
  • Byte-level BPE can encode anything: an unknown character falls back to its raw bytes.
  • Common English words are usually one token, with their leading space attached.
  • Token counts differ by language and tokenizer; costs and context limits are counted in tokens.

Key papers

Essential

Language Models are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu et al. · 2019 · OpenAI technical report

GPT-2: a 1.5-billion-parameter model trained on 40 GB of web text that performed tasks with no fine-tuning at all (zero-shot), just from how the prompt was phrased.

~35 min read✓ verified 2026-09-26

Watch

3 h 31 min

Andrej Karpathy

Deep Dive into LLMs like ChatGPT

A long, general-audience walk through the whole pipeline behind a chat model, from internet text to tokens to pretraining to post-training.

Should know