Train a Tokenizer

Learn byte-pair encoding by hand: merge the most frequent pair, again and again, and watch an unseen word get spelled from learned pieces.

From Chapter 8: The Rise of Large Language Models