Compute-Optimal Training (Kaplan vs Chinchilla)

The trade-off

With the budget C fixed, $D = C/6N$: double the model and you halve its training tokens. Too big, and the model is undertrained. Too small, and extra tokens barely help a model with too little capacity. Somewhere between is a minimum. Plot loss against N along a fixed budget (an IsoFLOP curve) and it is U-shaped.

Kaplan, then Chinchilla

Kaplan and colleagues (2020) estimated that the optimal model size grows as about C^0.73 and the data as C^0.27 Established: put most new compute into parameters. That is roughly what GPT-3 did, with 175B parameters trained on 300B tokens.

Hoffmann and colleagues (2022) trained over 400 models and found, by three different methods, that parameters and data should grow about equally, with exponents near 0.5 each Established. To test it they trained Chinchilla, 70B parameters on 1.4 trillion tokens, with the same compute as the 280B Gopher (trained on 300B tokens), and Chinchilla outperformed Gopher Established. That is 20 tokens per parameter, a ratio that became a rule of thumb.

Tiny example. A budget of 10²² FLOPs at 20 tokens per parameter: 6 × N × 20N = 10²² gives N ≈ 9 billion parameters and D ≈ 180 billion tokens.

The fit is a model too

Chinchilla also fitted a formula $L(N,D)=E+A/N^{\alpha}+B/D^{\beta}$. In 2024, Besiroglu and colleagues reconstructed the data from the paper's figure and found that the printed coefficients were inconsistent with the paper's other two methods and had implausibly narrow confidence intervals; their re-fit agreed with about 20 tokens per parameter Established. The lab below lets you switch between the two fits. At Chinchilla's budget the re-fit recommends about 72B parameters, the printed fit about 32B.

Optimal for whom?

"Compute-optimal" minimises training cost for a given loss. But a deployed model costs about 2N FLOPs for every token it generates, for as long as it is used. Sardana and colleagues extended the Chinchilla laws with inference demand and found that, with large expected demand (around a billion requests), the cheapest model to train and serve is smaller and trained on more data than Chinchilla-optimal Established. Llama 3's report says its flagship is approximately compute-optimal, while its smaller models were trained for much longer than compute-optimal and perform better than compute-optimal models at the same inference budget Established.

Labs now fit these laws on their own data. Meta's fit, extrapolated to Llama 3's budget of 3.8 × 10²⁵ FLOPs, suggested a 402B model on 16.55T tokens; noting that the IsoFLOP curves flatten near their minimum at large budgets, they chose 405B Established.