How an LLM Is Actually Built

The problem

The shopping list

Chapter 8 summed up how large language models are made in one sentence: predict the next token, at scale. This chapter is about the words "at scale". What would it actually take to train one?

Most frontier labs no longer say. One exception is Meta's report on Llama 3, which describes training a 405-billion-parameter model in unusual detail, failures included. It makes a good running example. Here is its bill of materials:

The token count, vocabulary, GPU count, interruptions and compute are all from the Llama 3 report Established. The 6.5 TB is our arithmetic: 405 billion parameters at 16 bytes each, a figure explained below.

Read down the list and it is less a machine-learning problem than a data-engineering and distributed-systems one. If you build data pipelines for a living, you already know half of this chapter: ETL, deduplication, sharding, checkpoints and capacity planning. The other half is what those words mean when the job is a neural network.

The raw material

Trillions of tokens

The text mostly comes from Common Crawl, a public archive of the web released as snapshots. Raw, it is mostly menus, cookie banners, product listings, spam and copies. A pretraining pipeline extracts the main text from each page, identifies its language, applies cheap rules (too short, too repetitive, too many symbols), scores quality with a classifier, removes unsafe and personal content, and deduplicates.

Most of what goes in does not come out. In the open FineWeb dataset, basic filtering of 96 Common Crawl snapshots left about 36 trillion tokens; deduplication left about 20 trillion; further filters left 15 trillion. An educational-quality classifier, trained on Llama 3's ratings of 460,000 pages, then selected 1.3 trillion of those for FineWeb-Edu Established.

Copies deserve their own step. One 61-word sentence appeared over 60,000 times in the C4 dataset, and models trained after deduplication emitted memorised text ten times less often Established. Exact hashing only catches identical pages, so pipelines compare documents as sets of overlapping word sequences (shingles) and estimate their overlap with compact MinHash signatures. Try the method on eleven made-up web pages. At the default settings, which are FineWeb's, the mirror and the syndicated copy of a news story are caught, as are two shop pages built from one template. A re-posted recipe with a reworded first sentence slips through. Lower the rows per band to catch it.

Even this has surprises. FineWeb's team found that deduplicating all 96 snapshots against each other made the data worse than deduplicating each snapshot on its own: in the oldest crawls, what survived global deduplication was lower quality than what it removed Established.

The recipe

What goes in the mix

A training set is not one source but a weighted blend. Sampling in proportion to size would bury the small, dense sources (books, encyclopedias, code, papers) under web text. LLaMA (2023) drew 67% of its tokens from Common Crawl and 4.5% from Wikipedia, but went over Wikipedia about 2.45 times and over Common Crawl about 1.1 times Established. Llama 3 chose its mix by training small models on candidate blends and extrapolating with scaling laws, ending at roughly 50% general knowledge, 25% maths and reasoning, 17% code and 8% multilingual text Established.

How often can the good data be repeated? Up to about 4 epochs, repeated data was almost as valuable as fresh data for a fixed compute budget; after that, returns faded Established. When even that is not enough, models can write data for other models. How far synthetic data can replace human writing is an open question Active research.

Scraping the web has a catch for evaluation: benchmarks live on the web too. GPT-3's authors tried to remove benchmark test data from their training set, but a bug left some in and retraining was too expensive Established. This is Chapter 3's data leakage at the scale of the internet.

The interface

Building the tokenizer

Chapter 8 trained a toy BPE tokenizer. A real one is trained the same way on a large sample of the training mix, and then frozen: the model's embedding table is built around it, so it can't change later without retraining.

The main dial is vocabulary size. A larger vocabulary packs more text into each token, so the same compute and context window cover more text, at the cost of a bigger embedding table. Llama 3 grew its vocabulary to 128,000 tokens, which raised English compression from 3.17 to 3.94 characters per token compared with Llama 2 Established: about 19% fewer tokens for the same text, for the model's whole life. Other choices are rules (LLaMA split every number into single digits and fell back to raw bytes for unknown characters Established) and special tokens that mark structure, such as the end of a document or, later, the turns of a chat.

The engine

The training loop

The loop is Chapter 4's: forward pass, loss, backward pass, update. What changes is size. Batches are counted in tokens: Llama 3 405B started at 4 million tokens per step and doubled twice, to 16 million Established. No GPU holds that many at once, so each processes a few sequences at a time and accumulates gradients until the batch is complete, and only then does the optimizer step. With trillions of tokens, most data is seen about once, so classic overfitting rarely shows up and the training loss on fresh batches is itself an honest measure of progress.

The learning rate is not fixed. It warms up from near zero, because early steps on random weights with immature optimizer statistics can wreck a run, then decays along a cosine curve so the model can settle. Gradient clipping caps any single oversized step.

A linear warmup to 8 × 10⁻⁵ over the first 8,000 steps (dashed line), then a cosine decay to 8 × 10⁻⁷ by step 1.2 million. The warmup is under 1% of the run, too short to see on the left.

The arithmetic

Numbers in sixteen bits

GPUs multiply 16-bit numbers much faster than 32-bit ones, and 16-bit activations take half the memory. But 16 bits lose things. In FP16, gradients smaller than about 6 × 10⁻⁸ become zero, and with only a few significant digits, 1.0 + 0.0001 rounds back to 1.0, so small updates vanish.

Mixed precision keeps the best of both. The forward and backward passes run in 16 bits, and a 32-bit master copy of the weights receives the updates. With FP16, the loss is scaled up so small gradients survive. BF16, a 16-bit format with FP32's range but fewer digits, trains without loss scaling Established, which is why it is the usual choice today.

The first wall

Where the memory goes

The ZeRO paper starts from a puzzle: a 1.5-billion-parameter model needs only 3 GB for its 16-bit weights, yet could not be trained on a 32 GB GPU Established. The answer is the training state. With mixed-precision Adam, every parameter carries a 16-bit weight and gradient (2 + 2 bytes) plus a 32-bit master weight and Adam's two running averages (4 + 4 + 4): 16 bytes per parameter. A 7-billion-parameter model's weights fit on one 80 GB GPU; its training state, about 107 GB, does not.

Then come the activations, the intermediate values saved for the backward pass. They grow with sequence length, batch size and depth. For LLaMA 7B with one 4,096-token sequence, storing all of them takes about as much memory again as the model states. Recomputation keeps only some and recomputes the rest during the backward pass, trading extra compute for memory.

Start the calculator below at its defaults: 211 GB per GPU, hopeless. Shard the model states across 8 GPUs with ZeRO-3 and it is still about 118 GB, because sharding doesn't touch activations. Now recompute the attention scores: about 32 GB, and it fits.

The second wall

Splitting the work

Even when memory allows it, one GPU would take centuries. The work has to be spread over thousands, in up to four ways at once.

• Data parallelism: every GPU holds a copy of the model and processes a different slice of the batch, and an all-reduce averages their gradients before each update. The maths is unchanged; it is map-then-reduce.
• Sharding (ZeRO, FSDP): instead of identical copies, each GPU keeps only a slice of the weights, gradients and optimizer states, gathering what it needs just in time. In ZeRO's example, a 7.5B model's states fall from 120 GB per GPU to 1.9 GB on 64 GPUs Established.
• Tensor parallelism: split each layer's matrices across the GPUs of one server, which then compute every layer together, talking constantly over very fast links.
• Pipeline parallelism: give each server a block of consecutive layers and stream micro-batches through them like an assembly line, at the cost of an idle "bubble" while the pipeline fills and drains.

Data parallelSharded (ZeRO-3 / FSDP)Tensor parallelPipeline parallel

Each GPU holds

A full copy of the model, and a different slice of the batch.

What they exchange

Once per step, every GPU averages its gradients with all the others (an all-reduce), so all copies stay identical.

Used

Always, as the outermost layer of parallelism: it is how a run uses thousands of GPUs.

The catch

The whole model, its gradients and optimizer states must fit on every GPU.

Real runs combine them. Llama 3 405B used tensor parallelism across the 8 NVLink-connected GPUs of each server, 16 pipeline stages, and sharded data parallelism (FSDP) across the rest, plus a fourth kind, context parallelism, for very long sequences: up to 16,384 GPUs.

Llama 3 405B combined all of them: tensor parallelism across the 8 GPUs in each server, 16 pipeline stages, 128-way sharded data parallelism, and context parallelism for very long sequences, at 38–43% of the GPUs' peak arithmetic rate Established.

The long run

When things break

A synchronous job on thousands of GPUs, running for months, will break. During one 54-day period of Llama 3 405B's training there were 466 interruptions, 419 of them unexpected, and about 78% of those were attributed to hardware, mostly GPUs Established: one about every three hours. Automation still kept more than 90% of the time spent on useful training Established. The defence is frequent checkpoints, automatic restarts and tools that find the faulty machine quickly.

The optimisation can break too. PaLM's largest run saw about 20 loss spikes; the team restarted from about 100 steps before each spike and skipped a few hundred batches Established. Why such spikes happen in very large models is not fully understood Active research.

The bill

Counting the compute

There is a one-line estimate of the whole bill. Each parameter is used about once per token on the forward pass (a multiply and an add, 2 FLOPs) and about twice as much on the backward pass (4 FLOPs). So training N parameters on D tokens takes

$$C \approx 6\,N\,D$$

floating-point operations. Check it: 6 × 405 × 10⁹ × 15.6 × 10¹² ≈ 3.8 × 10²⁵, exactly the figure Meta reports. For GPT-3, 6 × 175 × 10⁹ × 300 × 10⁹ ≈ 3.15 × 10²³, against a reported 3.14 × 10²³.

To turn FLOPs into time, divide by what the hardware actually sustains, not its peak. Llama 3 reached about 400 trillion FLOP/s per H100 Established. On 16,384 GPUs that gives 3.8 × 10²⁵ ÷ (16,384 × 4 × 10¹⁴) ≈ 5.8 million seconds: about 67 days of uninterrupted training. That is our estimate, before any time lost to failures.

The decision

Spending the budget

With C fixed, D = C / 6N. Double the model and you halve its tokens. Which split gives the best model? Kaplan and colleagues (2020) concluded that most extra compute should go into parameters Established, and models grew much faster than their datasets: GPT-3 saw under 2 tokens per parameter. Hoffmann and colleagues (2022) found instead that parameters and tokens should grow about equally, and showed it with Chinchilla: 70B parameters on 1.4 trillion tokens, the same compute as the 280B Gopher, and better Established. About 20 tokens per parameter became the rule of thumb.

The explorer below uses the scaling-law fit behind that result. It starts at Gopher's size for Gopher's budget. Slide the model size to the left and the predicted loss falls, then rises again: the bottom sits near 72B parameters, almost exactly Chinchilla. Then switch to the fit as printed in the 2022 paper. Its advice is quite different, and a 2024 replication traced the difference to the original fitting procedure and rounded coefficients Established. Influential numbers deserve checking.

"Optimal" here means cheapest to train. A deployed model also costs about 2N FLOPs for every token it generates, possibly trillions of them. Move the last slider and the cheapest model for the same quality gets smaller and is trained for longer. Llama 3's flagship is roughly compute-optimal, but its smaller models were trained far longer than compute-optimal, and performed better than compute-optimal models at the same inference cost Established.

Why it matters

Why it matters

Chapter 8 showed that next-token prediction at scale works. This chapter shows that "at scale" is mostly engineering: data pipelines, numerical formats, memory accounting, distributed systems, reliability and budgeting, each with its own trade-offs.

• For an engineer, the numbers here are napkin maths you can reuse: 16 bytes per parameter to train, 2 to run, 6ND FLOPs to train, about 2N per generated token, 40% of peak throughput. They tell you in a minute whether a fine-tuning job fits on your GPUs and roughly what it costs.
• For a researcher, the open questions are just as concrete: which data actually helps, how far synthetic data can go, why large runs spike, how far scaling laws extrapolate, and how to evaluate models trained on the whole web.

What comes next

What comes out

At the end of all this, months of compute and millions of dollars, you have a base model. It has read trillions of tokens and predicts the next one extremely well. But ask it a question and it may continue with three more questions, as a quiz page would. Chapter 10 covers how post-training (supervised fine-tuning and preference optimisation) turns this raw text predictor into an assistant.