Context Engineering

The window is a budget

Everything the model sees for a request competes for one window: system instructions, tool descriptions, the conversation so far, retrieved passages, the question. Each token is processed in prefill and stored in the KV cache for the rest of the reply. Retrieving 10 passages of about 150 tokens adds 1,500 tokens; for Llama 3 8B, whose cache takes 128 KiB per token in 16 bits, that's about 200 MB of cache for that one request (our arithmetic from Chapter 11's formula).

Position matters

Liu and colleagues moved the document containing the answer through a list of 20 retrieved documents: GPT-3.5-Turbo answered 75.8% correctly with it first, 53.8% with it in the middle (position 10) and 63.2% with it last, against 56.1% with no documents at all Established. In the middle, the model did worse than closed-book. The paper reports that performance is often highest when relevant information is at the beginning or end of the input, and that extended-context versions of models were not necessarily better at using their context Established.

What to do with that: rank passages well, keep the list short, and put the best ones where they are used best. Reversing the order so the best passage sits last, next to the question, is a common trick; test it on your model.

Less can be more

Shi and colleagues added a single irrelevant sentence to grade-school maths problems and found model accuracy dropped dramatically; telling the model to ignore irrelevant information helped Established. Retrieving 20 passages "to be safe" can hurt. Rerank, then keep the few that matter.

Long context or retrieval?

Context windows have grown into the hundreds of thousands of tokens, so why not paste everything? Cost and latency grow with length, and quality doesn't keep up. Xu and colleagues found a 4K-context model with simple retrieval comparable to a 16K-context model on long-context tasks with much less computation, and that retrieval helped regardless of window size Established. The RULER benchmark found that only half of 17 models claiming 32K tokens or more kept satisfactory performance at 32K Established. Long windows and retrieval are complements: retrieval picks what matters, a long window lets you include more of it when needed Interpretation.

History and compression

Conversations grow. Options: keep the last few turns verbatim, summarise older ones into a short running note, or treat past turns as another collection to retrieve from. Agents (Chapter 13) need the same for their tool results. Prompt compression methods such as LLMLingua drop tokens that a small model finds predictable and report up to 20× compression with little performance loss on the tasks they tested Established.

A template that works

1. Stable instructions first (and they can be cached between requests).
2. Retrieved passages, each with an ID, title and source, best placed deliberately.
3. Recent conversation, summarised beyond a budget.
4. The question last, with the reminder to answer only from the sources and cite them.