Embeddings, RAG & the LLM Application Stack

The problem

The frozen library

Chapter 11 ended with a model that is fast and cheap to serve. It still has two old problems. What it knows stopped at the end of its training data, and it can't say where any answer came from. Ask about last month's release notes, your company's wiki, or a fact mentioned in only a handful of web pages, and it either doesn't know or guesses fluently.

Facts do get into the weights. Kandpal and colleagues found that a model's accuracy on a factual question tracks how many of its pretraining documents mention the question's entities; BLOOM-176B's accuracy rose from about 25% to above 55% as that count went from 10 to 10,000 Established. Popular facts are learned; the long tail mostly isn't, and scaling barely helps there, while retrieval-augmented models beat far larger unaided ones on less popular facts Established.

This chapter builds the standard answer: keep the knowledge in documents, find the relevant pieces when a question arrives, and hand them to the model to read. We'll build it on a collection you already know, this site's own concept pages, so you can judge the results yourself:

Every number the labs show is computed from real embeddings of these pages, made offline with open models. Nothing generates text: where a language model would read the result, the labs show you exactly what it would read.

Representation

Meaning as a vector

Chapter 6 gave every word a vector. Search needs one vector per passage, so that "how do models reuse earlier work while writing?" can be compared with a paragraph about the KV cache. A text embedding model does this: a BERT-style encoder reads the passage, its token vectors are averaged, and the result is normalised to length 1. Similar meaning, similar direction; similarity is a dot product.

Reimers and Gurevych pointed out that BERT, which compares two sentences by reading them together, needs about 65 hours to find the most similar pair among 10,000 sentences; their Sentence-BERT encodes each sentence once and does it in about 5 seconds Established. That gap, compare-by-reading versus compare-by-vector, is why embeddings run search.

Where does the meaning come from? Encoders are fine-tuned on pairs that should match, such as a question and its answer passage, with contrastive learning: score the true partner against every other passage in the batch, softmax, cross-entropy. It is next-token prediction's loss with the batch as the vocabulary.

The old answer

Words still matter

Long before embeddings, search engines matched words. An inverted index maps each term to the documents that contain it: a database index for words. A query looks up its few terms and scores only those documents. BM25 scores them: rare terms count for more (inverse document frequency), the tenth repetition of a word adds less than the first, and long documents are discounted.

It needs no training, it explains itself, and it is very good at the things a single vector blurs: names, error codes, version numbers, function names. When the BEIR benchmark compared ten retrieval systems on 18 datasets they hadn't been trained for, BM25 was a robust baseline, and many methods that beat it in-domain did poorly elsewhere Established.

Try it

Find by meaning

Semantic search embeds the question and returns the passages whose vectors point most nearly the same way. Dense Passage Retrieval beat a strong BM25 system by 9–19% absolute in top-20 passage accuracy on open-domain question answering Established, and dense retrieval became the default first stage of RAG.

Below, all three run over this site's pages, which are cut into 723 chunks of about 100 words; a page ranks by its best chunk. Start with "Fading signal": keyword search ranks the vanishing-gradients page 13th, because the question shares almost no words with it, while meaning search ranks it first. Then try "ZeRO-3": keyword search finds the sharded-training page immediately; meaning search puts it 31st, since a short identifier's embedding carries little meaning.

Neither method wins everywhere, so hybrid search runs both and merges the rankings with reciprocal rank fusion: each list adds 1/(60 + rank). Agreement between the two lists wins; scores never have to be compared.

Then run the benchmark. Over the 44 labelled queries, hybrid is best on average (mean reciprocal rank 0.83; the right page in the top five for 41 of 44), keyword search is next (0.81; 39), and meaning search alone is last (0.72; 39). That is one small, hand-labelled test on one corpus, not a general ranking. The lesson is the habit, not the numbers: compare methods on many labelled queries from your own users, never on one impressive example.

Scale

Searching millions of vectors

Comparing a question with every stored vector is exact and simple. At 10 million passages of 768 numbers it's 7.7 billion multiply-adds per question, plus about 31 GB of vectors to read. Approximate nearest-neighbour indexes look at a small part instead. An inverted-file (IVF) index clusters the vectors with k-means (Chapter 3 promised this) and scans only the clusters nearest the query. HNSW builds a layered graph of neighbours and searches it greedily from a sparse top layer down, with logarithmic complexity scaling Established. Both trade a little recall for a lot of speed.

On this site's chunks, probing 1 of 32 lists scans about 4% of the vectors and finds under half of the true ten nearest; probing 8 scans about a quarter and finds about nine in ten. The same trade-off, with millions of vectors, is what vector databases tune.

Precision

A second opinion

An embedding summarises a passage before the question exists. A cross-encoder reads the question and the passage together, so every question word can attend to every passage word, and outputs one relevance score. It is far more accurate and far too slow to run on everything, so it runs second: retrieve 50–100 candidates cheaply, rerank them, keep a few. Re-ranking BM25's top 1,000 passages with BERT beat the previous state of the art on MS MARCO by 27% (relative) in MRR@10 Established.

The pattern

Retrieve, then generate

Put the pieces together and you have retrieval-augmented generation: two pipelines that meet at the prompt.

DrQA (2017) answered questions over all of Wikipedia with TF-IDF retrieval and a neural reader Established. In 2020, Lewis and colleagues named RAG: a pretrained generator conditioned on passages from a dense index of Wikipedia (21 million 100-word chunks), which set the state of the art on three open-domain QA tasks Established. Those systems trained the retriever and generator together. Most of today's don't: simply prepending retrieved documents to an unchanged model's input, with an off-the-shelf retriever, gave gains equivalent to a 2–3× larger model Established. Update a document and the next answer uses it, with no retraining, and the answer can cite where it came from.

Try it

Cutting documents

Before anything is embedded, documents are cut into chunks, and that choice quietly decides what can be found. Small chunks match sharply but can slice an answer in half; large chunks keep answers whole but blur their vectors, cost more prompt space and can exceed what the embedding model reads. On these pages, more than half of the 200-word chunks are longer than the model's 256 word pieces, so their vectors ignore their endings.

Start with the default: Chinchilla's comparison with Gopher, small chunks, three of them. Only 78% of the answer sentence arrives, because a chunk boundary cuts it and the other half isn't retrieved even at k = 5. Switch on overlap and the whole sentence arrives at k = 3; add the reranker and it comes first. Medium chunks hold it whole at k = 1. Then compare the prompt sizes.

The reranker is usually right, not always: on large chunks it pushes the LoRA answer out of first place. And there is no universal chunk size. The DPR authors found overlapping passages no better than non-overlapping ones in their setting Established, while the LoRA question here needs overlap. Pick a size that fits your embedding model, keep sentences and sections intact, put the document's title on every chunk, and measure.

The prompt

Packing the context

What the model reads is a built artefact: instructions, retrieved passages with their sources, the conversation so far, the question. Every token costs prefill compute and KV-cache memory (Chapter 11): ten retrieved passages of about 150 tokens add about 200 MB of cache per request on Llama 3 8B. And models don't use what they read evenly.

Liu and colleagues moved the answer-bearing document through 20 retrieved documents: GPT-3.5-Turbo was right 75.8% of the time with it first, 53.8% with it in the middle and 63.2% with it last, against 56.1% with no documents at all Established. In the middle, retrieval made it worse than not retrieving. Irrelevant sentences also lower accuracy Established, so retrieving twenty passages "to be safe" can hurt. Rerank, keep the few that matter, and put the best where they are used best.

Why not paste everything into a million-token window? Cost grows with length, and quality doesn't keep up: of 17 models claiming 32K-token contexts or more, only half kept satisfactory performance at 32K on the RULER benchmark Established. Long windows and retrieval are complements: retrieval chooses, a long window lets you include more of what was chosen Interpretation.

Measurement

Did it work?

A RAG system can fail in two places, so measure two layers. Retrieval: for labelled questions, is an answer-bearing passage in the top k (recall@k), and how high is the first one (MRR)? You ran exactly this in the search lab. Generation: is the answer correct, and is it faithful, every claim supported by the passages it cites? On the ALCE citation benchmark's ELI5 portion, even the best models of 2023 lacked complete citation support half the time Established.

Model judges make answer-level evaluation cheap enough to run on every change, but they are models too; check them against people on a sample. And when an answer is wrong, look at what was retrieved first. If the answer wasn't there, no prompt will fix it.

Try it

Talking to software

So far the model reads; an application also needs it to act. Three pieces turn a text generator into a component.

A system prompt sets the standing rules: the task, the constraints, the format, the tools. To the model it is just more tokens, placed first in the chat template (Chapter 10), which post-training taught it to weigh heavily. It is a strong default, not a lock.

Structured outputs make the reply parseable: JSON that matches a schema. Asking usually works and sometimes doesn't, and a pipeline that runs a million times hits "sometimes" constantly. Constrained decoding removes the "sometimes": at each step, zero the tokens that would break the format and renormalise the rest, the same mask-and-renormalise move as top-k and top-p in Chapter 8. Compiling the format into a finite-state machine and indexing the vocabulary against its states makes the allowed set cheap to look up, O(1) on average per step Established.

Tool calling puts structured output to work. The application describes its tools; the model, instead of prose, emits a call such as get_weather with JSON arguments; the application checks and runs it and returns the result as a message; the model continues. OpenAI announced function calling on 13 June 2023: the model chooses to output a JSON object containing arguments to call a described function Established. The model never runs anything itself.

In the lab, a toy model writes a weather-tool call. Sampled freely at temperature 1, about 23% of its calls break (the probabilities are made up, and real models break far less often); constrained, none do. Then ask without a date while the schema requires one: every constrained call is valid and every one contains a date the user never gave. Valid is not correct. Let the schema say null and most calls do; the fix was in the schema, not the decoder.

The whole system

The application stack

A production assistant wraps the model in ordinary software:

1. 1

   Check the input

   Is the request in scope? Route it, refuse it, or strip data that shouldn't be logged.
2. 2

   Retrieve

   Rewrite the query if needed; run keyword and vector search; rerank; filter by what this user may see.
3. 3

   Build the context

   System prompt, tool descriptions, the best few passages with sources, recent history, the question, within a token budget.
4. 4

   Generate

   Free text, structured output or a tool call.
5. 5

   Validate and act

   Check the schema and citations; check permissions before running any tool; confirm before anything irreversible.

Each step is code you can test. And each place the model reads outside text is an entry point: Greshake and colleagues showed that instructions planted in web pages likely to be retrieved could take over LLM-integrated applications, including Bing's GPT-4-powered chat Established. Retrieved passages and tool results are data, never instructions; the reliable defence is limiting what the model can do. Chapter 16 returns to attacks. The Model Context Protocol, open-sourced by Anthropic in November 2024, standardises how tools and data sources are offered to models Established.

Why it matters

Why it matters

For a researcher, retrieval separates what a model knows from what it can look up. That raises questions without settled answers: how models combine parametric and retrieved knowledge, when they should retrieve at all, how to attribute an answer to its sources, and how much memorisation retrieval can replace.

For an engineer, this chapter is most of the job. The majority of LLM features that answer from your documents are search plus a prompt plus validation, and most of their failures are search failures. The skills are familiar from data engineering: an indexing pipeline, query plans with a cheap stage and an expensive one, evaluation sets, monitoring.

What depends on it: agents use search and tools in a loop; coding assistants retrieve the right files; enterprise assistants stand or fall on permissions-aware retrieval.

Next

What comes next

Everything here was one round trip: retrieve once, call one tool, answer. Chapter 13 lets the model loop: choose a tool, read the result, decide what to do next, and repeat until a task is done. That loop is an agent, and it inherits every problem in this chapter, multiplied by the number of steps.