RAG and Vector Search

A question the model can't answer

Ask the smartest model on Earth "What's our refund policy?" and it cannot know. Your store's policy was never in its training text — the mountain of data from Chapter 15 stopped before your policy existed, and your private wiki was never in it at all. Worse, Chapter 9 taught us exactly what the machine does when it can't know: it produces something plausible anyway. Confident, fluent, and made up.

Run through the tools we already own and watch each one miss. We could retrain from scratch on text that includes your wiki — years of GPUs for a returns policy, absurd. We could fine-tune or add a LoRA adapter (Chapters 16–17) — but we drew an honest boundary there: adapters teach style and skills, and weights make an expensive, unreliable filing cabinet for facts that change every Tuesday. Or we could paste your whole document pile into the prompt — but Chapter 18 priced that: the context window is a desk, not a library, with quadratic teeth.

The actual fix is one homely sentence: don't memorize the library — get a library card. Fetch the few passages that matter for this question, put only those on the desk, and let the model read them. The fetching machine is vector search, and the whole pattern is RAG — retrieval-augmented generation. Both are the meaning-map of Chapter 8, industrialized; you already own every part.

Whole passages become arrows

This chapter needs exactly one upgrade. Chapter 8 gave an arrow to every token. An embedding model — a smaller sibling of the transformer you built in Chapter 11 — reads a whole sentence or paragraph and outputs one vector for the entire thing. One passage, one arrow. It is trained (the same downhill story as everything else on this site) so that texts people use interchangeably land in the same direction.

Here is the property that makes it magic, told with this chapter's two star texts: "How do I get my money back?" and "Items may be returned within 30 days for a full refund." They share zero important words — no "money," no "back" in the second; no "return," no "refund" in the first. A keyword search scores them as strangers. A good embedding model gives them nearly the same direction, because the two texts mean the same kind of thing.

Now the chapter's toy library, built with Chapter 8's honesty ritual: every vector below is hand-placed by me, invented for teaching — real embedding vectors have hundreds of dimensions, same recipes, longer lists. The question is $\mathbf{q} = [3, 4]$ for "How do I get my money back?" — written $\mathbf{q}$ on purpose (search people and attention people chose the same letter for the same reason: it is the thing that asks). The library is five passages from a small store's help pages, all length-5 arrows so every cosine will divide by 25: refund policy $\mathbf{v}_{\text{refund}} = [4, 3]$, warranty claims $[4.8, 1.4]$, holiday hours $[-3, 4]$, shipping times $[4, -3]$, careers page $[-3, -4]$. Here are the first two as columns:

In words: the question points to the spot (3, 4) on the plane; the refund passage points to (4, 3) — almost the same direction, with the across- and up-parts merely swapped.

And here is a wink you have earned. The dot product $\mathbf{q} \cdot \mathbf{v}_{\text{refund}} = 3 \times 4 + 4 \times 3 = 24$, over lengths $5 \times 5$, is the exact arithmetic of $\text{cat} \cdot \text{dog}$ from Chapter 8 — I chose these numbers on purpose, so you can check this whole chapter by eye. Directions carry meaning whether the arrow stands for a word or a paragraph.

A library of arrows: nearest-neighbor search

The work splits into two halves. The offline half is done once, before any question arrives: split your documents into chunks of a few hundred tokens each (Chapter 7's unit doing logistics work), run every chunk through the embedding model once, and file each (vector, original text) pair in a vector database. Your library is now a cloud of arrows, each dragging its passage behind it.

The online half happens at question time. Embed the question with the same model — $\mathbf{q} = [3, 4]$ — then find the stored arrows with the highest cosine similarity to it. That search has a name: nearest-neighbor search. Let's work the winner fully, using Chapter 2's cosine recipe, one display:

In words: multiply matching slots and add to get 24, divide by both lengths — the question and the refund passage agree 96 percent of the way.

Do the same for every shelf and the library summarizes into one ranking table. The careers page earns a special note: its arrow points exactly opposite the question, so its cosine bottoms out at $-1.00$ — the perfect anti-match. Permission to enjoy that.

Read that table as the chapter's thesis: the machine never understood the word "refund." It computed five dot products and sorted them. Meaning did the matching — because meaning was already baked into where the arrows point (that was Chapter 8's whole story). Relevance has become geometry.

One honest word on scale. A real library holds a million chunks, and — surprisingly — a million dot products per question is fine. Chapter 2 told us this operation is exactly what modern hardware was built to do; a million of them is milliseconds. At billions of chunks, engineers switch to approximate nearest-neighbor search: pre-sort the arrows into neighborhoods and search only the promising ones, accepting a tiny chance of missing the true winner in exchange for enormous speed. The idea is no more than "shelve the library by topic first"; the rest is names.

RAG: the full loop

Now assemble the pattern end to end — four steps, each one already yours. (1) Embed the question (this chapter). (2) Retrieve the top few chunks by cosine (this chapter). (3) Paste the winners into the prompt, above the question (Chapter 18's desk, used deliberately). (4) Generate — the frozen model reads its context and answers from it (Chapter 9's game, unchanged).

Step 3 is the one that makes the whole chapter click, so let's make it concrete. Using our toy library's top hits, the pipeline assembles a prompt that looks exactly like this:

Context from company documents:
— Items may be returned within 30 days for a full refund.
— Warranty claims: defects covered for one year.
Question: How do I get my money back?
Answer using only the context above.

And here is the beat worth sitting with: nothing about the model changed. No training, no weight updates, no new machinery. The retrieved text walks in through the front door as ordinary tokens in the context, and attention (Chapter 10) does the rest.

Why does this beat the alternatives for facts? Three reasons. Your knowledge updates by re-embedding the changed page — minutes and pennies, instead of the retraining Chapter 17 priced. The model can quote its sources, so answers become checkable — the honest antidote to Chapter 9's plausible-but-false failure mode. And access control stays outside the model: retrieve only what this user is allowed to see.

Now the honest failure modes, no doom. Retrieval is now the weakest link: if the right chunk isn't fetched — bad chunking, or a question phrased off the map — the model answers from vibes again, wearing the same confident tone. And retrieval is not obedience: a model can still ignore or garble the passages on its desk. Grounding reduces hallucination; it does not abolish it. The engineering craft — chunk sizes, overlap, how many chunks to fetch, blending keyword and vector search — is exactly the craft of making the right passage land on the desk.

See it move

The toy library, live: eleven passages drawn as arrows on the map, a draggable question arrow, a cosine ranking that updates as you move, and a panel showing the exact prompt RAG would assemble from the winners. This widget is a vector database with eleven rows — production systems swap in a million chunks and hundreds of dimensions, and change nothing else.

Why the LLM cares

"Chat with your PDF." Enterprise copilots that know the internal wiki. Support bots quoting the returns policy. Coding assistants that pull your codebase's own files. Search engines' AI answers with little citation numbers. Every one is this chapter's loop wearing a different shirt. When an answer arrives with sources attached, you are looking at RAG.

Keep this division of labor — it is the takeaway rule of thumb, and it echoes Chapters 16–18 exactly. Weights hold skills: language, reasoning patterns, tone — that is pretraining and fine-tuning's job. The context holds the working facts of the moment — that is the window's job. Retrieval is the bridge that keeps refilling the context from outside. Want the model to sound like your lab group? A LoRA adapter (Chapter 17). Want it to know this week's protocols? RAG. The two compose — tune the voice, retrieve the facts.

Embeddings have a second life, too. The same embed-and-compare move powers semantic search with no generation at all, duplicate detection, clustering feedback into themes, and recommendations ("this song's vector is near that one's"). Chapter 8 promised that embeddings escaped the LLM — this chapter is where you see the escape route.

And so we cash the site's oldest check one last time. Chapter 2 called the dot product the most important operation on this site. Today it ran your library. Attention asks it inside the model; retrieval asks it outside; and both are the same question — how much do these two directions agree?