RAG and Vector Search · How LLMs Actually Work

A question the model can't answer

“What's our refund policy?” Training may never have included our private documents. A fluent model can still invent a plausible policy. We need to supply evidence.

Where should changing facts live?
Approach	Cost or limit
Retrain or fine-tune	Changes weights; slow to refresh and unreliable for exact recall
Paste every document	Uses context space and attention computation
Retrieve relevant passages	Keeps documents outside the model; depends on search quality

Vector search finds candidates. RAG supplies those passages to the model. We can build both with Chapter 8's embeddings and Chapter 2's cosine similarity.

Whole passages become arrows

An embedding model represents a whole passage with one vector. Query and document encodings must share a compatible space. Training makes relevant pairs easier to retrieve; similarity is evidence of relevance, not proof.

“How do I get my money back?” and “Items may be returned within 30 days for a full refund” can match despite different wording. Our arrows are invented for teaching, as in Chapter 8; real vectors usually have many more dimensions.

Related wording can point the same way.

$$\mathbf{q} = \begin{bmatrix} 3 \\ 4 \end{bmatrix} \qquad \mathbf{v}_{\text{refund}} = \begin{bmatrix} 4 \\ 3 \end{bmatrix}$$

In words: the question points to the spot (3, 4) on the plane and the refund passage points to (4, 3), almost the same direction, with the across- and up-parts swapped.

Our question is $\mathbf q=[3,4]$. The refund passage is $[4,3]$. Both have length 5. We deliberately reused the cat–dog arithmetic so we can check the result by hand.

A library of arrows: nearest-neighbor search

We split documents into chunks, embed each, and store its text, source, and vector in a vector database. This preparation is called indexing.

At question time, nearest-neighbor search ranks candidates. Our measure is cosine:

$$\cos\theta = \frac{\mathbf{q} \cdot \mathbf{v}_{\text{refund}}}{\|\mathbf{q}\| \, \|\mathbf{v}_{\text{refund}}\|} = \frac{3 \times 4 + 4 \times 3}{5 \times 5} = \frac{24}{25} = 0.96$$

In words: multiply matching slots and add to get 24, then divide by both lengths. Their cosine is 0.96, a directional similarity score.

Worked example

Five passages, five dot products over the same $5 \times 5 = 25$, then sort:

Five passages scored against the question
Passage (its arrow)	q · passage	cos θ	reading
refund policy [4, 3]	3×4 + 4×3 = 24	0.96	the winner
warranty claims [4.8, 1.4]	3×4.8 + 4×1.4 = 20	0.80	related
holiday hours [−3, 4]	3×(−3) + 4×4 = 7	0.28	a little
shipping times [4, −3]	3×4 + 4×(−3) = 0	0.00	perpendicular
careers page [−3, −4]	3×(−3) + 4×(−4) = −25	−1.00	the anti-match

Look at the shipping row. $12 - 12 = 0$ means the shipping passage sits at a right angle to the money-back question: unrelated in the Chapter 2 sense, neither agreeing nor opposing.

Cosine ranks the toy library.

These are hand-placed geometry scores, not probabilities of relevance. A negative cosine is an opposite direction; in a learned embedding space it need not mean an opposite meaning.

Approximate nearest-neighbor search inspects promising regions instead of comparing every vector. The tradeoff is speed versus missed candidates. Library size, vector width, hardware, and filtering determine when approximation helps.

RAG: the full loop

Prepare the document index first; model weights stay fixed.
Step	Action
1. Embed the question	Use the library’s embedding space.
2. Retrieve passages	Rank matches and enforce access rules.
3. Assemble the prompt	Combine sources, question, and instructions.
4. Generate and check	Answer from context; verify claims against cited text.

Index once; retrieve for each question.

The assembled prompt (step 3, made visible)

Context from company documents:
— Items may be returned within 30 days for a full refund.
— Warranty claims: defects covered for one year.
Question: How do I get my money back?
Answer using only the context above.

The passages enter as ordinary context tokens. Attention reads them; training weights stay fixed. Source identifiers let the answer point back to evidence. Access restrictions must apply before unauthorized passages reach the prompt.

Where retrieval can fail
Failure	What we check
Relevant text never retrieved	Chunk boundaries, query encoding, and search coverage
Exact name or part number missed	Combine keyword and vector search
Outdated or unauthorized source	Refresh the index and filter permissions
Answer distorts the retrieved text	Compare claims with citations; allow an insufficient-evidence answer

Grounding helps us check answers. It cannot guarantee them. Hybrid search covers complementary strengths. Re-embed changed chunks when documents change; changing the embedding model usually requires rebuilding compatible document vectors.

See it move

Watch (1:44): what to notice. The question becomes a mint arrow, a cloud of blue chunk-arrows appears around it, and the nearest three light up by the same cosine you measured in Chapter 2. Those chunks then stack with the question into one boxed prompt that gets answered with a citation, and the last run shows a miss, where the nearest chunks are the wrong ones.

The eleven-passage widget lets us move the question vector and inspect the resulting prompt. It exposes the retrieval arithmetic; production systems add indexing, permissions, and evaluation.

Try it

Six things to do, in order.

The widget opens on the chapter's numbers. Confirm the top hit reads $\cos = 24 \div (5.0 \times 5.0) = 0.96$ for the refund policy.
The long "returns FAQ" arrow ties it at 0.96. It is twice as long and points the same way, and cosine cannot see length, which is Chapter 8's lesson doing real work.
Press "where's my package?" and watch the ranking reshuffle: track-your-package jumps to 0.98.
Drag the question arrow slowly from the returns cluster toward the store-info cluster and watch the rankings cross over on the way.
Switch to the "prompt" tab to see what would be pasted into the model's context. That panel is the whole of RAG.
Drag the question to the origin. A zero-length question matches nothing, and the widget says so.

Why the LLM cares

Weights contain learned skills and factual associations. Context supplies the current working material. Retrieval selects material from an external library. Fine-tuning and RAG can complement each other: adapt response behavior, then retrieve current evidence. The same vectors also support search without generation, duplicate detection, clustering, and recommendations.

Three homes for information

Retrieval moves library evidence into the context the model reads.
Home	Contains	Updated by
Weights	Learned facts and skills	Training; relatively slow
Context window	Current working material	Prompt assembly; immediate
Library	Documents, vectors, sources	Editing and re-embedding; typically minutes

Worked example: change the policy

Our stored policy says 30 days. If the approved document changes to 45 days, update its text and embedding, then retrieve that new passage. The assembled prompt now says 45 days while model weights remain unchanged. Check that the answer cites the current source.

What you now know

Embedding models represent passages and questions in compatible vector spaces.
Our refund passage wins with cosine 0.96.
RAG retrieves passages, adds them to context, then generates.
Index updates change available evidence without changing model weights.
Retrieval and generation can both fail; sources make checking possible.

Where we're headed

We have built the last mechanism. Next we place it on the wider map of machine learning.