Measuring Similarity

One number for agreement

How similar are "cat" and "dog"? Your gut answers before you finish the question. A machine has no gut. But this site is quietly heading somewhere specific: soon every word will become an arrow (that is the payoff we build toward in Part II). And once words are arrows, the fuzzy question "how similar?" turns into a sharp geometric one: do two arrows point roughly the same way?

Let's set the target precisely before we compute anything. We want one number that is positive when two arrows agree in direction, zero when they are unrelated, and negative when they oppose — one number the machine can compare, rank, and sort. Here is that number's behavior, in a picture, before a single symbol.

The operation that produces that number is the dot product, and let me be honest about its importance up front: it is the single most important computation on this entire site. Attention, embeddings, similarity search — the whole machinery of a model reading meaning is this one move, repeated billions of times a second. Learn it well here and half of what is coming will feel like review.

The recipe: multiply matching parts, add

Meet the two arrows we will use for the rest of the chapter — the same pair runs through every figure, the video, and the widget, so it is worth fixing them in mind now. Our first arrow is $\mathbf{u} = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$, drawn in mint; it points shallow and to the right. Our second is $\mathbf{v} = \begin{bmatrix} 1 \\ 3 \end{bmatrix}$, drawn in amber; it points steep and to the right. Picture them: both lean the same general way, up and to the right, so we should expect them to agree.

Here is the recipe in words, before any notation: multiply the two across-parts together, multiply the two up-parts together, then add the two results. That's it. We write the operation with a centered dot, $\mathbf{u} \cdot \mathbf{v}$, read aloud as "u dot v." Let's turn the handle on our pair, showing every digit:

In words: multiply the across-parts (2 and 1) to get 2, multiply the up-parts (1 and 3) to get 3, and add them: 2 plus 3 is 5. The answer is a single positive number — the arrows agree, exactly as our eyes predicted.

In general, for any two 2D vectors, the recipe reads:

In words: multiply the first components together, multiply the second components together, and add — matching slot with matching slot, never mixing across.

Let's test the promise from the opening — positive for agree, zero for unrelated, negative for oppose — on two more cases you can check by hand.

But why should "multiply matching slots and add" measure agreement at all? Here is the honest intuition. Think of each matching pair of components as casting a vote. If both parts have the same sign — both positive, or both negative — their product is positive, a vote that the arrows push the same way along that axis. Opposite signs multiply to a negative, a vote for disagreement. And big aligned components cast big votes. The sum tallies the election. That is a real intuition, not a proof; the geometric view in a moment will make it airtight.

How long is an arrow?

We owe a debt from the last chapter: we kept saying an arrow has a length but never computed one. Time to pay up — and it turns out length and the dot product are the same idea in different clothes, so this is the perfect moment. Picture first.

Look at the arrow to the point (3, 4). Its two dashed component lines are the legs of a right triangle, and the arrow itself is the hypotenuse. That is Pythagoras, which many of us met in school: the square of the hypotenuse equals the sum of the squares of the legs. So $3^2 + 4^2 = 9 + 16 = 25$, and the length is the square root, $\sqrt{25} = 5$. We write the length of a vector with double bars, $\|\mathbf{v}\|$, read aloud as "the length of v." In general:

In words: square each component, add the squares, and take the square root — the straight-line distance from the arrow's tail to its tip.

Now the lengths of our two stars. Keeping every number honest:

In words: our mint arrow has length root-five, a little over 2.2; our amber arrow has length root-ten, a little over 3.2.

Notice these do not come out to tidy whole numbers, and that is completely fine. We carry them along as $\sqrt{5}$ and $\sqrt{10}$ — exact, honest — and only round to a decimal at the very end, when a human wants to read the answer.

The shadow view

The dot product has a second face, and seeing it is where the whole idea clicks into place. Picture first. Shine a light straight down onto the line our mint arrow lies along; the amber arrow casts a shadow onto that line. The dot product turns out to equal the length of that shadow times the length of the mint arrow — agreement, in this view, is how much of one arrow lies along the other's direction.

To write this down we need the angle between the two arrows. We call it $\theta$ — theta, the Greek letter — and it measures how far apart the two directions are.

And here is one tool from school I am going to hand you rather than rebuild: the cosine. For everything on this site it is just a machine. Feed it the angle between two directions, and it hands back one number between $-1$ and $+1$ saying how aligned they are. Feed it 0° — the same direction — and it hands back exactly 1. Feed it 90° — a right angle — and it hands back exactly 0. Feed it 180° — dead opposite — and it hands back $-1$. In between it slides smoothly: at 45°, halfway to a right angle, it reads about 0.71. That behavior — 1, fading to 0, falling to $-1$ — is the only fact about the cosine this entire site ever uses. Pocket those four checkpoints and you own the tool.

With that machine in hand, the shadow's length works out to $\|\mathbf{v}\| \cos\theta$ (the cosine of the angle scales the full length down to just the part lying along $\mathbf{u}$), which gives the dot product's second form:

In words: multiply the two lengths together, then scale the result by $\cos\theta$ — a number that says how aligned the two directions are.

Let's check this against the component recipe on our stars — this is the satisfying part. I picked these two arrows so that the angle between them comes out to exactly 45°, chosen on purpose to keep the arithmetic clean. That one step you will have to take on trust — my protractor work — because reading an arrow's angle back out of its coordinates needs a tool called the arctangent, and we deliberately never build that one here. For the record, the mint arrow $\mathbf{u} = [2, 1]$ sits at about 26.6° above the horizontal, the amber arrow $\mathbf{v} = [1, 3]$ sits at about 71.6°, and $71.6° - 26.6° = 45.0°$. Everything after that you can check, because 45° is one of the four checkpoints we just pocketed: its cosine is about 0.71. So the shadow formula gives $\|\mathbf{u}\| \, \|\mathbf{v}\| \cos\theta = \sqrt{5} \times \sqrt{10} \times 0.71 \approx 2.24 \times 3.16 \times 0.71 \approx 5.0$. The component recipe said 5. Two recipes that look nothing alike, landing on the same number — and they always will.

One quirk of these particular numbers, worth flagging so you don't read a false rule into it: the shadow happens to end exactly at the tip of $\mathbf{u}$, the point (2, 1). That's only because the shadow's length, $\sqrt{5} \approx 2.24$, equals $\|\mathbf{u}\|$, which is also $\sqrt{5}$ — a coincidence of the numbers we picked, not a general rule. Usually the foot lands somewhere else along the line entirely.

Now we can cash out the whole sign story geometrically. When the angle is less than 90°, $\cos\theta$ is positive, the shadow falls forward along $\mathbf{u}$, and the dot product is positive. At exactly 90° the shadow collapses to a single point of zero length, and the dot product is zero. Past 90° the shadow falls backward, $\cos\theta$ goes negative, and so does the dot product. That is precisely why the component recipe measured agreement all along.

Cosine similarity: agreement, pure

There is a flaw in the raw dot product if what we truly want is direction-agreement: it grows with length, not just alignment. Double the amber arrow without turning it at all and the dot product doubles too — $\mathbf{u} \cdot 2\mathbf{v} = 2 \times 2 + 1 \times 6 = 10$ — even though the directions never budged. If length changes the score while the directions stay fixed, then length is contamination.

The fix is a single division. Rearrange the shadow formula to put $\cos\theta$ by itself — divide the dot product by both lengths — and the length dependence cancels clean away:

In words: take the dot product and divide out both lengths. What remains is the cosine of the angle — pure alignment, with size stripped away.

For our pair, $\cos\theta = 5 / (\sqrt{5} \times \sqrt{10}) = 5 / \sqrt{50} \approx 0.71$. And now the scale is universal: $+1$ means identical directions, $0$ means unrelated, $-1$ means exactly opposite. Those are the same three checkpoints we pocketed a page ago, now doing real work — and the reading holds no matter how long the vectors are.

This quantity has a name we will use constantly: cosine similarity. When you hear that embeddings are compared "by cosine similarity," it means literally this fraction — the same one we just computed — evaluated on arrows with thousands of components instead of two. Nothing about the recipe changes; the lists just get longer.

See it move

Two things are far easier to feel than to read: that the number climbs and falls smoothly as one arrow swings, flipping sign exactly at the right angle, and that the shadow view and the component view land on the very same 5. The short video puts both in motion.

Now take the controls. Drag either arrow's tip and watch the panel recompute the actual arithmetic — with the current digits, not a canned answer — alongside both lengths and the cosine on its $-1$-to-$+1$ gauge. Flip on the shadow to see the projection you just met.

Why the LLM cares

When a model reads a sentence and has to decide how much the word "it" should pay attention to the word "cat" mentioned earlier, it answers by computing a dot product between two vectors — one standing for "it," one for "cat." Every attention score, in every layer, is a dot product measuring how much two words agree. We build that machine in full in Chapter 10, Attention; the arithmetic will be exactly what you just learned.

The second use is search. "Find me the passage most relevant to this question" becomes "find the stored vector with the highest cosine similarity to the question's vector" — the same fraction from a few paragraphs ago, run across a whole library. That is what powers retrieval and the systems people call RAG, which we reach in Chapter 19.

And the scale is less mysterious than it sounds. Inside a real model the recipe is identical — multiply matching slots, add — just with the number of components $d$ in the thousands instead of two. Modern hardware exists in large part because this one operation had to be made fast at enormous scale; when you hear that "GPUs multiply matrices," the next chapter shows you that even that is dot products in bulk. Back at the start of Part I we said that direction is information — the dot product is how a machine reads it.