Part I · Ch. 2 — Measuring Similarity

Part I · Chapter 2 of 21

Measuring Similarity

The dot product: one number that says how much two arrows agree

You’ll need:vectors as arrows AND lists; components v₁, v₂scaling a vector changes length, not direction

One number for agreement

When words become arrows, we can ask whether their directions agree. We want one number: positive for agreement, zero at a right angle, negative for opposition.

Three small plane panels. In each, the same arrow u points up and to the right to (2,1). Panel 'agree': an arrow points the same way, longer, to (4,2); below it, u dot v = 10. Panel 'unrelated': an arrow points up-left to (−1,2) with a right-angle marker at the origin; u dot v = 0. Panel 'oppose': an arrow points down-left to (−2,−1); u dot v = −5.
Dot products track directional agreement.

The dot product does that. We will compute it two ways and see why length matters too.

The recipe: multiply matching parts, add

This chapter's figures, video, and interactive use $\mathbf{u}=[2,1]$ in mint and $\mathbf{v}=[1,3]$ in amber. Both lean up and right. Read $\mathbf{u}\cdot\mathbf{v}$ as “u dot v.”

$$\mathbf{u} \cdot \mathbf{v} = \begin{bmatrix} 2 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 3 \end{bmatrix} = 2 \times 1 + 1 \times 3 = 5$$

In words: multiply the across-parts (2 and 1) to get 2, multiply the up-parts (1 and 3) to get 3, and add them: 2 plus 3 is 5. The answer is one positive number, so the arrows agree.

$$\mathbf{u} \cdot \mathbf{v} = u_1 v_1 + u_2 v_2$$

In words: multiply the first components together, multiply the second components together, and add. Matching slot goes with matching slot, never mixing across.

Worked example

Take $\mathbf{w} = \begin{bmatrix} -1 \\ 2 \end{bmatrix}$, an arrow pointing up and to the left. Dot it with our mint arrow: $\mathbf{u} \cdot \mathbf{w} = 2 \times (-1) + 1 \times 2 = -2 + 2 = 0$. The result is zero, so $\mathbf{w}$ sits at a right angle to $\mathbf{u}$, neither agreeing nor opposing. Now flip $\mathbf{u}$ around to get $\begin{bmatrix} -2 \\ -1 \end{bmatrix}$, pointing the opposite way: $2 \times (-2) + 1 \times (-1) = -4 - 1 = -5$. That is as negative as this pair of lengths allows.

u = [2, 1] dotted with four partners
Partner of u = [2, 1]u · (partner)verdict
[4, 2], same way, longer+10agree
[1, 3], leans the same way+5agree
[−1, 2], right angle0unrelated
[−2, −1], reversed−5oppose

Those middle-to-bottom rows are the three panels of the figure above, now with their arithmetic filled in.

Matching signs multiply to positive contributions; opposite signs contribute negatively. Larger components contribute more. Adding the contributions measures agreement across all slots.

How long is an arrow?

The norm is the arrow's length. The figure shows a right triangle: the arrow is its hypotenuse.

Across 3, up 4 → length √(3² + 4²) = √25 = 5.

Pythagoras gives $3^2+4^2=9+16=25$, so the length is $\sqrt{25}=5$. Double bars, $\|\mathbf{v}\|$, mean “the length of v.”

$$\|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2}$$

In words: square each component, add the squares, and take the square root. That is the straight-line distance from the arrow's tail to its tip.

$$\|\mathbf{u}\| = \sqrt{2^2 + 1^2} = \sqrt{5} \approx 2.2 \qquad \|\mathbf{v}\| = \sqrt{1^2 + 3^2} = \sqrt{10} \approx 3.2$$

In words: our mint arrow has length root-five, a little over 2.2; our amber arrow has length root-ten, a little over 3.2.

Keep square roots exact until the last step, then round for display.

The shadow view

A projection is a shadow along a line. Drop v’s tip perpendicularly onto u’s line. The dot product equals the signed shadow length times the mint arrow's length.

A plane with the arrow u to (2,1) and a dashed thin line extending its direction. The arrow v points to (1,3). A dashed line drops perpendicularly from the tip of v onto u's line, meeting it at a dot at (2,1). A thick segment from the origin to that foot marks the shadow. A muted arc near the origin is labeled theta equals 45 degrees, and an annotation points to the shadow reading 'shadow length approximately 2.24'.
Projection measures the shared direction.

We call the angle $\theta$, “theta.” The cosine turns that angle into an alignment score.

A curve on axes labeled 'angle you feed in' along the bottom (marked 0, 45, 90, 135 and 180 degrees) and 'number it hands back' up the side (marked plus 1, 0 and minus 1). The curve starts at plus 1 above 0 degrees, slides down through 0 at 90 degrees and reaches minus 1 at 180 degrees. Four dots mark it: 0 degrees goes to 1, same direction; 45 degrees goes to 0.71; 90 degrees goes to 0, a right angle; and 180 degrees goes to minus 1, dead opposite. A faint dashed line marks the zero level.
Cosine falls as the angle opens.
Four cosine checkpoints
AngleCosineShadow
0°1Full length, forward.
45°About 0.71Shorter, forward.
90°0Zero length.
180°−1Full length, backward.

The shadow's signed length is $\|\mathbf{v}\|\cos\theta$. Therefore:

$$\mathbf{u} \cdot \mathbf{v} = \|\mathbf{u}\| \, \|\mathbf{v}\| \cos\theta$$

In words: multiply the two lengths together, then scale the result by $\cos\theta$, a number that says how aligned the two directions are.

Check the shadow

The arrows sit about 26.6° and 71.6° above horizontal, so their angle is 45°. We take those angle measurements as given; recovering them from coordinates needs a function we won't develop here. Then $\sqrt{5}\times\sqrt{10}\times0.71\approx5.0$, matching the component calculation.

In words: both lengths times the alignment score give the same dot product.

Here the shadow ends at [2,1], u's tip. That is a coincidence of these numbers rather than a rule. Orthogonal means perpendicular, not opposite.

Cosine similarity: agreement, pure

Doubling v gives $2\times2+1\times6=10$: twice the dot product with unchanged direction. Cosine similarity removes that length dependence.

A horizontal ruler from −1 to +1 with ticks at −1, −0.5, 0, +0.5, +1. Above −1, a pair of arrows point in opposite directions, labeled 'opposite'. Above 0, two arrows at a right angle, labeled 'unrelated'. Above +1, two arrows pointing the same way, labeled 'identical'. A marker dot sits at +0.71, labeled 'our pair: 0.71'.
Dividing out lengths isolates direction.

Read the alignment

$$\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \, \|\mathbf{v}\|} = \frac{5}{\sqrt{5} \times \sqrt{10}} \approx 0.71$$

In words: take the dot product and divide out both lengths. What remains is the cosine of the angle, which is alignment with size stripped away.

Our pair scores about 0.71. A positive stretch of either arrow changes numerator and denominator together, leaving this score unchanged. The zero vector has no direction, so this fraction is undefined for it.

See it move

Watch the score change sign at a right angle, then test the two recipes on your own arrows.

Watch (1:29): what to notice. The live number changes as the amber arrow swings, the sign flips at the right angle, and the shadow view lands on the same 5 the component recipe produced.

Why the LLM cares

Attention uses dot products between learned token vectors to score relevance. Vector search often compares embeddings by cosine similarity. Both repeat this chapter's arithmetic over thousands of components.

Next we will organize many dot products into one operation: a matrix multiplying a vector.

What you now know

  • The dot product multiplies matching entries and adds.
  • Its sign distinguishes agreement, perpendicularity, and opposition.
  • Dotting a vector with itself gives its squared length.
  • The shadow view gives the same dot product.
  • Cosine similarity removes length dependence for nonzero vectors.

Where we're headed

We can compare arrows. Now let's build matrices that transform them, using one dot product per output.