What is a list of numbers for?
A list of numbers is a place in a space, and once it is a place you can measure it: how far, which way, how much it agrees with another list. Every capability in this book is one of those measurements.
Open any model and you find lists of numbers. A word is a list. A sentence is a list. A face, a radar sweep, a song, your last email: by the time a machine touches them they are all lists of numbers. Ask what the individual numbers mean and you get nothing useful back. Ask what you can measure about the list and you get everything. This chapter builds the first of those measurements by hand, with numbers small enough to check on paper.
One list, three readings
The list $[3, 2]$ can be three different things, and all three are true at once.
Read as a place, $[3, 2]$ is a dot: go 3 across and 2 up from the origin and put your finger there. Read as a trip, it is an arrow, a vector, which is a direction plus a distance. That is the same instruction whether you start at the origin or at a street corner three towns over. Read as a heading, it is only the way the arrow points, with its length thrown away.
A vector gets a bold lowercase letter. The running pair for this whole book is $\mathbf{v} = [3, 2]$ and $\mathbf{u} = [1, 3]$, read "the vector v" and "the vector u." In every figure on this site the first is drawn in mint and the second in amber.
$$\mathbf{v} = \begin{bmatrix} 3 \\ 2 \end{bmatrix} \qquad \mathbf{u} = \begin{bmatrix} 1 \\ 3 \end{bmatrix}$$In words: v is the pair 3 and 2, three across and two up. u is the pair 1 and 3, one across and three up. Both lean up and to the right, but v leans flatter.
Which reading you want depends on the question. "How big is this?" is a question about the trip. "Are these two things alike?" is a question about the heading, and it is the one a search engine, a memory system and an attention head each ask thousands of times a second. Keep the three readings separate in your head and the rest of the book follows.
Agreement, measured
We want one number that answers "do these two arrows point the same way?" It should be a sliding scale. Strong agreement scores high, a right angle scores zero, and pointing against each other scores negative.
The instrument that does this is the dot product. Two steps: multiply matching entries, then add.
Worked example: v dot u, by hand
Our pair is $\mathbf{v} = [3, 2]$ and $\mathbf{u} = [1, 3]$. Multiply the across-parts: $3 \times 1 = 3$. Multiply the up-parts: $2 \times 3 = 6$. Add them: $3 + 6 = 9$.
$$\mathbf{v} \cdot \mathbf{u} = 3 \times 1 + 2 \times 3 = 3 + 6 = 9$$Now a second partner. Let $\mathbf{w} = [-2, 3]$, which leans up and to the left. Against v: $3 \times (-2) + 2 \times 3 = -6 + 6 = 0$. The two contributions canceled. Draw v and w on a scrap of paper and you will find they meet at a right angle.
And a third: reverse u to get $[-1, -3]$. Then $3 \times (-1) + 2 \times (-3) = -3 - 6 = -9$. The same size as before, now negative.
In words: the dot product of v and u is 9, so they agree. Swing a partner around to a right angle and the score falls to 0. Swing it all the way around to the opposite direction and the score becomes −9, as negative as these two lengths allow.
Written in general, for lists of any length:
$$\mathbf{v} \cdot \mathbf{u} = v_1 u_1 + v_2 u_2 + \cdots + v_d u_d = \sum_{i=1}^{d} v_i u_i$$In words: run down both lists together, multiply each pair of matching entries, and add up all the products. The big Greek S, sigma, is shorthand for "add up all of these," and $d$ is how many entries the lists have.
Two things about that formula matter. Matching slots never mix: entry 5 of one list only ever meets entry 5 of the other. And nothing in the recipe cares whether $d$ is 2 or 2,000. The arithmetic we just did by hand is, entry for entry, the arithmetic a model does with a thousand-number list. The list has been cut short so a person can hold it, and nothing else has been changed.
The shadow that turns it into a measurement
The dot product is a shadow, scaled by a length. One picture does the work here.
Stand the amber arrow up as a line and shine a light perpendicular to it. The mint arrow casts a shadow onto that line. That shadow, called a projection, is a signed length: positive if it falls forward along u, negative if it falls backward past the origin.
Call the angle between the two arrows $\theta$, "theta." The shadow's signed length is $\|\mathbf{v}\|\cos\theta$, where the double bars $\|\mathbf{v}\|$ mean "the length of v" and cosine is the function that turns an angle into an alignment score between $+1$ and $-1$. The dot product is that shadow times the length of the arrow we projected onto:
$$\mathbf{v} \cdot \mathbf{u} = \|\mathbf{v}\| \, \|\mathbf{u}\| \cos\theta$$In words: multiply the two lengths together, then scale the answer by how aligned the directions are. Perfect alignment scores the full product of the lengths. A right angle wipes it out to zero. Opposition makes it negative.
Both faces agree
Our pair: $\|\mathbf{v}\| = \sqrt{3^2 + 2^2} = \sqrt{13} \approx 3.61$ and $\|\mathbf{u}\| = \sqrt{1^2 + 3^2} = \sqrt{10} \approx 3.16$. The recipe already told us $\mathbf{v} \cdot \mathbf{u} = 9$, so the alignment score must be $9 \div (3.61 \times 3.16) \approx 0.79$, which is the cosine of about 38 degrees. Measure the angle in the figure above with a protractor and that is what you get. The shadow itself is $\|\mathbf{v}\|\cos\theta \approx 3.61 \times 0.79 \approx 2.85$, the violet segment in the figure.
In words: the counting recipe and the picture hand back the same 9 by two different routes. The rest of the book depends on those two routes agreeing.
The proof that the two faces always match is trigonometry, and I leave it out. The interactive below lets you test it on any pair of arrows you can drag.
A unit arrow is a ruler
One special case carries most of the weight in this book: projecting onto an arrow of length 1.
Such an arrow is a unit vector, marked with a little hat: $\hat{\mathbf{a}}$, "a-hat." Put $\|\hat{\mathbf{a}}\| = 1$ into the formula and one of the three factors disappears:
$$\mathbf{v} \cdot \hat{\mathbf{a}} = \|\mathbf{v}\| \cos\theta$$In words: dotting with a unit vector hands back the shadow itself, with no scaling left over. The number is how far along that direction v reaches.
So a unit vector is a ruler you can lay down anywhere in the space, pointing any way you like, and the dot product reads the measurement off it. That is what a coordinate is. A point does not own its coordinates. You get them by choosing a direction and taking a reading.
Reading v along a chosen direction
Take $\hat{\mathbf{a}} = [0.6, 0.8]$. Check first that it is a unit vector: $0.6^2 + 0.8^2 = 0.36 + 0.64 = 1$, and the square root of 1 is 1. Length 1, which is why those two numbers were picked.
Now read v along it: $3 \times 0.6 + 2 \times 0.8 = 1.8 + 1.6 = 3.4$.
Lay a different ruler down and the reading changes. Along $[1, 0]$, straight across, we get $3 \times 1 + 2 \times 0 = 3$, which is v's first entry. Along $[0, 1]$ we get 2, its second entry.
In words: the two numbers we started with, 3 and 2, were never intrinsic to the point. They were the readings from two particular rulers, across and up, that somebody chose before we arrived. Choose different rulers and the same point is described by different numbers.
Chapter 4 is about choosing better rulers on purpose, and almost every picture of embeddings you will ever see is the result of somebody doing that.
See it move
Drag both arrows and watch one number do all three jobs. The sign flips as you cross a right angle. Lengthening an arrow without turning it changes the dot product but not the alignment score. The shadow, when you switch it on, lands where the arithmetic says it will.
Where this shows up
The dot product turns up almost everywhere in machine learning, usually under a different name. Four places you will meet it:
- Retrieval. When a system finds the right document for your question, it turns both into lists of numbers and takes dot products. The winner is the one that agrees most.
- Attention. Inside a language model, every token asks every other token how relevant it is. That relevance score is a dot product, computed millions of times per sentence. The attention chapter of the LLM book builds the full mechanism; the score at its heart is the one we just built.
- The final prediction. A model's confidence in each possible next word is a dot product between its current internal position and a stored direction for that word. The Prediction Game follows that all the way to a probability.
- Interpretability. When researchers say they found a "direction for formality" inside a model, they mean they found a unit vector, and they read it with the dot product the way we read v along a-hat above.
That is the pattern of the book: a small number of measurements, applied in a space with enough room to make them meaningful. How much room a space needs is the next chapter but one.
What you now know
- A list of numbers can be read as a place, as a trip, or as a heading, and the question you are asking decides which reading you want.
- The dot product multiplies matching entries and adds them, giving one number that is positive for agreement, zero at a right angle, and negative for opposition.
- That same number equals the two lengths multiplied together and scaled by the cosine of the angle between them, so it mixes alignment with size.
- Dotting with a unit vector strips the scaling away and hands back a pure coordinate: how far a point reaches along a direction you chose.
- A point's coordinates are readings from rulers somebody picked, not properties the point owns.
Where we're headed
We have one instrument that mixes direction and length together. Sometimes that mixing is what you want and sometimes it is in the way, so the next chapter separates the two: a ruler for how far apart two points are, a protractor for which way they face, and a way of putting every point on the same sphere so that only the heading is left to talk about.