The Geometry of Models

What is a list of numbers for?

Open any model and you find lists of numbers. A word is a list. A sentence is a list. A face, a radar sweep, a song, your last email — by the time a machine touches them they are all lists of numbers. The obvious question is what the numbers mean, and the honest answer is that individually they mean nothing at all. The useful question is different: what can we ask of a list?

This chapter answers that. A list of numbers is a place in a space, and once it is a place we can measure it — how far, which way, how much it agrees with another. Every capability in this book is one of those measurements. We are going to build the first and most important of them by hand, with numbers small enough to check on paper.

One list, three readings

Take the list $[3, 2]$. There are three different things we can decide it is, and all three are true at once.

The same two numbers, three jobs. A place is where you are; a trip is how you got there; a heading throws away how far you went and keeps only which way you faced.

Read as a place, $[3, 2]$ is a dot: go 3 across and 2 up from the origin and put your finger there. Read as a trip, it is an arrow — a vector — a direction plus a distance, which is the same instruction whether you start at the origin or at a street corner three towns over. Read as a heading, it is only the way the arrow points; its length has been thrown away.

We write a vector as a bold lowercase letter. Our running pair, kept for the whole book, is $\mathbf{v} = [3, 2]$ and $\mathbf{u} = [1, 3]$ — read them "the vector v" and "the vector u." In every figure on this site the first is drawn in mint and the second in amber.

$$\mathbf{v} = \begin{bmatrix} 3 \\ 2 \end{bmatrix} \qquad \mathbf{u} = \begin{bmatrix} 1 \\ 3 \end{bmatrix}$$

In words: v is the pair 3 and 2 — three across, two up. u is the pair 1 and 3 — one across, three up. Both lean up and to the right, but v leans flatter.

Which reading you want depends on the question. "How big is this?" is a question about the trip. "Are these two things alike?" — the question a search engine, a memory system and an attention head all ask thousands of times a second — turns out to be a question about the heading. Keeping the three readings separate in your head is most of the skill.

Agreement, measured

Here is the question we want a number for: do these two arrows point the same way? Not "are these lists equal" — we want a sliding scale. Strong agreement should score high. A right angle should score zero. Pointing against each other should score negative.

The instrument that does this is the dot product. The recipe has two steps and no cleverness in it: multiply matching entries, then add.

Worked example: v dot u, by hand

Our pair is $\mathbf{v} = [3, 2]$ and $\mathbf{u} = [1, 3]$. Multiply the across-parts: $3 \times 1 = 3$. Multiply the up-parts: $2 \times 3 = 6$. Add them: $3 + 6 = 9$.

$$\mathbf{v} \cdot \mathbf{u} = 3 \times 1 + 2 \times 3 = 3 + 6 = 9$$

Now a second partner. Let $\mathbf{w} = [-2, 3]$, which leans up and to the left. Against v: $3 \times (-2) + 2 \times 3 = -6 + 6 = 0$. Exactly zero, and it is worth pausing on how clean that is — the two contributions cancelled each other out perfectly. Draw v and w on a scrap of paper and you will find they meet at a right angle.

And a third: reverse u to get $[-1, -3]$. Then $3 \times (-1) + 2 \times (-3) = -3 - 6 = -9$. The same size as before, now negative.

In words: the dot product of v and u is 9 — they agree. Swing a partner round to a right angle and the score falls to 0. Swing it all the way round to the opposite direction and the score becomes −9, as negative as these two lengths allow.

Written in general, for lists of any length:

$$\mathbf{v} \cdot \mathbf{u} = v_1 u_1 + v_2 u_2 + \cdots + v_d u_d = \sum_{i=1}^{d} v_i u_i$$

In words: run down both lists together, multiply each pair of matching entries, and add up all the products. The big Greek S, sigma, is shorthand for "add up all of these," and $d$ is how many entries the lists have.

Two things about that formula deserve saying out loud, because they are the whole reason this operation runs the world. First, matching slots never mix — entry 5 of one list only ever meets entry 5 of the other. Second, nothing in the recipe cares whether $d$ is 2 or 2,000. The arithmetic we just did by hand is, entry for entry, the arithmetic a model does with a thousand-number list. We are not looking at a toy version of the real thing. We are looking at the real thing, with the list cut short so a person can hold it.

The shadow that turns it into a measurement

The recipe gives us a number, but it does not yet tell us what that number is. For that, one picture.

The dot product is a shadow, scaled. Shine a light straight down onto u's line and v casts a shadow. The dot product is that shadow's length times u's length.

Stand the amber arrow up as a line and shine a light perpendicular to it. The mint arrow casts a shadow onto that line. That shadow — called a projection — is a signed length: positive if it falls forward along u, negative if it falls backward past the origin.

Call the angle between the two arrows $\theta$, "theta." The shadow's signed length is $\|\mathbf{v}\|\cos\theta$, where the double bars $\|\mathbf{v}\|$ mean "the length of v" and cosine is the function that turns an angle into an alignment score between $+1$ and $-1$. The dot product is that shadow times the length of the arrow we projected onto:

$$\mathbf{v} \cdot \mathbf{u} = \|\mathbf{v}\| \, \|\mathbf{u}\| \cos\theta$$

In words: multiply the two lengths together, then scale the answer by how aligned the directions are. Perfect alignment scores the full product of the lengths; a right angle wipes it out to zero; opposition makes it negative.

Both faces agree

Our pair: $\|\mathbf{v}\| = \sqrt{3^2 + 2^2} = \sqrt{13} \approx 3.61$ and $\|\mathbf{u}\| = \sqrt{1^2 + 3^2} = \sqrt{10} \approx 3.16$. The recipe already told us $\mathbf{v} \cdot \mathbf{u} = 9$, so the alignment score must be $9 \div (3.61 \times 3.16) \approx 0.79$, which is the cosine of about 38 degrees. Measure the angle in the figure above with a protractor and that is what you will get. The shadow itself is $\|\mathbf{v}\|\cos\theta \approx 3.61 \times 0.79 \approx 2.85$ — the violet segment in the figure.

In words: the counting recipe and the picture hand back the same 9, by two completely different routes. That agreement is the hinge this whole book turns on.

We are not going to prove the two faces always match; that proof is a pleasant afternoon with some trigonometry and it would pull us off the road. The widget below lets you test it on any pair of arrows you can drag, which is a real kind of confidence.

A unit arrow is a ruler

Now the special case that matters more than any other, and the one the rest of this book keeps coming back to.

Suppose the arrow we project onto has length exactly 1. We call that a unit vector and mark it with a little hat: $\hat{\mathbf{a}}$, "a-hat." Put $\|\hat{\mathbf{a}}\| = 1$ into the formula and one of the three factors disappears:

$$\mathbf{v} \cdot \hat{\mathbf{a}} = \|\mathbf{v}\| \cos\theta$$

In words: dotting with a unit vector hands back the shadow itself — no scaling left over. The number is how far along that direction v reaches.

So a unit vector is a ruler you can lay down anywhere in the space, pointing any way you like, and the dot product reads the measurement off it. That is what a coordinate is: not something a point possesses, but something you obtain by choosing a direction and taking a reading.

Choose a direction, get a coordinate. The unit arrow is the ruler; the dot product reads it. Change the ruler's direction and the same point reports a different number.

Reading v along a chosen direction

Take $\hat{\mathbf{a}} = [0.6, 0.8]$. Check first that it really is a unit vector: $0.6^2 + 0.8^2 = 0.36 + 0.64 = 1$, and the square root of 1 is 1. Good — length exactly 1, which is why we picked these two numbers.

Now read v along it: $3 \times 0.6 + 2 \times 0.8 = 1.8 + 1.6 = 3.4$.

Lay a different ruler down and the reading changes. Along $[1, 0]$ — straight across — we get $3 \times 1 + 2 \times 0 = 3$, which is just v's first entry. Along $[0, 1]$ we get 2, its second entry.

In words: the two numbers we started with, 3 and 2, were never intrinsic to the point. They were the readings from two particular rulers — across and up — that somebody chose before we arrived. Choose different rulers and the same point is described by different numbers.

Hold on to that last sentence. Chapter 4 is about choosing better rulers on purpose, and almost every picture of embeddings you will ever see is the result of somebody doing exactly that.

See it move

Drag both arrows and watch one number do all three jobs. Three things to notice, in order: the sign flips exactly when you cross a right angle; lengthening an arrow without turning it changes the dot product but not the alignment score; and the shadow, when you switch it on, always lands where the arithmetic says it will.

Where this shows up

Almost everywhere, and usually without being named. Here is the honest list, so you can recognize the instrument when you meet it later wearing a different word.

Retrieval. When a system finds the right document for your question, it turns both into lists of numbers and takes dot products. The winner is the one that agrees most.
Attention. Inside a language model, every token asks every other token how relevant it is. That relevance score is a dot product, computed millions of times per sentence. The attention chapter of the LLM book builds the full machine; the score at its heart is the one we just built.
The final prediction. A model's confidence in each possible next word is a dot product between its current internal position and a stored direction for that word. The Prediction Game follows that all the way to a probability.
Interpretability. When researchers say they found a "direction for formality" inside a model, they mean they found a unit vector, and they read it with the dot product exactly as we read v along a-hat above.

One instrument, four jobs. That is the pattern of this whole book: a small number of measurements, applied in a space with enough room to make them meaningful. Which raises the question of how much room, and that is where we are going.

What you now know

A list of numbers can be read as a place, as a trip, or as a heading, and the question you are asking decides which reading you want.
The dot product multiplies matching entries and adds them, giving one number that is positive for agreement, zero at a right angle, and negative for opposition.
That same number equals the two lengths multiplied together and scaled by the cosine of the angle between them, so it mixes alignment with size.
Dotting with a unit vector strips the scaling away and hands back a pure coordinate: how far a point reaches along a direction you chose.
A point's coordinates are readings from rulers somebody picked, not properties the point owns.

Where we're headed

We now have one instrument that mixes direction and length together. Sometimes that mixing is what we want and often it is not, so next we separate the two — a ruler for how far apart two points are, a protractor for which way they face, and a way of putting every point on the same sphere so that only the heading is left to talk about.