Meaning as Direction

From catalog numbers to arrows

Chapter 7 left us holding library cards. "Cat" is token 3061; "dog" is token 5892 — two numbers no more related than two strangers' phone numbers (the IDs are invented, as they were last chapter). And yet every chatbot you have ever used plainly treats cat and dog as similar things. Somewhere between the tokenizer and the reply, meaning gets in. This chapter is where.

Here is the whole move, in one sentence — and then we spend the rest of the chapter earning it: give every token an arrow — a vector, straight from Chapter 1 — and place the arrows so that tokens with similar meaning point in similar directions. The vector a token is handed has a name: it is called an embedding. Trading a bare ID for an arrow is embedding the token.

This is the payoff chapter for the entire mathematical toolkit of Part I. The arrows of Chapter 1 become word-meanings. The dot product of Chapter 2 becomes a meaning-o-meter. The matrix of Chapter 3 becomes the table that holds it all. If Part I ever felt abstract — arrows and grids for their own sake — today it stops being abstract.

A table with one row per token

The mechanism is almost disappointingly plain: it is one big table. It has $V$ rows — one per vocabulary token, where $V$ is the vocabulary size we met in Chapter 7 — and $d$ columns, where $d$ is the number of dimensions, the length of each token's list of coordinates. A table of numbers with rows and columns is exactly Chapter 3's object: a matrix. Looking up token 3061's embedding means reading row 3061. No arithmetic — a fetch.

For teaching we set $d = 2$, so every arrow lives on the flat plane we know from Chapter 1. Our toy embeddings for this chapter are invented, hand-placed numbers — real ones are learned, which is the story two sections from now. With that promise on the record, here are our first two arrows as column vectors:

In words: cat's arrow goes 3 across and 4 up; dog's goes 4 across and 3 up. From here on we tag a vector with the word it stands for — a little word under a bold v — so $\mathbf{v}_{\text{cat}}$ is read aloud as "the vector for cat."

Real models are not this cozy. They use $d$ in the hundreds or thousands; a popular mid-size open model gives each token $d = 4096$ numbers. With about $V \approx 50{,}000$ rows of $d \approx 4{,}096$ numbers each, the table alone holds about 200 million numbers — and every one of them is a learned parameter, part of the $\theta$ — theta, the collective name for every tunable number in the model — that training will tune in Part III. (You met $\theta$ in passing back in Chapter 5.)

And notice where this table sits: it is literally the model's front door. IDs arrive from the tokenizer; each one pulls its row; and from that instant the model computes with arrows and never sees the IDs again.

Similar meaning, similar direction

Let's finish the toy map with two more words: $\mathbf{v}_{\text{car}} = [4, -3]$ and $\mathbf{v}_{\text{truck}} = [3, -4]$. Now we have four arrows, and here is a small delight: every one of them has length exactly 5, because each is a 3-4-5 right triangle — the very triangle that kept Chapter 2's arithmetic clean. Cat and dog point up-and-to-the-right together; car and truck huddle down-and-to-the-right together. The picture already tells the whole thesis: similar meaning is similar direction, and clusters are topics.

Now let's measure instead of squint — with Chapter 2's tool, worked in full exactly once. The dot product of the cat and dog arrows is:

In words: multiply the matching components and add — 3 times 4 is 12, 4 times 3 is 12, together 24.

Both arrows have length 5, so cosine similarity — the dot product divided by both lengths — comes out to:

In words: divide the 24 by both lengths; the directions agree 96 percent of the way to identical.

The remaining pairs follow the same recipe. Rather than turn every handle again, here is the scoreboard — each cosine is the dot product over $5 \times 5 = 25$:

Widen the lens past four words and the same picture holds. In a fuller map, "nurse" sits near "hospital", "doctor", and "medicine"; "Paris" sits near "France"; "Tuesday" sits next to "Wednesday". Neighborhoods are topics, and cosine similarity is the ruler that turns the vague word "near" into a number — the single most-used measurement in this entire field.

Nobody drew this map

The obvious question: who placed the arrows? For our toy map, Sean did, last Tuesday. For a real model, the honest answer is nobody. The table's 200 million numbers start as random noise — every arrow pointing in a random direction — and then get nudged, over and over, by the training process (the downhill steps of Chapter 5) so that the model's next-token predictions improve. That is the entire recipe.

Why should prediction-pressure organize meaning? In plain language: words that show up in the same kinds of sentences must earn similar predictions. "Cat" and "dog" both appear before "chased the ball", after "fed the", near "vet". The cheapest way for the model to predict well across all those contexts is to give the two words nearby arrows. Meaning emerges as a side effect of practicing prediction. The linguist J. R. Firth put it best in 1957: "You shall know a word by the company it keeps."

One honest consequence: the map encodes how words are used in the training text, not dictionary truth. If the text uses a word in biased ways, the geometry inherits the bias — "nurse" can end up nearer to "she" than to "he." Real systems measure this and partially correct for it; what matters here is that the map is an empirical object, not a moral one.

Arithmetic on meaning

Time for the party trick — with full disclosure. We now build a second toy map, separate from the cat/dog map, and this time we cheat on purpose: we choose axes that mean something, so the effect is visible. Read dimension 1 as roughly "royalty" and dimension 2 as roughly a "gender-coded direction." In real embeddings the same structure genuinely exists, but it lies diagonal to the axes — the trick still works there, it just can't be read off the coordinates. (Yes, the previous section warned you not to read the axes; here we own the cheat out loud.)

Four points: $\mathbf{v}_{\text{man}} = [1, 1]$, $\mathbf{v}_{\text{woman}} = [1, 5]$, $\mathbf{v}_{\text{king}} = [6, 1]$, and $\mathbf{v}_{\text{queen}} = [6, 5]$. Start by computing the offset from man to woman:

In words: to get from man to woman you move 0 across and 4 up — call that the woman-minus-man direction. The key idea is that this arrow is a relationship, detached from any particular word.

Now the famous computation, componentwise so you can check every digit yourself:

In words: start at king, remove the man-direction, add the woman-direction — and you land exactly on queen. In the figure, man→woman and king→queen are the same arrow $[0, 4]$ drawn from two different starting points, closing a parallelogram.

Honesty, mandatory: in real, learned embeddings this held only approximately. In a famous 2013-era word-vector system (word2vec), king − man + woman landed closer to queen than to any other word in the vocabulary, and headlines were duly written. It is approximate; it works for some analogies (countries and their capitals, verb tenses) and misses on others; and modern LLMs do not navigate by party tricks. The durable lesson is quieter and bigger: relationships have consistent directions. The same little arrow that turns man into woman turns king into queen — meaning has a geometry you can do arithmetic on.

Name what you just did: you added and subtracted meanings using Chapter 1's vector arithmetic, and it worked. That was the promise on the tin of Part I.

From two dimensions to thousands

Why not stay in 2D forever? Room. On a flat plane you can have only two fully perpendicular directions — but meaning has far more independent shades than two: singular/plural, past/present, big/small, formal/casual, French/English, animal/vehicle, and on and on. Each independent shade wants its own direction that does not interfere with the others, and $d$ dimensions supply room for a huge number of nearly-perpendicular directions.

A gentle notation upgrade. Chapter 1 called the plane of all 2-number lists $\mathbb{R}^2$, read "r-two." The space of all $d$-number lists is written $\mathbb{R}^d$, read "r-d." It is not a mystical place; it means "lists of length $d$," nothing more. One word of comfort: nobody — not Sean, not the people who build these models — can picture 4096 dimensions. You reason there by trusting the recipes, not by squinting harder.

And the recipes genuinely do not change. Here is a dot product in $d = 4$, worked in one line so you feel the sameness:

In words: multiply matching slots, add them up — the same recipe, just a longer list. Cosine, lengths, and offsets are all identical, component by component, whether $d$ is 2 or 4096.

What does the extra room buy? Concretely: in $d = 2$ our four words already crowd each other — cat and truck are forced to relate somehow, because on a plane every direction is near every other. In $d = 4096$, "cat" can be close to "dog" along one direction, close to "pet" along another, and far from "carburetor" along all of them at once. High dimension is what lets 50,000 tokens each keep the right neighbors without striking up false friendships.

See it move

Two dead catalog numbers become arrows; the cosine 24/25 = 0.96 appears from Chapter 2's recipe; and the woman-minus-man arrow slides across the map to turn king into queen. The short video puts the whole chapter in motion — the same numbers you just worked by hand.

Now take the controls. The explorer below is a toy meaning-map you can poke. It has two modes: Explore (tap words, read their arrows and cosines) and Word math (run the analogy machine).

Why the LLM cares

Locate the table in the machine, unambiguously: the embedding table is layer zero of every LLM. Your prompt becomes IDs (Chapter 7); each ID pulls its row; and from that instant the model computes exclusively on arrows. Every later chapter of this site — attention, the transformer, training, LoRA — is arithmetic performed on the vectors born right here.

There is a limitation, though, and it sets up the next two chapters like a puzzle. The table gives one arrow per token, fixed no matter the sentence. But "bank" in "river bank" and "bank" in "bank account" should not mean the same thing — and the table hands both the exact same row. Something after layer zero must let context reshape each token's arrow. That something is called attention, and it is Chapter 10's whole story.

This trick escapes the LLM entirely, too. Text in, arrow out, cosine to compare — the same three moves power search engines, recommendation systems, and the "give the model a library card" pattern called RAG that Chapter 19 builds. Learning today's chapter is learning half of modern AI infrastructure.

So you can now say precisely what it means for a machine to "know" that cat and dog are similar: their arrows' cosine is large. And, equally precisely, what it does not mean: there are no definitions, no understanding-shaped storage — just geometry that prediction-pressure carved.