The trick everyone has seen
king โ man + woman = queen is a statement about one map's shape, and it holds only because a rule nobody prints is switched on. This chapter takes it apart.
If you have read one popular article about how language models work, you have met that line. Take the position of "king," subtract the position of "man," add the position of "woman," and the place you land is "queen." Doing arithmetic on words and getting a sensible answer is surprising. It is also the most over-interpreted result in the field. What would it look like for this to be false? By the end of the chapter you will have watched the same procedure, run on the map we built by hand in the previous chapter, return the wrong word.
A difference is a direction
In Chapter 1 we subtracted one arrow from another and got a third arrow, the one that takes you from the tip of the first to the tip of the second. The arithmetic is the same here. The name changes when the two arrows are words.
The difference between two word positions is called an offset. It is a direction with a length, and it belongs to neither word but to the change between them. Subtract man from king and the offset you hold is not royalty and not masculinity. It is whatever is left when you cancel everything the two words share.
Stated carefully, the claim behind the famous example has two ifs in it. If a relation holds between many pairs of words, and if the map places every one of those pairs the same distance apart in the same direction, then that relation has become a direction you can travel in. We test both.
The parallelogram, closed by hand
Start with the famous one on a map small enough to check. Four words and four positions, the same coordinates the interactive further down uses, and the same four the LLM book's Meaning as Direction uses, where you met this arrow already. What that chapter does not do, and what we do next, is run the same procedure on coordinates nobody arranged. The four positions:
man at $[1, 1]$, woman at $[1, 5]$, king at $[6, 1]$, queen at $[6, 5]$.
Worked example: closing the parallelogram
Step 1, take the offset. Subtract man from king, component by component: $6 - 1 = 5$ across, and $1 - 1 = 0$ up. The offset is $[5, 0]$: five steps to the right, no change in height.
Step 2, apply it somewhere else. Start at woman, $[1, 5]$, and travel that offset. Across: $1 + 5 = 6$. Up: $5 + 0 = 5$. We land on $[6, 5]$.
Step 3, look at what is there. $[6, 5]$ is queen's position. The general form of what we just did is one line:
$$\mathbf{q} = \mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}}$$Step 4, check the other side. Subtract woman from queen: $6 - 1 = 5$ across, $5 - 5 = 0$ up. Also $[5, 0]$. The two offsets are the same pair of numbers, so the cosine between them, Chapter 2's measure of agreement in heading, is 1.000.
In words: the query point $\mathbf{q}$ is king's position minus man's position plus woman's position. On this map that arithmetic lands on queen's position with nothing left over, because the arrow from man to king and the arrow from woman to queen are the same arrow drawn in two places.
Four points arranged as a parallelogram are all there is to it, and the analogy is the statement that the fourth corner is where you expect. Nothing in the procedure knows anything about kings or gender. It knows that two subtractions produced the same pair of numbers.
Why a map would have that shape at all
The parallelogram follows from how the positions were earned. A word's coordinates are a summary of the company it keeps, so two words whose company differs in a consistent way end up displaced in a consistent way.
Think about what "king" and "man" are surrounded by in ordinary text. They share a great deal: both appear near people, near pronouns, near verbs of speaking and walking. Where they differ, they differ in one direction, toward thrones, crowns, kingdoms and reigns. Now look at "queen" and "woman." They differ along that same direction. Two pairs, one consistent difference in the counts, and the counts are what set the positions.
Now do it on the map we built
The royal map above is a teaching prop. Somebody chose those four positions to make the example come out. Our six-word map is different: every coordinate was summed from tally marks, and nobody arranged anything. So run the same procedure there.
Our map, from the previous chapter: tea $[6, 0]$, coffee $[5, 0]$, milk $[3, 1]$, cat $[1, 4]$, horse $[1, 5]$, dog $[1, 6]$. Across is counts near drink and hot, up is feed and walk.
A reasonable analogy to ask of it: milk is drunk and also fed to the cat, while tea is only drunk. So "tea minus milk" ought to be drink-ness with the animal-ness taken out, and applying that offset to cat should give us the drink least like an animal.
Worked example: the same procedure, counted coordinates
The offset. Tea minus milk: $6 - 3 = 3$ across, $0 - 1 = -1$ up. The offset is $[3, -1]$.
The query. Start at cat, $[1, 4]$, and travel it: $1 + 3 = 4$ across, $4 - 1 = 3$ up. The query point is $[4, 3]$.
The ranking. Now ask which word points most nearly the same way as $[4, 3]$, using Chapter 2's cosine. Take milk, $[3, 1]$: the dot product is $4 \times 3 + 3 \times 1 = 15$; the lengths are $5$ and $\sqrt{10} \approx 3.162$; so the cosine is $15 \div 15.811 = 0.949$. Do the same for the rest and the full ranking is milk 0.949, then coffee 0.800 and tea 0.800 tied to the last decimal, then cat 0.776, horse 0.745, dog 0.723.
In words: the winner is milk, at a cosine of 0.949, well ahead of everything else. And milk was one of the three words we typed into the question. The procedure handed us back our own input.
This is not a defect in our little map. It is the normal behavior of the offset method, and the reason is geometric. All six of our words point somewhere in the same quarter-turn of the plane, and the query sits angularly between tea and cat, the two words the offset was built from and landed on. Milk is the one word already sitting in that gap, so milk wins, even though milk is the word we subtracted. The famous demonstrations do not escape this. They handle it.
The exclusion rule
The handling is one line of code, and it almost never appears in the article you read: before ranking the candidates, delete the three input words from the list. Written out, the procedure is this:
$$\arg\max_{w \,\notin\, \{a,\, b,\, c\}} \cos(\mathbf{q},\, \mathbf{v}_w)$$In words: of all the words $w$ that are not one of the three you typed in, return the one whose direction agrees most with the query. That subscript is a rule about the candidate list, not about the geometry, and it is doing visible work.
Do that to our result. Strike out milk, tea and cat, and the ranking restarts from what is left: coffee 0.800, horse 0.745, dog 0.723. The answer becomes coffee, a good answer, a drink that has nothing to do with animals, and also the answer only because we changed the rules after seeing the first one.
There is something sharper in that second row. Tea scores 0.800 too, the same number to the last decimal, and no number of decimal places will ever separate the two. Tea would have been a defensible answer, and the only reason it is not the answer is that tea was an input. The rule separates words you typed from words you did not, which is not the same as separating good answers from bad ones.
| Map | Query | Top-1, inputs allowed | Top-1, inputs removed | Cosine |
|---|---|---|---|---|
| Royal, hand-placed | king โ man + woman | queen | queen | 1.000 |
| Six-word, counted | cat โ milk + tea | milk (an input) | coffee | 0.949 / 0.800 |
On the map built to make the answer come out, the rule changes nothing. On the map built from counts, it turns the answer from milk, one of the inputs, into coffee. Every benchmark score for this task was computed with the rule switched on, which is a defensible engineering choice, since nobody wants a search engine that answers a question with the question. But it means a reported analogy accuracy is not measuring "can the space do analogies." It measures something narrower, and the narrower thing has a name.
Run it yourself
The map below starts on our counted six words. Press the analogy and watch the offset arrow lift off milk-to-tea and re-land on cat, then read the ranking underneath. The checkbox is the rule we just named, and it starts off.
So what survives
- Offsets are real and useful. That a consistent relation in the text becomes a consistent direction in the space is a checkable property of these maps, and a great deal of later work is built on it. When you read about steering a model's behavior by adding a direction to its internal state, this chapter is what is being exploited.
- Offsets are also uneven. They hold well for mechanical, frequent relations such as singular to plural or country to capital city, and poorly for loose or abstract ones. The same space can score high on one relation and near zero on another, and a single headline number hides that.
- The benchmark measures something specific. What an analogy test reports is offset consistency on a curated list of word pairs, with the input words excluded. That is a real property, worth measuring. It is not "the model understands relations," and the gap between those two sentences is where most of the confusion about embeddings lives.
Two papers make the careful version of this argument: Tal Linzen's Issues in evaluating semantic spaces using word analogies (2016), which separates how much of the score comes from the offset and how much from the neighborhood, and Rogers, Drozd and Li's The (too many) problems of analogical reasoning with word vectors (2017). The exclusion rule itself is spelled out in Levy and Goldberg's Linguistic Regularities in Sparse and Explicit Word Representations (2014).
Where you'll meet this
- Activation steering. Part III takes the difference between two sets of internal states, polite replies minus rude ones for instance, and adds that offset back into a running model to shift its behavior. That is this chapter's arrow, computed on activations instead of word positions, and it inherits this chapter's caveat: the direction is only as consistent as the examples it was averaged over.
- Model cards. When a card reports an analogy benchmark, you now know what the row means and what it does not. It is offset consistency on a curated pair list with the inputs deleted, and a single headline figure can hide a relation the space handles badly.
- Retrieval that hands you back your own question. If you have ever searched a document store and seen your query text returned as the top hit, that is the failure we watched on our map: the query sits nearest to the thing it was built from. Production search fixes it the way the analogy literature does, by excluding the source from the candidate list, and now you know to ask whether it did.
If you arrived here wanting the plain-language version of what an embedding is before any of this arithmetic, the Q&A answer What are embeddings? covers it without the geometry.
What you now know
- The difference between two word positions is an offset: a direction and a length that stands for the change between them and belongs to neither word.
- An analogy closes when the same offset is reused between two different pairs, which makes the four points a parallelogram and the answer the fourth corner.
- A map has that shape when the corpus used a distinction consistently, so the geometry is evidence about the text that was counted and not about language itself.
- Run the same procedure on counted coordinates and the top answer is usually one of the words you typed in, because the query sits near all three of its ingredients by construction.
- Every implementation silently deletes the three input words before ranking, and on our map that exclusion rule, rather than the geometry, is what turns the answer from milk into coffee.
- What an analogy benchmark reports is offset consistency on curated pairs with the inputs removed, which is a narrower and more useful claim than the one the famous example seems to make.
Where we're headed
We have a map whose positions were earned by counting, and we have seen what its geometry encodes and what it only appears to. Counting is not the best we can do. The next chapter puts the map on a workbench and shapes it on purpose: you say which pairs of words should be pulled together and which pushed apart, write that down as a loss, and let the positions move. One step, by hand, with numbers you can check.