From catalog numbers to arrows
Cat has invented token ID 3061; dog has 5892. Those numbers tell us nothing about similarity. An embedding replaces an ID with an arrow, giving our vector tools something useful to compare.
A table with one row per token
The embedding table has V rows and d columns: vocabulary size by vector dimension. A lookup reads the row selected by the ID.
In words: cat's arrow goes 3 across and 4 up; dog's goes 4 across and 3 up. From here on we tag a vector with the word it stands for, as a small word under a bold v, so $\mathbf{v}_{\text{cat}}$ is read aloud as "the vector for cat."
Our two-dimensional coordinates are invented for teaching. In a model, the table's numbers are $\theta$, “theta,” the learned parameters. With 50,000 rows and 4,096 columns, it holds 204,800,000 numbers, about 200 million.
Similar meaning, similar direction
Add car=[4,−3] and truck=[3,−4]. All four arrows have length 5. Cat and dog share a direction; car and truck form another cluster.
![Four toy word vectors: cat [3,4], dog [4,3], car [4,−3], truck [3,−4]. The cat–car arc spans 90 degrees, giving cosine 0; cat–dog gives 0.96.](media/meaning-as-direction/meaning-map.svg)
Compare cat with dog
$$\mathbf{v}_{\text{cat}} \cdot \mathbf{v}_{\text{dog}} = 3 \cdot 4 + 4 \cdot 3 = 24$$In words: multiply the matching components and add. 3 times 4 is 12, 4 times 3 is 12, together 24.
$$\cos\theta = \frac{\mathbf{v}_{\text{cat}} \cdot \mathbf{v}_{\text{dog}}}{\|\mathbf{v}_{\text{cat}}\| \, \|\mathbf{v}_{\text{dog}}\|} = \frac{24}{5 \cdot 5} = \frac{24}{25} = 0.96$$In words: divide the 24 by both lengths; the directions agree 96 percent of the way to identical.
Every pair, by cosine
| Pair | dot product | $\cos\theta$ | reading |
|---|---|---|---|
| cat & dog | 24 | 0.96 | same cluster |
| car & truck | 24 | 0.96 | same cluster |
| cat & car | 0 | 0.00 | perpendicular, unrelated |
| dog & truck | 0 | 0.00 | perpendicular, unrelated |
| cat & truck | −7 | −0.28 | leaning apart |
| dog & car | 7 | 0.28 | leaning slightly together |
Within a cluster the score is near 1; between clusters it is near 0. Cat and car are perpendicular: $3 \cdot 4 + 4 \cdot (-3) = 12 - 12 = 0$, directions with nothing in common. A slightly negative score means two arrows leaning apart; a slightly positive one, leaning a little together.
Cosine 0.96 is an alignment score, not a 96% probability of shared meaning. Our toy cat and car have cosine zero. In richer learned maps, nurse may be near hospital and doctor; Paris near France. Such neighborhoods describe patterns in the data.
Nobody drew this map
Real embedding tables start with random values. Prediction training adjusts them along with the rest of the model. Cat and dog occur in similar contexts, such as “fed the” and “visited the vet,” so useful representations can develop related directions.
The map reflects use, not dictionary truth. Biased associations in the training text can become biased geometry. Nor must every raw input embedding have a perfect semantic neighborhood: its usefulness depends on the trained model and how we compare representations.
Arithmetic on meaning
Use a separate toy map with deliberately interpretable axes: one for royalty, one for a gender-coded relationship. Its points are man=[1,1], woman=[1,5], king=[6,1], queen=[6,5]. This is a drawing convenience, not a claim that learned coordinates have those labels.
![A quadrant plot with dimension 1 across and dimension 2 up. Dots at man (1,1), woman (1,5), king (6,1), and a dot at queen (6,5). Two identical arrows point straight up: man to woman labeled 'woman − man = [0, 4]', and king to queen labeled 'the same offset'. Muted dashed lines connect man to king and woman to queen, completing a parallelogram. An annotation reads 'king − man + woman lands here' pointing at the queen dot.](media/meaning-as-direction/analogy-parallelogram.svg)
An offset represents a relationship between points.
Reuse the relationship
$$\mathbf{v}_{\text{woman}} - \mathbf{v}_{\text{man}} = \begin{bmatrix} 1 \\ 5 \end{bmatrix} - \begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \\ 4 \end{bmatrix}$$In words: to get from man to woman you move 0 across and 4 up. Call that the woman-minus-man direction. This arrow is a relationship, detached from any particular word.
$$\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} = \begin{bmatrix} 6 - 1 + 1 \\ 1 - 1 + 5 \end{bmatrix} = \begin{bmatrix} 6 \\ 5 \end{bmatrix} = \mathbf{v}_{\text{queen}}$$In words: start at king, remove the man-direction, add the woman-direction, and you land on queen. In the figure, man→woman and king→queen are the same arrow $[0, 4]$ drawn from two different starting points, closing a parallelogram.
Real word-vector analogies are approximate: some country/capital and tense relationships work, others fail. Modern language models do not answer every question with this subtraction trick. The useful idea is that some relationships can share directions.
From two dimensions to thousands
| Space | Representation | What changes |
|---|---|---|
| 2D | Two numbers. | We can draw every arrow, but only two independent coordinate directions fit. |
| 4D | Four numbers. | The lists lengthen; matching-slot arithmetic stays the same. |
| 4,096D | 4,096 numbers. | More room for relationships such as tense, topic, and style without forcing them onto a plane. |
We write the space of d-number lists as $\mathbb{R}^d$, “R-d”; extra dimensions add room, not a new arithmetic rule.
$$\begin{bmatrix} 1 \\ 2 \\ 0 \\ 3 \end{bmatrix} \cdot \begin{bmatrix} 2 \\ 1 \\ 1 \\ 1 \end{bmatrix} = 1 \cdot 2 + 2 \cdot 1 + 0 \cdot 1 + 3 \cdot 1 = 7$$In words: multiply matching slots and add them up. It is the same recipe on a longer list. Cosine, lengths and offsets are identical, component by component, whether $d$ is 2 or 4096.
See it move
Watch IDs become arrows and the offset move between pairs. In Explore, compare directions; in Word math, test the analogy.
Why the LLM cares
The context enters as a sequence of embedding vectors. A lookup alone gives “bank” the same vector in “river bank” and “bank account.” Attention will let those surroundings reshape it.
Embedding whole passages also lets us search by similarity. That is the foundation of the retrieval chapter.
What you now know
- An embedding lookup trades a token ID for a vector.
- The table has V rows and d columns.
- Cosine measures direction agreement, not probability.
- Training learns the coordinates from usage.
- Offsets can capture relationships, approximately.
- Later layers add context to the fixed input vectors.
Where we're headed
Our tokens now have arrows. Next we'll turn the model's scores into predictions, including the softmax and temperature calculations.