This one came from LT David Robinson, USN, and he found the right seam. He had met embeddings twice: once as the thing that turns a token into numbers at the front of a language model, once as the thing that powers semantic search. The two stories did not fit together. If embeddings give a general answer when you search with them, never an exact text match, how can the same idea hand a token an exact value?
The answer has three parts. A token's embedding is an exact lookup: a token identification number is a row address, the model reads that row, and nothing is searched, averaged or guessed. A search over embeddings is only ever approximate, in two separate ways, and I keep those two apart below. Embeddings also do a great deal more than search. The lookup at the front of a model is the job they were invented for, and retrieval is one customer among several.
The three jobs one word does
An embedding is a way of standing something up as a vector, an ordered list of numbers, so that a machine can do arithmetic on it. That is the whole of the definition. It says nothing about what is being represented, where the numbers came from, or what you are going to do with them, and those three choices are where all the confusion lives.
Door one, look up: a token's identification number selects a stored row. Door two, learn: training nudges that row a little at a time. Door three, compare: two vectors are scored against each other, and the scores put a list in order. David met door one and door three, and they behave differently because they are different operations, and not because one is a softer version of the other.
A token's embedding is a row in a table
Before a model sees your text, a tokenizer chops it into tokens and replaces each one with its number in the model's vocabulary. That number is an address. It measures nothing. Token 43 is not more like token 42 than it is like token 900; the numbering is an accident of how the vocabulary was built.
The model carries a table with one row per vocabulary entry and one column per coordinate. Pretend for the rest of this chapter that the whole vocabulary is three tokens of two coordinates each, so we can draw all of it:
| Token ID | Token | Stored row |
|---|---|---|
| 0 | river | [1, 0] |
| 1 | bank | [2, 1] |
| 2 | money | [0, 2] |
Reading row one
Write the table as $\mathbf{E}$, a bold capital letter because it is a matrix, a rectangle of numbers. Square brackets after it mean "read the row at this address." A token vector is written with the word in the subscript, the way the rest of this family writes them.
$$\mathbf{v}_{\text{bank}} = \mathbf{E}[1] = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$$In words: the vector for bank is whatever sits in row one of the embedding table, and in our pretend model that is the two numbers 2 and 1. Every time token number one appears, in any sentence, in any position, on any day, the model reads back those same two numbers.
That is the answer to the first half of David's question. The exactness has nothing to do with meaning and everything to do with addressing. It is the same exactness as pulling file 1 out of a filing cabinet: you do not find file 1 by searching for something that looks like file 1.
Two qualifications. "Exact" means the stored numbers are read back unchanged, not that 2 and 1 are the uniquely correct coordinates of the English word bank; another model would store different numbers there and work just as well. And nobody decided that the first coordinate means water and the second means money. The row is learned, which is the next section.
Where the rows come from: training moves the rows
The table is not a reference work the model consults. It is part of the model's parameters, and it is trained by the same process as every other weight. The model reads an example, predicts, and is scored by a loss. Then the arithmetic runs backward and reports, for every adjustable number in the model, which direction would make the loss worse. That report is the gradient, written $\nabla L$ and read aloud as "the gradient of the loss." Each row of the embedding table gets its own slice of it.
![Two panels. On the left, a bowl-shaped loss surface drawn in three dimensions over the two coordinates of the row for bank, with a dot on its side at the stored row [2, 1] and an arrow from the dot running down the slope. On the right, the same bowl seen from above as contour rings around the minimum at [1.5, 2]. At the dot [2, 1] a red arrow labeled the gradient, [0.5, −1], points outward, uphill and away from the rings' center; a shorter mint arrow labeled minus 0.1 times the gradient points the opposite way, to [1.95, 1.10], one step closer to the center.](media/what-are-embeddings/gradient.svg)
One learning step on bank
Say this example produced a gradient of $[0.5, -1]$ for row one, and the run uses a learning rate of $\eta = 0.1$. That symbol is the Greek letter eta, and it sets how big each step is. Gradient descent takes a small step against the gradient, because the gradient points the way the loss goes up:
$$\begin{aligned} \mathbf{v}_{\text{bank}} &\;\leftarrow\; \mathbf{v}_{\text{bank}} - \eta\,\nabla L \\ &= \begin{bmatrix} 2 \\ 1 \end{bmatrix} - 0.1 \begin{bmatrix} 0.5 \\ -1 \end{bmatrix} = \begin{bmatrix} 1.95 \\ 1.10 \end{bmatrix} \end{aligned}$$In words: multiply each gradient number by the step size 0.1, subtract the results from the stored row, and write the answer back into the table. First coordinate: 0.1 × 0.5 = 0.05, and 2 − 0.05 = 1.95. Second coordinate: 0.1 × (−1) = −0.1, and 1 − (−0.1) = 1.10. The second coordinate went up, because subtracting a negative number adds. Row 1 now reads [1.95, 1.10], and the very next lookup of token 1 returns those.
The arrow slid a little. It was not re-invented. Repeat that a few hundred billion times under the pressure of predicting real text and the rows arrange themselves into something useful. Nobody described meaning to the model; rows that make the next token easier to predict are the ones that survive. That step is the subject of Rolling Downhill.
Inside the model, the row stops being the word
This part surprises people who have only met embeddings as search. The looked-up row is the input to the model, not its reading of your sentence. It is identical for every occurrence of the token, which by itself would leave bank in "we sat on the river bank" indistinguishable from bank in "she moved the money to the bank." The layers fix that.
The mechanism is attention, covered properly in Attention. Each position gathers information from the other positions, so the vector sitting at bank's position picks up river in one sentence and money in the other. Between those mixing steps sit layers that multiply by learned weights and then bend the result through a fixed curve, the nonlinear operations. The result at that position is a contextual representation, and the two banks end up in different places.
Two things to hold on to. The separating is deterministic: same sentence, same weights, same output vector, every time. And the object that does the work inside a language model is the contextual representation rather than the table row.
Search is a different embedding, of a different thing
Now David's other encounter. When a search product says it uses embeddings, it is almost never using the table above. It is running a separate trained model, an encoder, over an entire passage and emitting one vector for the whole passage. The encoder looks up token rows internally, then mixes and gathers them, often by averaging the final per-token vectors, a step called pooling. A different thing is represented, with a different training objective and a different table.
Your query goes through that same encoder, or through a partner encoder trained alongside it. That is not a detail: two embeddings are comparable only if they were trained together, equal length proves nothing, and swapping the encoder means recomputing every stored vector in the index.
Comparison is by direction. If two vectors both have length one, the dot product of the pair is the cosine of the angle between them, and that is the score nearly every vector search reports.
$$\cos\theta = \frac{\mathbf{q} \cdot \mathbf{a}}{\|\mathbf{q}\|\,\|\mathbf{a}\|}$$In words: take the dot product of the query vector $\mathbf{q}$ and a passage vector $\mathbf{a}$, then divide by both of their lengths. The two vertical bars mean "the length of". A vector's length is the square root of the sum of the squares of its coordinates. Dividing by the lengths is what makes the score about direction only, so a long passage cannot win by being long.
One query, two passages
A different model and a different table from the one above: this is the search space now. The query is $\mathbf{q} = [1, 0]$, passage A is $\mathbf{a} = [0.8, 0.6]$, and passage B is $\mathbf{b} = [0, 1]$. All three already have length one: for A, $0.8^2 + 0.6^2 = 0.64 + 0.36 = 1$, and the square root of one is one. So the division does nothing and the cosine is the dot product.
$$\begin{aligned} \mathbf{q} \cdot \mathbf{a} &= 1 \times 0.8 + 0 \times 0.6 = 0.8 \\ \mathbf{q} \cdot \mathbf{b} &= 1 \times 0 + 0 \times 1 = 0.0 \end{aligned}$$In words: passage A scores 0.8 and passage B scores 0.0, so A is ranked first and B last. Both numbers came out of definite coordinates, with no approximation anywhere in the arithmetic. What is approximate is the claim that 0.8 means A answers the question.
One more approximation sits on top of all this. Scoring a query against ten million stored passages one at a time is too slow, so production systems build an approximate nearest-neighbor index that skips most of the comparisons. It can miss the mathematically closest vector. You can switch that approximation off by scanning everything; you cannot switch off the other one.
Why "general, not exact" is right here and wrong elsewhere
So there are two approximations in a vector search and they have nothing to do with each other. The index may not check every candidate, which is an engineering trade you can buy your way out of. The representation may rank a passage that does not answer you above one that does, which is a property of what the encoder learned and no amount of compute repairs it.
The second is the behavior people call "general," and for its intended job it is the point. Someone asking "how do I reset my password?" should be handed a passage reading "recover access by choosing a new passphrase," which shares not one significant word with the query. Matching on words fails there. Matching on direction works, because direction is a loose learned summary rather than a transcript.
That gives a clean rule for the desk. When you need this string and no other, such as a hull number, a quotation, a date or a negation, use a tool that matches characters: grep, a database WHERE clause, a word-matching search index. Those are exact, auditable, and will never hand you a near miss. When you need whatever is about this, use vectors. Serious retrieval systems run both and merge the two result lists, which the field calls hybrid search. Reach for the vector when the wording is unpredictable, and for the exact matcher when it is not.
Same word, five spaces
The word stretches further still. Every row below is called an embedding in ordinary use, every one is a list of numbers, and none of them can be compared with any other.
| Space | What one vector stands for | Where you meet it |
|---|---|---|
| Token table | One vocabulary entry | The front of every language model |
| Contextual states | One token in one sentence | Inside the model, after the layers |
| Passage encoder | A whole query or passage | Vector search and retrieval |
| Image and text together | A picture or a caption, in one shared space (CLIP is the best-known) | Image search, captioning, content filters |
| Speaker and face | One person's voice or face | Biometric matching, and splitting a recording up by who is speaking |
The fourth row shows what compatibility requires. In a jointly trained image-and-text model, a photograph of a destroyer and the words "a destroyer under way" land near each other, because the two encoders were trained side by side with the objective of putting matched pairs close together. Pictures and words are not secretly the same kind of thing. The compatibility was manufactured.
And nowhere in any of these spaces do the individual coordinates carry tidy names. There is no universal map of meaning that everyone samples from: each model has its own space, and the only reason a number in it means anything is that the rest of that same model knows how to read it.
Retrieval finds; the model answers
The last confusion worth clearing is about what comes back. An embedding search does not return an answer at all. It returns identification numbers, and a separate store hands back the original text those numbers point to. The vector is not a compressed copy of the passage and the system never decodes it; the text comes from the store.
The general-sounding answer arrives later, when a language model is handed those retrieved passages and asked to write from them. That two-stage arrangement is the subject of RAG and Vector Search, and the division of labor is the point: the embedding finds roughly relevant material fast and is bad at being exact, while the language model reads and connects and is bad at knowing when it has nothing to read. Skip the finding step and you get the failure described in Why It Makes Things Up.
So keep the sources and check them. Retrieval tells you where to look, and certifies nothing about what you find there.
What you now know
- An embedding is anything represented as a vector of learned numbers, and the word covers at least three different operations: looking a row up, learning that row, and comparing two vectors.
- A token's embedding is an exact lookup: the token's identification number is a row address, and the model reads back the stored row unchanged, with no searching and no averaging involved.
- That row was learned rather than derived from meaning. One gradient step moved the row for bank from [2, 1] to [1.95, 1.10], and the finished row is the sum of a vast number of such nudges.
- The row is only the input: attention and the layers after it turn the identical starting row into different contextual representations for "river bank" and "money bank."
- Search uses a different model over a different object, compared by cosine (0.8 and 0.0 for our passages), and it is approximate twice over: once in the index and once in the representation, and only the first of those can be switched off.
- For an exact string, such as a hull number, a date or a negation, reach for a character-matching tool, and let vectors handle the part where the wording is unpredictable.
Where we're headed
David, I hope that closes the gap: exact at the address, approximate at the angle, and a good deal more going on than search. One thing in this chapter went by without inspection, though. I asked you to pretend the vocabulary had three tokens, and the table had three rows because of it. A real table has tens of thousands, one row per vocabulary entry, and that count is a decision somebody made rather than a fact about English. The next question asks why there are so many tokens in a vocabulary. That is the same table, counted down its left edge instead of read across.