Counting has a ceiling
Counting cannot bring together two things that belong together but never appear near each other. To fix that you stop taking the positions you are given and start moving them on purpose.
Two chapters ago we built a map out of tally marks, and it worked. Tea and coffee landed together because they keep the same company, and the animals landed apart because they keep different company: a real map for the price of a few evenings with a notebook. But consider "pay stub" and "remittance advice." They mean the same thing to an accounts department, and one lives in American documents while the other lives in British ones. A corpus holding both may never put them near anything they share, so they get different coordinates, and no window width fixes it.
This chapter is the workbench, the clamp, and the one arithmetic step that shapes a map into the shape you asked for.
Say who should be close
The mechanism is unglamorous. You hand the training procedure a list of pairs, and for each pair you say one of two things: these two belong close together, or these two belong apart.
A pair you want close is a positive pair; a pair you want apart is a negative pair. That is the whole vocabulary, plus one name for the word being moved: the anchor.
We will use one triple from the map we already have. The anchor is milk at $[3, 1]$. Its positive is tea at $[6, 0]$, because both are drinks and I want them nearer. Its negative is dog at $[1, 6]$, because I want milk further from the animals than it sits.
A loss is a rule about pairs
To move anything we need a number that says how badly the arrangement disagrees with our wishes and gets smaller as it improves. That number is the loss, and we build it one wish at a time.
First wish: milk should be near tea, and the squared distance between them measures how far that is from true. Second wish: milk should be far from dog, where the same quantity is something we want large, so we subtract it. With $\mathbf{a}$ the anchor, $\mathbf{p}$ the positive and $\mathbf{n}$ the negative:
$$L = \|\mathbf{a}-\mathbf{p}\|^2 - \|\mathbf{a}-\mathbf{n}\|^2$$In words: the loss is the squared distance from the anchor to the thing it should be near, minus the squared distance from the anchor to the thing it should be far from. Making the first term small and the second large both push this number down, which is what we wanted from a single score.
Worked example: the loss right now
Anchor to positive. Milk $[3, 1]$ to tea $[6, 0]$: across, $3 - 6 = -3$; up, $1 - 0 = 1$. Square and add: $9 + 1 = 10$.
Anchor to negative. Milk $[3, 1]$ to dog $[1, 6]$: across, $3 - 1 = 2$; up, $1 - 6 = -5$. Square and add: $4 + 25 = 29$.
Subtract. $10 - 29 = -19$.
In words: the loss starts at โ19.000. A negative loss is fine here. This score has no floor of zero, and all we ask of it is that it goes down.
The gradient is a displacement
Now differentiate. In Rolling Downhill we met the gradient as the direction of steepest increase, and the rule that you move against it to go down. Here the gradient turns out to be something you can draw.
Differentiate $L$ with respect to the anchor's position, term by term. The derivative of $\|\mathbf{a}-\mathbf{p}\|^2$ is $2(\mathbf{a}-\mathbf{p})$, an arrow from the positive toward the anchor. The derivative of $-\|\mathbf{a}-\mathbf{n}\|^2$ is $-2(\mathbf{a}-\mathbf{n})$, an arrow from the anchor toward the negative. Add them and the anchor cancels:
$$\nabla_{\mathbf{a}} L = 2(\mathbf{n} - \mathbf{p})$$In words: the gradient of the loss with respect to the anchor is twice the vector from the positive to the negative. The anchor's own position vanished in the addition, so where the anchor sits does not change which way it is told to go.
Worked example: one step, in full
The gradient. Dog minus tea: across, $1 - 6 = -5$; up, $6 - 0 = 6$. Double it: $[-10, 12]$.
The update rule. Move against the gradient, scaled by the learning rate $\eta$, eta, the number that sets how big a step is:
$$\mathbf{a} \leftarrow \mathbf{a} - \eta \, \nabla_{\mathbf{a}} L$$We take $\eta = 0.05$. Smaller and the move is too small to see on a six-by-six grid; larger and milk overshoots into empty space below the map, which you can make happen in the interactive below.
The arithmetic. Across: $3 - 0.05 \times (-10) = 3 + 0.5 = 3.500$. Up: $1 - 0.05 \times 12 = 1 - 0.6 = 0.400$. Milk moves from $[3, 1]$ to $[3.500, 0.400]$.
The new loss. To tea: $(3.5-6)^2 + (0.4-0)^2 = 6.25 + 0.16 = 6.41$. To dog: $(3.5-1)^2 + (0.4-6)^2 = 6.25 + 31.36 = 37.61$. So $L = 6.41 - 37.61 = -31.200$.
In words: one step of 0.05 took the loss from โ19.000 to โ31.200. Milk got closer to tea and further from dog in one move, because both wishes were encoded in one number and one number has one gradient.
The enlargement is worth a second look. The pull contributes $0.05 \times 2([6,0]-[3,1]) = [0.3, -0.1]$ and the push contributes $0.05 \times 2([3,1]-[1,6]) = [0.2, -0.5]$. Laid tip to tail they reach $[0.5, -0.6]$, which is the move. The push is doing most of the work, and it is doing it because dog lies in a direction we want to leave rather than because dog is close.
One step is a nudge; training is a settling
Look at what else changed while milk was moving half a unit.
Five numbers moved and every one moved the way we wanted, which is worth being suspicious of, because the loss only ever mentioned distance. Chapter 2 was a whole chapter on distance and cosine being different rulers that routinely disagree. Here they happened to agree. The cosine to tea climbed from 0.949 to 0.994 as a by-product, not because anything in $L$ was watching it. A contrastive loss built on squared distance makes a promise about distances and none about angles.
And one step is a nudge. Real training runs thousands of steps over thousands of triples, where every point is an anchor in some and a negative in others, so everything moves at once.
Those twelve points started scattered at random. The grouping visible by step 200 was never drawn. It was specified, as a list of pairs, and the arithmetic did the drawing. When someone says an embedding model "learned that these documents are related," this is the machine they mean, and what it learned was somebody's pair list.
The margin, and why the push stops
Our loss has a flaw you may already have spotted: it has no bottom. Every extra unit of separation between milk and dog lowers $L$ forever, so if we keep stepping, milk flees rather than settling anywhere sensible.
The fix, introduced for face recognition by Schroff and colleagues in FaceNet (2015), is to state how much separation is enough and stop paying for more. That number is the margin, written $m$:
$$L = \max\!\left(0,\ \|\mathbf{a}-\mathbf{p}\|^2 - \|\mathbf{a}-\mathbf{n}\|^2 + m\right)$$In words: take the same difference as before, add the margin, and if the result is negative, call the loss zero. A triple that already has the separation you asked for contributes nothing to the loss and therefore nothing to the gradient, so it stops pushing.
Take $m = 25$, measured in the same squared units $L$ is. Before our step the gap, meaning how much further dog sits than tea, was $29 - 10 = 19$, short of 25, so the triple was unhappy by 6 and pushed. After the step the gap is $37.61 - 6.41 = 31.2$, past the margin, so this triple's loss is zero and it sits out the rest of training.
That is why the loss curve in the settling figure hits zero and stays there. The points did not run out of room. Every triple got the gap it asked for and switched itself off.
Where the pairs come from
Nobody writes millions of pairs by hand. They are harvested, and the harvesting rule is the most consequential decision in building an embedding model.
- The next sentence. A sentence and the one after it, called a positive pair. Cheap, unlimited, and it teaches that adjacent text is similar.
- The same sentence, twice. Run one sentence through the model twice with different random dropout and call the results a positive pair. That is the trick behind SimCSE (Gao, Yao and Chen, 2021), where the pair list comes from nothing but the model's own noise.
- The click log. A query and the result somebody clicked. This is how retrievers are built, and it shapes the space by what users found satisfying rather than by what is true.
- Everything else in the batch. Rather than choosing negatives, call every other item in the mini-batch one. This is the in-batch negative, formalized as InfoNCE by van den Oord and colleagues in Representation Learning with Contrastive Predictive Coding (2018), and it is why batch size appears in embedding papers as a quality knob rather than a memory detail.
Choosing negatives deliberately is called negative sampling, and it is where much of the craft lives. Easy negatives teach nothing, because the gap is already past the margin. Negatives that are too hard are often just positives you failed to label.
Run the forces yourself
Every word below is draggable and the pair list under the map is editable. It opens on our one triple: milk pulled toward tea, pushed away from dog. Step once and read milk's coordinates.
Where you'll meet this: reading a model card
When you pick an embedding model off a shelf, its page tells you the pair list, usually without calling it that. It is the training-data section, and it is the most load-bearing paragraph there.
| Model family | What counted as a positive pair | Good at | Bad at |
|---|---|---|---|
| General sentence encoder | Adjacent sentences, paraphrases, forum question-answer pairs | Judging whether two sentences say the same thing | Matching a short query to a long passage |
| Question-passage retriever | A real question and the passage answering it | Retrieval, where query and document differ wildly in length | Sentence similarity: it calls a question and its answer near-identical |
| Code retriever | A docstring and the function beneath it | Finding code from a description of what it should do | Prose of any kind, including the poetry test people enjoy |
| Face encoder | Two photographs of the same person | Deciding whether two faces are the same person | Everything else: it has no notion of a word |
Read the second column first. A model is not better or worse in the abstract. It is shaped for the pairs it was shown. The plain-language answer on embeddings covers what these vectors are for without the geometry, and Meaning as Direction puts finished spaces to work.
What you now know
- A contrastive objective is a list of pairs with a verdict on each, and everything the finished space knows comes from that list.
- The loss $L = \|\mathbf{a}-\mathbf{p}\|^2 - \|\mathbf{a}-\mathbf{n}\|^2$ starts at โ19.000 for milk held between tea and dog, and its gradient with respect to the anchor is $2(\mathbf{n}-\mathbf{p})$, so one step of 0.05 moves milk from $[3, 1]$ to $[3.500, 0.400]$ and the loss to โ31.200.
- The loss promises nothing about angle: the cosine to tea rose from 0.949 to 0.994 as a side effect of a rule that only ever mentioned distance.
- A margin $m$ tells a triple when it has the separation it asked for, at which point it contributes zero and stops pushing; without it the push never ends.
- Pairs are harvested rather than written, from the next sentence, a paraphrase, a click log or everything else in the batch, and choosing the negatives is called negative sampling.
- A space is an artifact of its pair list, so a model card's training-data paragraph tells you more about what the model is good at than any benchmark number does.
Where we're headed
We can now build a map by counting and reshape one on purpose, which means we have a map worth interrogating. The next chapter stops making maps and starts using them: what a nearest-neighbor search computes, what a centroid claims about a group of things, what happens when two different words come back with the same score, and why the obvious way of searching a million positions is the one nobody ships.