Part II ยท Ch. 7 โ€” Learning the Map

Part II ยท Chapter 7 of 16

Learning the Map

Contrastive training: which pairs pull and which push

Youโ€™ll need:the six-word map built from countsdistance and cosine are different rulersthe gradient step, linked rather than re-taught

Counting has a ceiling

Counting cannot bring together two things that belong together but never appear near each other. To fix that you stop taking the positions you are given and start moving them on purpose.

Two chapters ago we built a map out of tally marks, and it worked. Tea and coffee landed together because they keep the same company, and the animals landed apart because they keep different company: a real map for the price of a few evenings with a notebook. But consider "pay stub" and "remittance advice." They mean the same thing to an accounts department, and one lives in American documents while the other lives in British ones. A corpus holding both may never put them near anything they share, so they get different coordinates, and no window width fixes it.

This chapter is the workbench, the clamp, and the one arithmetic step that shapes a map into the shape you asked for.

Say who should be close

The mechanism is unglamorous. You hand the training procedure a list of pairs, and for each pair you say one of two things: these two belong close together, or these two belong apart.

A pair you want close is a positive pair; a pair you want apart is a negative pair. That is the whole vocabulary, plus one name for the word being moved: the anchor.

We will use one triple from the map we already have. The anchor is milk at $[3, 1]$. Its positive is tea at $[6, 0]$, because both are drinks and I want them nearer. Its negative is dog at $[1, 6]$, because I want milk further from the animals than it sits.

A loss is a rule about pairs

To move anything we need a number that says how badly the arrangement disagrees with our wishes and gets smaller as it improves. That number is the loss, and we build it one wish at a time.

First wish: milk should be near tea, and the squared distance between them measures how far that is from true. Second wish: milk should be far from dog, where the same quantity is something we want large, so we subtract it. With $\mathbf{a}$ the anchor, $\mathbf{p}$ the positive and $\mathbf{n}$ the negative:

$$L = \|\mathbf{a}-\mathbf{p}\|^2 - \|\mathbf{a}-\mathbf{n}\|^2$$

In words: the loss is the squared distance from the anchor to the thing it should be near, minus the squared distance from the anchor to the thing it should be far from. Making the first term small and the second large both push this number down, which is what we wanted from a single score.

Worked example: the loss right now

Anchor to positive. Milk $[3, 1]$ to tea $[6, 0]$: across, $3 - 6 = -3$; up, $1 - 0 = 1$. Square and add: $9 + 1 = 10$.

Anchor to negative. Milk $[3, 1]$ to dog $[1, 6]$: across, $3 - 1 = 2$; up, $1 - 6 = -5$. Square and add: $4 + 25 = 29$.

Subtract. $10 - 29 = -19$.

In words: the loss starts at โˆ’19.000. A negative loss is fine here. This score has no floor of zero, and all we ask of it is that it goes down.

The gradient is a displacement

Now differentiate. In Rolling Downhill we met the gradient as the direction of steepest increase, and the rule that you move against it to go down. Here the gradient turns out to be something you can draw.

Differentiate $L$ with respect to the anchor's position, term by term. The derivative of $\|\mathbf{a}-\mathbf{p}\|^2$ is $2(\mathbf{a}-\mathbf{p})$, an arrow from the positive toward the anchor. The derivative of $-\|\mathbf{a}-\mathbf{n}\|^2$ is $-2(\mathbf{a}-\mathbf{n})$, an arrow from the anchor toward the negative. Add them and the anchor cancels:

$$\nabla_{\mathbf{a}} L = 2(\mathbf{n} - \mathbf{p})$$

In words: the gradient of the loss with respect to the anchor is twice the vector from the positive to the negative. The anchor's own position vanished in the addition, so where the anchor sits does not change which way it is told to go.

Worked example: one step, in full

The gradient. Dog minus tea: across, $1 - 6 = -5$; up, $6 - 0 = 6$. Double it: $[-10, 12]$.

The update rule. Move against the gradient, scaled by the learning rate $\eta$, eta, the number that sets how big a step is:

$$\mathbf{a} \leftarrow \mathbf{a} - \eta \, \nabla_{\mathbf{a}} L$$

We take $\eta = 0.05$. Smaller and the move is too small to see on a six-by-six grid; larger and milk overshoots into empty space below the map, which you can make happen in the interactive below.

The arithmetic. Across: $3 - 0.05 \times (-10) = 3 + 0.5 = 3.500$. Up: $1 - 0.05 \times 12 = 1 - 0.6 = 0.400$. Milk moves from $[3, 1]$ to $[3.500, 0.400]$.

The new loss. To tea: $(3.5-6)^2 + (0.4-0)^2 = 6.25 + 0.16 = 6.41$. To dog: $(3.5-1)^2 + (0.4-6)^2 = 6.25 + 31.36 = 37.61$. So $L = 6.41 - 37.61 = -31.200$.

In words: one step of 0.05 took the loss from โˆ’19.000 to โˆ’31.200. Milk got closer to tea and further from dog in one move, because both wishes were encoded in one number and one number has one gradient.

Left: the six-word map with milk, tea and dog picked out. A mint arrow leaves milk pointing toward tea, labeled 'pull toward tea'. An amber arrow leaves milk pointing down and to the right, directly away from dog, labeled 'push away from dog'. A small dashed violet box around milk is labeled 'shown enlarged'. Right: that box enlarged, showing the two displacements added tip to tail โ€” a mint arrow labeled 'pull: plus 0.3, minus 0.1', then an amber arrow labeled 'push: plus 0.2, minus 0.5' โ€” with a dashed violet arrow labeled 'one move' running straight from milk to the endpoint, labeled 3.500 comma 0.400.
Two forces, one move. The pull and the push are separate arrows that are added before anything happens, so the anchor takes one step rather than two.

The enlargement is worth a second look. The pull contributes $0.05 \times 2([6,0]-[3,1]) = [0.3, -0.1]$ and the push contributes $0.05 \times 2([3,1]-[1,6]) = [0.2, -0.5]$. Laid tip to tail they reach $[0.5, -0.6]$, which is the move. The push is doing most of the work, and it is doing it because dog lies in a direction we want to leave rather than because dog is close.

One step is a nudge; training is a settling

Look at what else changed while milk was moving half a unit.

Left: the six-word map. Milk appears twice โ€” a hollow gray dot labeled 'milk, before' at 3 comma 1, and a filled violet dot labeled 'milk, after' at 3.5 comma 0.4 โ€” with a short violet arrow between them. A mint line runs from the new milk to tea, labeled 'shorter now'; an amber line runs from the new milk to dog, labeled 'longer now'; faint dashed lines show where both used to run. Right: a before-and-after list reading the loss L minus 19.000 to minus 31.200, distance to tea 3.162 to 2.532, distance to dog 5.385 to 6.133, cosine to tea 0.949 to 0.994, cosine to dog 0.468 to 0.275.
One move, five measurements. The distance and the angle both changed, and in the same direction. Nothing here asked for that, and nothing guarantees it in general.

Five numbers moved and every one moved the way we wanted, which is worth being suspicious of, because the loss only ever mentioned distance. Chapter 2 was a whole chapter on distance and cosine being different rulers that routinely disagree. Here they happened to agree. The cosine to tea climbed from 0.949 to 0.994 as a by-product, not because anything in $L$ was watching it. A contrastive loss built on squared distance makes a promise about distances and none about angles.

And one step is a nudge. Real training runs thousands of steps over thousands of triples, where every point is an anchor in some and a negative in others, so everything moves at once.

Four small square panels in a row labeled start, step 5, step 20 and step 200. Twelve dots in three colors begin scattered and end in three tight separated groups, one per color. Beneath, a red curve labeled total loss falls from about 7,600 at step 0 to zero by about step 45 and stays flat, with a note reading 'every pair satisfied: the pushing stops'.
The clusters are the outcome, not the input. Nobody placed a single one of these twelve points. The only thing supplied was a list saying which pairs belong together.

Those twelve points started scattered at random. The grouping visible by step 200 was never drawn. It was specified, as a list of pairs, and the arithmetic did the drawing. When someone says an embedding model "learned that these documents are related," this is the machine they mean, and what it learned was somebody's pair list.

The margin, and why the push stops

Our loss has a flaw you may already have spotted: it has no bottom. Every extra unit of separation between milk and dog lowers $L$ forever, so if we keep stepping, milk flees rather than settling anywhere sensible.

The fix, introduced for face recognition by Schroff and colleagues in FaceNet (2015), is to state how much separation is enough and stop paying for more. That number is the margin, written $m$:

$$L = \max\!\left(0,\ \|\mathbf{a}-\mathbf{p}\|^2 - \|\mathbf{a}-\mathbf{n}\|^2 + m\right)$$

In words: take the same difference as before, add the margin, and if the result is negative, call the loss zero. A triple that already has the separation you asked for contributes nothing to the loss and therefore nothing to the gradient, so it stops pushing.

Take $m = 25$, measured in the same squared units $L$ is. Before our step the gap, meaning how much further dog sits than tea, was $29 - 10 = 19$, short of 25, so the triple was unhappy by 6 and pushed. After the step the gap is $37.61 - 6.41 = 31.2$, past the margin, so this triple's loss is zero and it sits out the rest of training.

A graph with the gap โ€” how much further the negative sits โ€” across the bottom and loss contributed up the side. A gray dashed line labeled 'bare difference: falls forever' descends steadily below zero. A red line labeled 'with a margin: stops at zero' descends until it reaches zero at a gap of 25, marked by a violet dashed vertical line labeled 'the margin, m equals 25', then runs flat. An amber dot labeled 'before the step' sits on the red line at a gap of 19; a mint dot labeled 'after the step' sits on the flat part at a gap of 31.2.
A satisfied triple contributes nothing. Past the margin the red line goes flat, so extra separation earns no reward, which is the only reason anything settles.

That is why the loss curve in the settling figure hits zero and stays there. The points did not run out of room. Every triple got the gap it asked for and switched itself off.

Where the pairs come from

Nobody writes millions of pairs by hand. They are harvested, and the harvesting rule is the most consequential decision in building an embedding model.

  • The next sentence. A sentence and the one after it, called a positive pair. Cheap, unlimited, and it teaches that adjacent text is similar.
  • The same sentence, twice. Run one sentence through the model twice with different random dropout and call the results a positive pair. That is the trick behind SimCSE (Gao, Yao and Chen, 2021), where the pair list comes from nothing but the model's own noise.
  • The click log. A query and the result somebody clicked. This is how retrievers are built, and it shapes the space by what users found satisfying rather than by what is true.
  • Everything else in the batch. Rather than choosing negatives, call every other item in the mini-batch one. This is the in-batch negative, formalized as InfoNCE by van den Oord and colleagues in Representation Learning with Contrastive Predictive Coding (2018), and it is why batch size appears in embedding papers as a quality knob rather than a memory detail.

Choosing negatives deliberately is called negative sampling, and it is where much of the craft lives. Easy negatives teach nothing, because the gap is already past the margin. Negatives that are too hard are often just positives you failed to label.

Run the forces yourself

Every word below is draggable and the pair list under the map is editable. It opens on our one triple: milk pulled toward tea, pushed away from dog. Step once and read milk's coordinates.

Where you'll meet this: reading a model card

When you pick an embedding model off a shelf, its page tells you the pair list, usually without calling it that. It is the training-data section, and it is the most load-bearing paragraph there.

Four embedding model families, and the pair list each one was shaped by.
Model familyWhat counted as a positive pairGood atBad at
General sentence encoderAdjacent sentences, paraphrases, forum question-answer pairsJudging whether two sentences say the same thingMatching a short query to a long passage
Question-passage retrieverA real question and the passage answering itRetrieval, where query and document differ wildly in lengthSentence similarity: it calls a question and its answer near-identical
Code retrieverA docstring and the function beneath itFinding code from a description of what it should doProse of any kind, including the poetry test people enjoy
Face encoderTwo photographs of the same personDeciding whether two faces are the same personEverything else: it has no notion of a word

Read the second column first. A model is not better or worse in the abstract. It is shaped for the pairs it was shown. The plain-language answer on embeddings covers what these vectors are for without the geometry, and Meaning as Direction puts finished spaces to work.

What you now know

  • A contrastive objective is a list of pairs with a verdict on each, and everything the finished space knows comes from that list.
  • The loss $L = \|\mathbf{a}-\mathbf{p}\|^2 - \|\mathbf{a}-\mathbf{n}\|^2$ starts at โˆ’19.000 for milk held between tea and dog, and its gradient with respect to the anchor is $2(\mathbf{n}-\mathbf{p})$, so one step of 0.05 moves milk from $[3, 1]$ to $[3.500, 0.400]$ and the loss to โˆ’31.200.
  • The loss promises nothing about angle: the cosine to tea rose from 0.949 to 0.994 as a side effect of a rule that only ever mentioned distance.
  • A margin $m$ tells a triple when it has the separation it asked for, at which point it contributes zero and stops pushing; without it the push never ends.
  • Pairs are harvested rather than written, from the next sentence, a paraphrase, a click log or everything else in the batch, and choosing the negatives is called negative sampling.
  • A space is an artifact of its pair list, so a model card's training-data paragraph tells you more about what the model is good at than any benchmark number does.

Where we're headed

We can now build a map by counting and reshape one on purpose, which means we have a map worth interrogating. The next chapter stops making maps and starts using them: what a nearest-neighbor search computes, what a centroid claims about a group of things, what happens when two different words come back with the same score, and why the obvious way of searching a million positions is the one nobody ships.