Part III · Ch. 13 — The Surprise Surface

Part III · Chapter 13 of 16

The Surprise Surface

From one point to four probabilities, and the bill in bits


One point, and a vocabulary

A language model's last step turns one position into a probability for every word it knows, and this chapter does that arithmetic by hand on four words. The position is the one chapter 11 left us with, $\mathbf{h} = (7.000,\ 1.000)$, and the vocabulary is four words long: tea, coffee, milk, dog.

The model stores one direction per word. That table of directions is the unembedding, and it is the embedding table read in the other direction: at the start a word is looked up to get a position, and at the end a position is scored against every word. The four rows are $\mathbf{u}_{\text{tea}} = (0.400,\ 0.000)$, $\mathbf{u}_{\text{coffee}} = (0.400,\ 0.200)$, $\mathbf{u}_{\text{milk}} = (0.200,\ 0.200)$ and $\mathbf{u}_{\text{dog}} = (0.000,\ 0.400)$, with lengths 0.400, 0.447, 0.283 and 0.400 and headings 0.000, 26.565, 45.000 and 90.000 degrees.

Hold on to the lengths. They do more work than anybody expects.

A logit is a shadow, scaled by the row's length

The score for a word is the dot product of its row with the position:

$$z_w = \mathbf{u}_w \cdot \mathbf{h}$$

In words: the raw score for word $w$ is the row for $w$ multiplied into the position, component by component, and added up. That raw score has a name, and the name is about to do a lot of work.

That number is the logit. For coffee it is one line of arithmetic you can check without a calculator:

Worked example: coffee's logit

Coffee's row is $(0.400,\ 0.200)$ and the position is $(7.000,\ 1.000)$. Multiply across: $0.4 \times 7.0 = 2.800$. Multiply up: $0.2 \times 1.0 = 0.200$. Add them: $2.800 + 0.200 = 3.000$.

Do the same for the other three and you get tea 2.800, milk 1.600, dog 0.400.

In words: a logit is chapter 1's dot product, wearing a new job title.

Chapter 1 also drew the dot product as a shadow, and that picture survives with one correction that matters. A shadow, properly, is the projection onto a direction of length one. Coffee's row is 0.447 long, so its logit is the shadow scaled by 0.447: the shadow measures 6.708, and $6.708 \times 0.447 = 3.000$. Every logit is a shadow times the length of the row it fell on.

That scale factor decides the winner here. Tea's row agrees with the position better than coffee's does, at a cosine of 0.990 against 0.949, and tea still loses, because coffee's row is 0.447 long and tea's is 0.400. Heading alone does not settle it.

A grid with a mint arrow from the origin to h at 7 across and 1 up. Four short arrows leave the origin, one per vocabulary word, with faint dashed rays carrying their directions across the plane. A thick amber band runs from the origin along coffee's direction and is labeled 'shadow 6.708'; a dotted perpendicular drops from h onto the end of that band. A panel on the right lists, per word, the cosine with h, the row length and the logit: tea 0.990, 0.400, 2.800; coffee 0.949, 0.447, 3.000; milk 0.800, 0.283, 1.600; dog 0.141, 0.400, 0.400. Below it reads 'coffee: 6.708 x 0.447 = 3.000' and 'tea is better aligned and still loses'.
Two factors, one logit. Notice that tea has the better cosine and the shorter row, and the shorter row is what costs it the top spot.

Four shadows become four shares

Four logits are not four probabilities. They are not between zero and one and they do not add to one. Softmax fixes both in one move: exponentiate each logit, then divide by the total.

$$p_w = \frac{e^{z_w}}{\sum_{w'} e^{z_{w'}}}$$

In words: raise $e$ to the power of each logit, add the four results together, and give each word its share of that total. Exponentiating makes every number positive, and dividing by the total makes them add to one.

The LLM book's Chance and Choice covers what a model does with these numbers once it has them. This chapter is about where they come from.

The whole output step at $\mathbf{h} = (7.000,\ 1.000)$. Each column is computed from the one to its left.
WordRowLogitProbabilitySurprise (bits)Surprise (nats)
tea(0.400, 0.000)2.8000.3831.3860.961
coffee(0.400, 0.200)3.0000.4671.0970.761
milk(0.200, 0.200)1.6000.1153.1172.161
dog(0.000, 0.400)0.4000.0354.8483.361

The four probabilities add to 1.000, coffee is on top at 0.467, and the whole distribution has an entropy of 1.571 bits, which is a perplexity of 2.971 words. The last two columns are the next section.

The bill, in bits

Say the text went on with the word coffee. The model gave that word 0.467 of its belief, and the cost of being that confident and right is:

$$s_w = -\log_2 p_w$$

In words: take the probability the model gave the word that came next, take the base-two logarithm, and flip the sign. A word the model was sure of costs almost nothing; a word it ruled out costs a lot.

For coffee at 0.467 that is 1.097 bits. Had the next word been dog, at 0.035, the bill would have been 4.848 bits. Surprise is the same quantity the Q&A page Entropy and Information builds from scratch, and this book counts it in the unit that page uses.

The books in this family disagree about units on purpose. This book counts bits and so does Entropy and Information, while the LLM book's Measuring Wrongness counts nats, because it takes the natural logarithm instead of the base-two one. One nat is 1.443 bits and one bit is 0.693 nats, so coffee's 1.097 bits is the same reading as 0.761 nats, and neither book is wrong. When you meet a loss number in the wild, find out which logarithm made it before you compare it with anything.

One direction in logit space does nothing at all

Add 5.000 to all four logits and the probabilities do not move. Subtract 10.000 from all four and they still do not move: tea 0.383, coffee 0.467, milk 0.115, dog 0.035, every time. Softmax divides by the total, and adding a constant to every logit multiplies the top and the bottom of that fraction by the same thing.

So one whole direction in the space of logits, the one where all four go up together, carries no information. That space has a second surprise in it. Our four logits come from two coordinates through a table of four rows, and the rank of that table is 2, so the reachable logits form a plane inside a four-dimensional space. Most of that space cannot be produced at any position: a four-word vocabulary read off a two-dimensional stream has two degrees of freedom.

Temperature is the point sliding toward the origin

Temperature is usually taught as a knob on the logits. With zero output bias, which is what our toy has, it is a move of the position and nothing else:

$$\frac{z_w}{T} = \mathbf{u}_w \cdot \left(\frac{\mathbf{h}}{T}\right)$$

In words: dividing every logit by the temperature $T$ gives the same four numbers as leaving the logits alone and dividing the position by $T$ instead. Turning the temperature up and sliding the point toward the origin are one operation, as long as the output layer adds no bias of its own.

Temperature, read as a position. The rows are five different places on the same line through the origin.
$T$$\mathbf{h}/T$p(tea)p(coffee)p(milk)p(dog)Entropy (bits)
0.500(14.000, 2.000)0.3860.5760.0350.0031.184
1.000(7.000, 1.000)0.3830.4670.1150.0351.571
2.000(3.500, 0.500)0.3380.3740.1860.1021.846
4.000(1.750, 0.250)0.2990.3150.2220.1641.956
10.000(0.700, 0.100)0.2710.2760.2400.2131.992

Follow the row down and the point walks in toward the origin, and the four probabilities close on each other. At the origin every logit is 0.000, every word sits at 0.250, and every one of them costs 2.000 bits. Chapter 1 said a zero-length arrow has no heading. This is that sentence with consequences: a model whose position has been scaled to nothing has no opinion left.

The plane tiles into regions

Every point on the plane has a top word, so the plane divides into areas that each belong to one word. Picture a bearing-and-range plot cut into sectors: where you are decides which sector you are in, and that is all a decision region is.

The boundary between two words is where their logits are equal, which is where $(\mathbf{u}_a - \mathbf{u}_b) \cdot \mathbf{h} = 0$. That is a straight line through the origin, perpendicular to the difference of the two rows, and it passes through the origin because there is no bias to shift it. The six pairs give boundaries at 0.000, 45.000, 63.435 and 90.000 degrees.

A square of the plane from minus 8 to 8 in both coordinates, shaded into three regions. A violet region below the horizontal axis is labeled 'tea 0.377'. An amber wedge between the horizontal axis and a ray at 63.435 degrees is labeled 'coffee 0.187' and contains a mint dot labeled h. A blue region covering the rest is labeled 'dog 0.436'. Three solid white rays leave the origin at 0.000, 63.435 and 45.000 degrees; two dotted rays mark boundaries that never separate two winners. A panel on the right is headed 'milk has no region, anywhere' and explains that milk's row is the midpoint of tea's and dog's, that the largest value of milk's logit minus the better of tea's and dog's is 0.000, and that this holds while the output bias is zero.
Three regions for four words. Notice that the boundaries all run through the origin, which is what a model with no output bias buys you, and that one of the four words has no region at all.

Measured over a grid of 321 by 321 points across that square, dog owns 0.436 of the area, tea 0.377 and coffee 0.187. Those three add to 1.000. Milk gets 0.000.

A word that can never win

Milk is not unlucky. In a model with no output bias, a word whose row is a convex combination of other rows can never be the top prediction at any position, and milk's row is the midpoint of tea's and dog's: $(0.200,\ 0.200)$ is the average of $(0.400,\ 0.000)$ and $(0.000,\ 0.400)$.

The proof is one line. Milk's logit is the average of tea's logit and dog's logit at every position, and an average is never strictly bigger than both of the things it averages. Over the whole grid the largest value of milk's logit minus the better of tea's and dog's is 0.000, and it is never positive. Milk can tie. It cannot win.

Both conditions are load-bearing. Drop the convex combination and the argument evaporates. Drop the zero bias and it evaporates too, because a bias is added after the dot product and can lift a dominated word above its neighbors. Models that carry an output bias are not covered by this argument at all.

Reading the stream at every depth

Chapter 11 recorded thirteen positions, one per write, ending at $(7.000,\ 1.000)$. Nothing stops us running each of the earlier twelve through the same four rows, and the technique has a name: the logit lens, introduced by nostalgebraist in 2020. The caveat belongs in the same breath as the name, because the lens reads an intermediate position through the final rows, which is an interpretation and not a measurement of what any layer believed.

The logit lens on chapter 11's recorded trace, seven of the thirteen states.
DepthPositionp(coffee)Entropy (bits)
0(1.000, 1.000)0.2891.994
3(2.500, 1.600)0.3521.953
5(4.400, 1.300)0.4151.808
7(5.700, 1.000)0.4421.681
8(5.100, 1.400)0.4371.751
10(6.200, 1.100)0.4551.642
12(7.000, 1.000)0.4671.571

Coffee leads from the embedding onward, so what the six layers add is confidence rather than a change of mind. That is a property of this toy's embedding and not a finding about models. The interesting row is depth 8. That is the erasing write, MLP 4, and the probability goes down, from 0.442 to 0.437, while the entropy goes back up from 1.681 bits to 1.751 bits. A later block can undo an earlier one, and on the surprise surface that shows up as a step backward.

A line chart with depth from 0 to 12 across. A solid mint line rising from 0.289 to 0.467 is the probability of coffee, read on the left axis. A dashed amber line falling from 1.994 to 1.571 is the entropy in bits, read on the right axis. At depth 8 the mint line dips and the amber line rises; that point is ringed in red and annotated 'MLP 4 erases, and p(coffee) falls to 0.437'.
Confidence can go backward. Notice that the dip and the rise happen at the same step, because they are two readings of one write.
A heatmap of the plane from minus 8 to 8 in both coordinates, showing the surprise of the word coffee in bits. The color runs from mint at low surprise in the upper right through amber to red at high surprise in the upper left. Contour lines are drawn and labeled at 1 bit, 2 bits and 4 bits; they curve rather than forming circles. The origin is marked and annotated 'the origin, 2.000 bits', and h at 7 comma 1 is marked and annotated 'h, 1.097 bits'.
The bill, drawn as a landscape. Notice the contours are not circles: surprise depends on the heading and the length of the position together, so its level curves bend.

Steering, and a flip predicted before it is measured

Chapter 12 built a direction by hand, the unit vector $(0.707,\ -0.707)$, and showed that adding $\alpha$ times it to the position moves the model's behavior. That same knob, with the output attached, reads like this.

Adding $\alpha$ times the steering direction to $\mathbf{h}$, and reading the result.
$\alpha$Positionp(tea)p(coffee)p(dog)Top
−8.000(1.343, 6.657)0.0620.2360.522dog
−6.132(2.664, 5.336)0.1170.3410.342dog
−2.000(5.586, 2.414)0.2910.4720.082coffee
0.000(7.000, 1.000)0.3830.4670.035coffee
1.414(8.000, 0.000)0.4460.4460.018tie
2.000(8.414, −0.414)0.4720.4340.014tea

The flip point was computed before the table was. Coffee beats tea while the second coordinate of the position is positive and not otherwise, because coffee's row and tea's row differ only in that coordinate. The position starts at a second coordinate of 1.000 and the steering direction's second coordinate is −0.707, so the crossing sits at $1 \div 0.707 = 1.414$. The table agrees: at 1.414 the two words are level at 0.446 apiece.

Push the other way and dog takes over, but not until much further out. Dog beats coffee where $-0.4x + 0.2y > 0$, and substituting the steered coordinates gives $-2.6 - 0.424\alpha > 0$, so $\alpha$ has to go below −6.128.

Drag the point yourself

The interactive below is the plane from the tiling figure with the numbers attached. Drag the dot and the four logits, the four probabilities and the surprise of your chosen target word all move with it. The temperature slider does not repaint the shading, because temperature does not change which word wins where; it slides the point toward the origin instead.

A two-token exam

One number is not a loss. A loss is an average over the tokens the model was asked about, so the smallest version that counts is two positions with two right answers.

Worked example: cross-entropy on two tokens

First token. The position is $(7.000,\ 1.000)$, the word that came next was coffee, the model gave it 0.467, and the bill is 1.097 bits.

Second token. The mirror sequence ends at $(1.000,\ 7.000)$, where the four probabilities are tea 0.052, coffee 0.209, milk 0.171 and dog 0.568. The word that came next was dog, so the bill is 0.815 bits.

The average. $(1.097 + 0.815) \div 2 = 0.956$ bits, which is 0.663 nats. Raise two to that power and you get a perplexity of 1.940 words.

In words: cross-entropy is the mean surprise on the words that came next, and perplexity is that number turned back into a count. A perplexity of 1.940 on a four-word vocabulary says the model has cut the field roughly in half.

The model's top prediction was correct both times, and the loss is not zero. That gap is the whole reason training keeps going after a model starts getting answers right.

Where you'll meet this

  • A training dashboard. The loss curve on a training run is this arithmetic, averaged over millions of tokens instead of two. When it falls, the model did not get more answers right; the average surprise on the observed token went down, and the unit it went down by depends on the logarithm. This book and the Q&A page Entropy and Information count in bits, while the LLM book's Measuring Wrongness counts in nats, with one nat equal to 1.443 bits and one bit equal to 0.693 nats, so coffee's 1.097 bits and its 0.761 nats are one reading of one model.
  • Temperature in an API call. When you set temperature on a request, you are sliding the model's position toward the origin, in a model whose output layer adds no bias. The knob is a scale factor on a point, and at a high enough setting there is no point left to read.
  • Interpretability write-ups. A plot of "what the model believed at layer 8" is a logit lens plot, and it read that layer's position through the final layer's rows. Ours goes backward at one step, and that step is a write that erased.
  • Vocabulary design. A token whose row lands inside the hull of other rows cannot win in a model with no output bias. Real vocabularies have tens of thousands of rows in hundreds of dimensions, and how often a row is trapped inside the others is not something I have measured.

What you now know

  • A logit is a dot product between a word's unembedding row and the position, which is chapter 1's shadow scaled by the length of that row, and tea loses to coffee at a better cosine because its row is shorter.
  • Softmax turns four logits into four shares of one whole, and the surprise of a word is minus the base-two logarithm of its share: 1.097 bits for coffee at the position chapter 11 handed us.
  • Bits and nats are the same measurement in different units, with one nat equal to 1.443 bits, so a loss number means nothing until you know which logarithm produced it.
  • Adding a constant to every logit changes nothing, and our four logits are produced by a table of rank 2, so most of the logit space is unreachable.
  • Temperature, in a model with no output bias, is the position sliding toward the origin, and at the origin all four words sit at 0.250 and cost 2.000 bits apiece.
  • The plane tiles into decision regions with straight boundaries through the origin, and in a model with no output bias a word whose row is a convex combination of other rows has no region at all: milk wins nowhere, at any temperature.
  • Running the logit lens over chapter 11's trace shows confidence climbing with depth and going backward once, at the erasing write, which is an interpretation of intermediate positions rather than a record of what a layer believed.

Where we're headed

Part III followed one position from its embedding to a probability. Part IV asks what a single position is able to carry in the first place. A retrieval system compresses a whole document into one point and then compares points, and there are score patterns no such system can produce, no matter how it is trained, because of the dimension it has to fit them into. Chapter 14 works that bound out on a three-by-three table you can check with a pencil, then walks it back to what the bound does and does not forbid, because the bound is about scores and retrieval only needs an order.