Measuring Wrongness

A guessing machine needs a report card

Part II ended with a machine that bets. Given "the cat sat on the ___", it laid two-thirds of its belief on mat, a quarter on couch, a sliver on keyboard. Part III now asks the one question that turns a bettor into a learner: how wrong was that bet? Not roughly wrong, not "pretty good" — wrong as a single number, because Chapter 5's downhill walk cannot descend a vibe. It needs a height.

That number has a name. We call it the loss, written $L$. Here is the plan for building it, in four moves. First we decide what to grade — it turns out we look at only one thing. Then we design the ruler: why the penalty is $-\ln p$ and not something gentler. Then we grade a whole exam, adding the penalties across many questions. And finally we hand the finished mountain to the next chapter, which learns to walk down it.

One honesty sentence up front: nothing in this chapter is new machinery. It is Chapter 4's logarithm meeting Chapter 6's probabilities — two old friends taking on one new job. You were twice promised that the number 2.3 "returns wearing a suit." This is the fitting.

Grade only the truth

Set the scene with Chapter 9's exact bars. The context is "the cat sat on the ___", and the model reported $p(\text{mat}) = 0.665$, $p(\text{couch}) = 0.245$, $p(\text{keyboard}) = 0.090$. Now the answer key arrives. In training, the answer key is just the actual text — the model reads real sentences and is graded on what genuinely came next — and the storybook says mat.

Here is the first design decision, and it is smaller than you would expect. The grade looks at exactly one of those bars: the probability the model gave the correct next token, $p = 0.665$. Everything else is ignored. We never look at what it wasted on couch or keyboard.

That should feel wrong at first — shouldn't we also punish the belief squandered on the two losers? No, and the reason is Chapter 6's conservation law. The model owns exactly one unit of belief, no more. Every drop it pours on couch is a drop it has already taken away from mat. Punishing the truth's low share and punishing the wrong bars' high shares would be charging the model twice for the same crime.

So the grading pipeline is short: run the model, read off one number — the belief it placed on what actually came next — and convert that belief into a penalty. The whole chapter now reduces to a single question: what should the conversion curve look like?

Designing the wrongness ruler

Before we pick a curve, let's write the wish list — three requirements any honest penalty-of-$p$ curve must meet. (1) Perfect confidence in the truth ($p = 1$) should cost nothing. (2) The penalty should grow as $p$ falls — and grow brutally as $p$ approaches 0, because a model that was nearly certain the truth was impossible has failed in the worst way there is. (3) Grades must add across questions, so a thousand-token exam produces one honest total.

Audition the obvious candidate first: penalty $= 1 - p$. It passes requirement 1 — at $p = 1$ it costs 0 — and it is beautifully plain. But it fails requirement 2. Sitting at $p = 0.1$ costs 0.9; sitting at $p = 0.001$ — a hundred times more dismissive of the truth — costs 0.999. Barely more. The gentle ruler tops out at 1 and cannot tell "wrong" from "catastrophically, confidently wrong." It has no room left to punish the real disasters.

Now the winner: the natural log of Chapter 4, flipped positive.

In words: the loss is minus the natural log of the probability the model gave the correct token, given the context.

Check it against the wish list. Requirement 1: $-\ln 1 = 0$ — perfect confidence, zero penalty ($\ln 1 = 0$ was a Chapter 4 anchor). Requirement 2: as $p$ slides toward 0, $\ln p$ dives toward minus infinity, so $-\ln p$ climbs a cliff with no ceiling. Watch the numbers march:

In words: half-belief in the truth costs 0.69, one-in-ten costs 2.3, one-in-a- hundred costs 4.6. Every division of belief by ten adds the same 2.3 to the penalty — that is Chapter 4's each-÷10-subtracts-2.3 pattern, seen in a mirror.

Requirement 3 is the deep one, and it is why no other simple curve survives. Grades must add — and adding is exactly what Chapter 4's rule $\ln(a \cdot b) = \ln a + \ln b$ does for a living. The probability of getting a whole sequence right multiplies: belief 0.5 on the first token times 0.5 on the second is 0.25 on the pair. The log converts that multiplication into addition — $-\ln(0.25) = 0.69 + 0.69 = 1.39$. Multiplied beliefs become added penalties. That single property is what makes $-\ln p$ the right ruler and not merely a steep one.

Now walk the finished ruler's landmarks, each read as a feeling. $p = 1 \to L = 0$: flawless. $p = 0.9 \to L \approx 0.11$: nearly sure, a tiny sting — this is "a loss of 0.1" territory. $p = 0.5 \to L = 0.69$: a coin flip's worth of doubt. $p = 0.37 \to L \approx 1$ (that is $e^{-1}$, one full unit of penalty). $p = 0.1 \to L = 2.3$: the number from Chapter 4, now in its suit — one-in-ten belief in the truth. $p = 0.01 \to L = 4.6$: the model all but ruled the truth out.

One sentence of vocabulary honesty: engineers often call this "log loss" or "negative log probability." Our log is always the natural log, $\ln$, base $e$ — same idea as Chapter 4's decibel aside, a different ruler. And one intuition worth naming out loud: $-\ln p$ measures surprise. The model is charged not for being wrong in the world's eyes, but for how surprised it was by the answer — mild surprise, mild fee; "that's impossible!", a ruinous one.

Grading the cat

Time to apply the ruler to Chapter 9's bars, fully by hand. The truth is mat, and $p(\text{mat}) = 0.665$. That looks like it demands $\ln 0.665$ — an impossible number to do in your head. But we can dodge it with a Chapter 4 trick, because $p(\text{mat})$ was built as a fraction: $7.39 / 11.11$. Meet the sibling of the product rule — dividing inside a log turns into subtracting outside it:

In words: the log of a divided by b is the log of a minus the log of b — the exact mirror of Chapter 4's $\ln(a \cdot b) = \ln a + \ln b$.

So the penalty splits into two logs we can almost read off. We know $\ln 7.39 = 2$ from Chapter 4 (that was the anchor $e^2 \approx 7.39$). And we grant ourselves exactly one calculator value for the whole chapter: $\ln 11.11 \approx 2.41$. Trust the calculator for this one number; everything else follows by hand.

In words: the penalty is the log of everyone's total minus the log of mat's own share — about 0.41.

Now the delight. Grade the same bet against each possible truth, using the same two logs. If the book had said couch ($p = 0.245 = 2.72/11.11$, and $\ln 2.72 = 1$): $L = 2.41 - 1 = 1.41$. If it had said keyboard ($p = 0.090 = 1/11.11$, and $\ln 1 = 0$): $L = 2.41 - 0 = 2.41$. Line them up:

Look at the spacing: the losses sit exactly 1 apart, just as the logits 2.0 / 1.0 / 0.0 did back in Chapter 9. This is that chapter's "each +1 of logit multiplies belief by $e$" seen in the mirror — each $+1$ of logit shaves exactly 1 off the loss, because $\ln e = 1$. Logit gaps are loss gaps. You have just connected raw scores, probabilities, and penalties into a single arithmetic chain you can verify on paper.

The whole exam

One question is not an exam. Training grades the model on run after run of text, every position a fresh question (a seed Chapter 15 will grow into billions). The total grade is the average of the per-token losses — we add them up, as requirement 3 demanded, and then divide by the number of questions so that a long exam and a short one can be read on the same scale. Work a three-question exam using the ruler's own landmark numbers: on question 1 the model gave the truth $p = 0.5$, so $L = 0.69$; on question 2, $p = 0.9$, so $L = 0.11$; on question 3, a bad miss, $p = 0.1$, so $L = 2.3$.

In words: add the three penalties and divide by three — the exam score is about 1.03.

Read the exam like a teacher. The single bad question (2.3) contributes more to the total than the two decent ones combined (0.80). That asymmetry is the design working as intended — the cliff end of the ruler is where the training pressure piles up. A model improves its exam score fastest by rescuing its worst surprises, not by polishing its best answers.

This finished recipe — grade only the truth's probability, charge $-\ln$, average over all the data — has a name: cross-entropy loss. The name has a history in information theory; on this site the recipe is the definition, and you have now built every part of it by hand. Written in general form, over our $N$ questions:

In words: the loss of the parameters $\theta$ is the average, over all $N$ questions, of minus the log of the belief the model gave each right answer.

The new wrinkle is the $\theta$ in parentheses — theta, the collective name for every one of the model's billions of dials. Same exam, different dial settings, different score. The loss is a function of $\theta$ (Chapter 4's machine sense: feed in a full setting of every weight, get out one number). Change the dials and the score changes — which is the whole hinge on which the rest of the site turns.

See it move

Why the LLM cares

Here is the reveal, plainly: this number is the single most-watched quantity in modern AI. Every training run of every LLM is an attempt to push this chapter's $L(\theta)$ down; the multi-million-dollar dashboards of Chapter 15 display little else. When researchers say a model "got better," the literal sentence underneath is "the average $-\ln p(\text{correct token})$ went down."

And now the two halves of this site's story click together. Part I built a walker that can descend any landscape it can read a height from (Chapter 5's preview, $\theta_{\text{new}} = \theta - \eta \cdot \nabla L$ — step against the slope, size $\eta$). This chapter built the height. Loss is the mountain; training is the walk. Chapter 13 takes the first real steps and discovers that the terrain is nothing like Chapter 5's friendly parabola.

Let me plant one seed for later with hand arithmetic. A freshly initialized model knows nothing, so it spreads its belief evenly — $p = 1/50{,}000$ on every token (Chapter 7's $V \approx 50{,}000$). Its loss on every single token is therefore $-\ln(1/50{,}000) = \ln 50{,}000 \approx 10.8$. Remember that number the way you remembered 2.3: it is where every loss curve in Chapter 15 begins.

Close warm. You can now do something most people who use AI every day cannot — read a loss value as a sentence. "The run is at 2.3" means "on average, the model is giving the actual next token about a 10% share of its belief." One number, fully decoded, in a language you built yourself.