A guessing machine needs a report card
The model assigns shares to mat, couch, and keyboard. How good was that prediction? The downhill rule needs one height. A loss, written L, supplies it.
Grade the observed next token
Reuse the shares 0.665,0.245,0.090 from Chapter 9. The target is mat because that is what came next in the training text.
For this loss, read the target's probability, 0.665. The other shares affect it through normalization: allocating more belief elsewhere leaves less for the target.
Designing the wrongness ruler
We want zero penalty at p=1, increasing penalty as p falls, and additive scores for a sequence. The negative log probability supplies those properties.
In words: the loss is minus the natural log of the probability the model gave the correct token, given the context.
| Rule | p=0.1 | p=0.001 | Behavior |
|---|---|---|---|
| 1−p | 0.90 | 0.999 | Bounded by one; limited penalty for extreme misses. |
| −ln p | 2.3 | 6.9 | Unbounded as target probability approaches zero. |
The log also connects sequence probabilities to total loss. Conditional probabilities multiply along a sequence; their negative logs add. Two successive probabilities of 0.5 give 0.25 together and total loss about 1.39.
In words: half-belief in the truth costs 0.69, one-in-ten costs 2.3, one-in-a- hundred costs 4.6. Every division of belief by ten adds the same 2.3 to the penalty, which is Chapter 4's each-÷10-subtracts-2.3 pattern with the sign flipped.
Additional checkpoints: p=0.9 costs about 0.11; p=0.37 costs about 1. The natural-log unit is a nat. We can read loss as surprise.
Grading the cat
Use the log quotient rule to avoid calculating a new logarithm for each candidate:
$$\ln\!\left(\frac{a}{b}\right) = \ln a - \ln b$$In words: the log of a divided by b is the log of a minus the log of b. It is Chapter 4's $\ln(a \cdot b) = \ln a + \ln b$ with division in place of multiplication.
Grade mat by hand
We know $e^2\approx7.39$, so $\ln7.39\approx2$. Take one calculator value, $\ln11.11\approx2.41$.
$$L = -\ln 0.665 = \ln 11.11 - \ln 7.39 \approx 2.41 - 2 = 0.41$$In words: the penalty is the log of everyone's total minus the log of mat's own share, about 0.41.
For couch, subtract 1 instead, giving 1.41. For keyboard, subtract zero, giving 2.41.
The three possible grades
| if the truth were… | $p$ | loss $L$ |
|---|---|---|
| mat | 0.665 | 0.41 |
| couch | 0.245 | 1.41 |
| keyboard | 0.090 | 2.41 |
These three grades differ by the logit gaps because they score the same fixed distribution against different targets. Raising one logit also changes the common normalization total, so it does not generally reduce its own loss by exactly that amount.
The whole exam
A mean lets us compare exams of different lengths. Each token contributes its own loss.
Average three prediction rounds
Correct-token probabilities 0.5,0.9,0.1 produce losses about 0.69,0.11,2.3.
$$L = \frac{0.69 + 0.11 + 2.3}{3} = \frac{3.10}{3} \approx 1.03$$In words: add the three penalties and divide by three, giving an exam score of about 1.03.
The largest surprise contributes more than the other two combined. That does not alone determine the parameter gradient; the backward computation does.
Cross-entropy gives this score. Our cross-entropy loss averages it over N questions. The parameters $\theta$, “theta,” determine the predicted shares $p_i$:
$$L(\theta) = \frac{1}{N} \sum_{i=1}^{N} \big( -\ln p_i \big)$$In words: the loss of the parameters $\theta$ is the average, over all $N$ questions, of minus the log of the belief the model gave each right answer.
| Measure | What it asks |
|---|---|
| Cross-entropy | How much probability reached the observed targets? |
| Accuracy | How often did the top choice match? |
| Mean target probability | What is the arithmetic average of the shares? |
These measures differ. Exponentiating minus mean log loss gives the geometric mean target probability, not the arithmetic mean.
See it move
Why the LLM cares
Loss gives each parameter setting a height. Gradient descent uses its slopes to improve the model's predictions. A uniform distribution over V=50,000 tokens has loss $\ln50000\approx10.8$, roughly where an untrained model starts.
A mean loss of 2.3 corresponds to a geometric mean target probability near 0.1. We can now read a training curve with that interpretation in mind.
What you now know
- Loss gives training one objective to reduce.
- Next-token log loss scores the observed token's probability.
- −ln p strongly penalizes near-zero probability on the target.
- Our three candidates cost 0.41,1.41,2.41 against the same prediction.
- Mean loss is not an error count or an arithmetic mean probability.
Where we're headed
The loss defines the hill. Next we'll choose how to step across terrain less friendly than our first parabola.