Choose a ruler for the prediction
The model gave class 1 a probability of about 0.682. If class 1 is the supplied answer, how wrong is that prediction? A ruler that only checks whether the winning label matched throws away most of what happened. Different tasks need different rulers, so we build two and calculate their answers before either one trains anything.
A label is the supplied answer for an example. A loss is a scalar score comparing a prediction with its target. An objective is the quantity a training procedure tries to reduce. A loss turns a many-entry prediction into one comparable number, and that number means nothing until you state how it was calculated.
For numerical targets, mean squared error averages the squared differences between predictions and targets. Squaring stops positive and negative misses from canceling. It also makes big misses cost more: double a miss and its square goes up fourfold. The units come out as the square of the target’s units, which matters when you compare datasets on different scales.
These three numerical examples stand apart from our classifier. A prediction can be close without being right, and an average can hide which individual cases were bad. Look at the aggregate loss and at some of the examples behind it.
Three numerical misses
Predictions [1,2,3] and targets [1,1,2] give misses [0,1,1]. Their squares are [0,1,1]. Adding and dividing by three gives mean squared error (0+1+1)/3=2/3. The first example contributes nothing; the other two contribute equally.
Let N be the number of examples, y the target, and y-hat the prediction. The superscript in parentheses selects an example; it does not raise a number to a power. The square outside the difference is the squaring operation. The summation visits all N examples before the final division.
$$ L=\frac{1}{N}\sum_{i=1}^{N}(\hat y^{(i)}-y^{(i)})^2 $$In words: square each prediction’s miss, add the squares, and divide by the number of examples.
An unlikely outcome carries more surprise
Classification asks a different question: how much probability did the model give to the answer that occurred? An unlikely observed outcome should carry more surprise than a likely one. We also want independent probabilities multiplied together to become surprises added together. A logarithm gives that relationship.
A logarithm asks which exponent produces a number in a chosen base. Log base two of eight is three, because two cubed is eight. For probability one-half, negative log base two is one, because log base two of one-half is minus one. A bit is a base-two information unit, so one bit of surprise is an outcome assigned probability one-half.
A nat is the information unit built on the natural logarithm, written ln, the inverse of the exponential with base e. Changing the log base changes the numerical scale of surprise, so a loss in nats can’t be compared with the same quantity in bits without converting. Our classifier’s cross-entropy uses natural logarithms.
Average surprise for two coins
Entropy is average surprise under a probability distribution. For a fair coin, −(1/2)log₂(1/2)−(1/2)log₂(1/2)=1 bit. For probabilities (0.9,0.1), −0.9log₂(0.9)−0.1log₂(0.1)=0.136803+0.332193=0.468996 bits. The rare outcome is more surprising, and it shows up less often in the average.
Let p be the list of outcome probabilities and c select an outcome. Its surprise is weighted by how often that outcome happens under p. The expression H(p) names the entropy that results. That is uncertainty in the distribution, not a fraction of predictions a classifier got wrong.
$$ H(p)=-\sum_c p_c\log_2 p_c $$In words: weight each outcome’s surprise by its probability, then add.
An outcome with probability one carries zero surprise, because it was certain under that distribution. As an observed outcome’s assigned probability heads toward zero, its negative logarithm grows without bound. At zero the penalty is infinite in the limit.
Grade the model’s assigned probability
Back to the unchanged fixed model, input x=(1,0), and target class 1. A one-hot target puts a 1 on one class and 0 everywhere else. With class 0 first and class 1 second, the target is (0,1). Cross-entropy averages negative log probabilities using the target entries as weights.
Grade the fixed prediction
The probabilities are p=(0.318300,0.681700). With target (0,1), the loss is −0 ln(p₀)−1 ln(p₁)=−ln(p₁)=0.383166 nats. Had the supplied label been class 0, the loss would be 1.144760 nats. The prediction never moved; the target picked a different probability to grade.
Likelihood here means the probability the model gives the observed answer. Raise that probability and its negative logarithm falls.
Let y_c be the target entry for class c and p_c the model’s assigned probability. The negative sign turns the nonpositive logarithm of a probability into a nonnegative penalty. A one-hot target leaves one contributing term. A more general target distribution can weight several.
$$ L=-\sum_c y_c\ln p_c $$In words: multiply each log probability by the target’s entry, add, then negate; a one-hot label selects the correct class.
For the other pair of logits from the last chapter, (2,1), class 0 costs −ln(e/(e+1))=0.313262 nats and class 1 costs −ln(1/(e+1))=1.313262 nats.
See it move
Move the probability assigned to the true class toward zero, then toward one. Compare the logarithmic penalty with mean squared error against the one-hot target. Flip the label without moving the probabilities and see which share gets graded now. Coin mode weights surprise by the coin distribution itself and gives entropy in bits.
Scalar mode goes back to the three numerical predictions and targets. Edit a cell and the matching miss changes before the average is recomputed. A loss number on its own is incomplete: you need the prediction type, the target convention, the averaging rule, and the units to read its size.
A surface over the weights
So far the predictions moved directly. A training algorithm changes the parameters that produce those predictions instead. Try many pairs of weights and calculate the loss at each pair. Every pair gives one height, and the heights form a surface over weight space. The horizontal coordinates now name parameters, not input features.
This small linear model has two weights and no more, so the surface is its whole objective over those two directions. The contours and the three-dimensional view come from the same grid of computed losses. The path its weights took during training is drawn on top, and its final loss is 0.006484.
That surface belongs to a numerical fitting task. It is not the cross-entropy surface of our seventeen-parameter classifier. A larger model can only be drawn through selected directions or slices in a three-dimensional picture. Rotating such a picture changes the view and not the number of parameter directions inside it.
Contour spacing tells you how fast the loss changes across the displayed plane. Equal-height curves close together mean a larger change over a short horizontal distance than widely spaced ones. That sets up the next question: can we measure the local change with respect to each weight and pick a small move that lowers the loss?
Where this shows up when you train
For what these loss numbers look like across a real training run, see Reading the Graphs.
During training a prediction can hold many entries while the loss supplies one scalar to differentiate. The objective has to match what the task rewards. Numerical distance, probability assigned to a label, and a downstream task score are not the same quantity. Record the loss formula and the reduction convention alongside the run.
The fixed model is graded and unchanged. The target is still class 1 and the loss is still 0.383166 nats. Next: measure how one weight affects that number, use the slope to choose an update, and recompute the prediction to see whether the step helped.
What you now know
- A loss grades a prediction using a defined ruler.
- Entropy is average surprise under a distribution.
- Cross-entropy for a one-hot label is the negative log probability of that label.
Where we’re headed
We have a score. Next: which direction each weight has to move to bring it down. Continue to the next chapter.