Part IV · Ch. 22 — Did It Actually Learn? Evaluation

Part IV · Chapter 22 of 29

Did It Actually Learn? Evaluation

Compare held-out predictions with a baseline


What did the model beat?

Training finished and printed a score. Did the model learn something useful, or run a complicated version of an easy guess? Answering takes a comparison rule, an evaluation population, and a definition of success. The final loss line is evidence and not a verdict.

A majority-class baseline always predicts the most common training label. A mean baseline predicts the mean training target. A persistence baseline predicts the last observed value as the next one. Fit each comparison rule on training information and score it on the same evaluation examples as the model. A baseline tested on different data is not an equivalent opponent.

Two panels on separate axes: the digit CNN’s validation accuracy beside its baseline, where higher is better, and the stream GRU’s mean squared error beside its baseline, where lower is better.
Compare a model and baseline on the same examples using the same metric.

Reading the Graphs follows a weather model’s validation margin over an advection baseline across recorded epochs, and shows how a “beat the baseline” checkpoint score can stall while loss keeps falling.

The digit CNN and its baseline are compared by validation accuracy, where higher is better. The stream GRU and its baseline are compared by mean squared error, where lower is better. The figure puts each baseline beside the model’s final score. The two panels keep separate axes, because an accuracy fraction and a squared numerical error have different units and different meanings.

High accuracy, no positive detections

In a hand set with nine negatives and one positive, always predicting negative gets 9/10 accuracy. It finds 0/1 positives. The aggregate score is high because the common class dominates the count. If the positive event is the thing you care about, that rule fails at the part of the task that matters. Count the kinds of mistakes before deciding whether a model improved anything.

Count the kinds of error

A confusion matrix records truth rows and prediction columns. In a binary task, “positive” names the event you choose to focus on. True positive means a positive prediction on a real positive; false positive means a positive prediction on a real negative. False negative means a missed real positive, and true negative means a correctly rejected real negative. The same event keeps the same definition across all four counts.

The confusion table [[4,1],[2,3]] with truth rows and prediction columns, each cell named true negative, false positive, false negative or true positive, linked to the precision, recall and F1 fractions.
Different error counts answer different practical questions.

Our hand matrix is [[4,1],[2,3]], ordered negative then positive on both axes. So true negatives are four, false positives one, false negatives two, and true positives three. Accuracy is (4+3)/(4+1+2+3)=7/10. Precision asks what fraction of predicted positives were right. Recall asks what fraction of real positives were found. Their denominators answer different questions, even though both numerators count the same three true positives.

Fractions from the counts

Precision is 3/(3+1)=3/4. Recall is 3/(3+2)=3/5. F1 combines them through the harmonic mean and can be written straight from counts as 2×3/(2×3+1+2)=2/3. Don’t average precision and recall arithmetically. With no predicted positives, precision’s denominator is zero and the result is undefined. With no real positives, recall is undefined. Neither case means perfect performance.

Numerical prediction errors

For predictions [1,2,3] and truths [1,1,2], the errors are [0,1,1]. Mean absolute error, MAE, is (0+1+1)/3=2/3. Root mean squared error is sqrt((0+1+1)/3)=sqrt(2/3). A square root is the nonnegative number whose square gives its input. Taking it after averaging squared errors brings the metric back to the target’s units. Squaring first makes larger misses count for more than they do in absolute error.

In the formulas, TP, FP, and FN are the counts just defined. N is the number of evaluated examples, y is the true numerical target, and y with a hat is the prediction. RMSE names root mean squared error. Write the counts or error values beside each formula so a score stays traceable to individual observations:

$$\mathrm{precision}=\frac{TP}{TP+FP},\quad\mathrm{recall}=\frac{TP}{TP+FN}$$

In words: precision asks what fraction of predicted positives were right; recall asks what fraction of actual positives were found.

$$F_1=\frac{2TP}{2TP+FP+FN}$$

In words: combine precision and recall through their harmonic mean, written here using counts.

$$\mathrm{RMSE}=\sqrt{\frac{1}{N}\sum_i(\hat y_i-y_i)^2}$$

In words: average squared misses, then take the square root to return to the target’s units.

Scores allow different decision rules

A classifier produces scores or probabilities before anyone picks a decision rule. For one selected digit, call the case positive when that digit’s probability is at least a threshold. Lowering the threshold usually admits more positives and more false alarms; raising it rejects more of both. The threshold states a decision tradeoff and changes none of the model’s learned weights.

Precision–recall and ROC curves for one digit, traced by sweeping the threshold across the digit CNN’s validation probabilities. Tied scores move together, and points with a zero denominator are marked undefined.
Changing the threshold trades different kinds of errors.

The digit CNN saved validation probabilities and labels at 31 checkpoints: steps 0,100,…,3000. Each holds all 597 validation examples and ten class probabilities. The widget defaults to digit one as positive, with every other digit negative. It predicts positive when the saved probability is at least the threshold. Tied scores cross the threshold together.

A calibration example

Calibration asks whether predicted probabilities match observed frequencies. Suppose four predictions all assign probability 1/2, and two of the outcomes are positive. Their observed positive frequency is 2/4=1/2, matching the predicted probability for that group. Four observations are far too few to judge calibration. In the measured view each bin shows its sample count, and an empty bin is left blank.

Use the final assessment once

Per-class recall comes from every stored confusion checkpoint: divide the diagonal count by that truth row’s total. It shows whether improvements spread across classes or gathered in a few. Choose the development checkpoint before you open the final test. Picking whichever checkpoint wins on that final set turns it into another development set.

The digit CNN’s confusion matrices at its first, a middle and its final saved checkpoint, labeled with step and epoch on one color scale, beside per-class recall curves drawn at the saved checkpoints.
Checkpoint matrices expose which classes improved and which mistakes remained.

Each matrix is labeled with the step and epoch at which it was saved. When you move to a step between checkpoints, the widget keeps showing the last saved checkpoint, and its step badge tells you which one.

Distribution shift is a difference between development data and the inputs met later. A model selected on these digit images can meet new stroke styles, image scales, or upstream preprocessing. A strong validation score measures the given split, not every later population. An accuracy drop alone can’t diagnose an external failure; inspect the changed inputs and the procedure that produced the original score.

Where this shows up when you train

Before evaluating a candidate, write down the baseline, the metric, the split, the positive class if there is one, and the acceptance criterion. If missed events and false alarms cost different amounts, put that tradeoff in writing. A probability threshold should serve the intended use, not get tuned until one attractive scalar appears. Keep the confusion counts beside the scalar so anyone can reconstruct how the criterion was applied.

What you now know

  • A baseline must use the same evaluation examples and metric.
  • Accuracy, precision, recall and reconstruction error measure different outcomes.
  • Development choices consume validation information, so final testing needs separate data.

Where we’re headed

Evaluation is covered. Next: using the measurements to diagnose what failed and to check a repair. Continue the story.