The four curves
The loss is falling and the saved best model is six epochs old. Is training still helping? That question has no answer until each curve is named. Here are four curves from our weather-model runs, with the task’s score beside the error that training reduces.
Training loss per step is one mini-batch’s error. Training loss per epoch normally averages a whole pass, weighting batches by their sizes where that matters. Our runs record the loss only every few steps, so our epoch training curve is the mean of those recorded step losses: an estimate of the whole-pass average.
A validation score measures performance on held-out examples the model never fits during its parameter updates. The learning rate is a recipe setting, not a measurement of the model; it is plotted to show when the update scale changed. Keep the loop’s batch, epoch, and schedule vocabulary and loss units in view.
The band and the level
Each mini-batch holds different examples. Some are harder, so a batch loss can rise while the run improves over many updates. Read the height of the band across an interval before you read one wiggle. Sampling noise is variation from which examples were drawn, and the changing parameters add variation of their own.
The horizontal axis counts recorded samples, not optimizer updates or elapsed seconds. The recording ends with evaluations for epochs 1–8 and ten step samples from epoch 9, which has no validation point. The at-home exercise below shows the related curve trainkit records.
From scratch and from a checkpoint
A fine-tune starts from learned weights. Our main nowcasting run started from scratch; an Alaska run continued from its checkpoint on Alaskan data. Both use a model with 27,223,297 parameters. A fine-tune can start near the earlier run’s finishing loss, with less distance to travel, and a different regional dataset can change the starting difficulty.
Validation loss is the error objective on held-out examples. The main run’s falls from 0.1661 to 0.1393 over eight epochs, and the Alaska run’s from 0.1435 to 0.1310 over five, starting close to where the main run ended. The runs use different evaluation sets, so the curves compare shape, not which run is better.
The loss falls while the score stalls
A plateau here means an interval with no new best task score. It needn’t be a flat line. The Alaska run’s score peaks at epoch 2, then sits below that peak while validation loss keeps falling through epoch 5.
Check the margin, then the checkpoint
At epoch 4 of the main run, the model’s 30-minute CSI at 20 dBZ is 0.6757 and the advection baseline’s is 0.5044. The margin is 0.6757 − 0.5044 = 0.1713. Average the four margins (two thresholds, two lead times) to get this chapter’s validation score.
The Alaska run’s scores, to five decimal places, are 0.20952, 0.21453, 0.19554, 0.20015, and 0.19894. Compare each with 0.21453: none of the last three wins, and the run’s own saved-best record agrees.
An improving loss proxy doesn’t mean the thing you care about is still improving. That is a different situation from training loss falling while validation loss rises; the Alaska run’s validation loss falls too. When It Goes Wrong separates the diagnostic symptoms. This curve alone establishes neither overfitting nor its cause.
Read the recipe beside the result
The main and Alaska runs both use constant learning rates: 0.0002 for the base parameter group and 0.001 for the output head, five times as large. A parameter group is a set of weights given the same optimizer settings. Neither has a warm-up or a scheduled drop.
The loop’s constructed step and warm-up sketches show the other shapes. A step schedule drops at a chosen point; warm-up ramps upward at the start. Either changes the update recipe and can change the loss trend or its variability. Neither guarantees a visible kink. The dashboard chapter has a constructed schedule example.
Steady, spiking, or pinned at the ceiling?
A gradient norm measures gradient magnitude. A steady trace stays in a narrow range; occasional spikes leave it. A trace pinned at a clipping ceiling keeps reaching the imposed cap. Each shape calls for a different check.
274 of the 307 recorded norms equal 1.0: 89% at whole-percent precision. The smallest is 0.199. Clipping limits the gradient handed to the optimizer, so repeated contact with the ceiling is worth inspecting. It doesn’t show how far the unmodified gradients went over the cap, and an adaptive optimizer’s parameter movement can’t be reconstructed from this norm alone.
Throughput, utilization, and memory
The main run’s throughput ranges from 0.61 to 4.11 samples per second. The same model processes examples at quite different observed rates. Waiting for data, evaluation, and checkpoint work are candidates to investigate. Our recording doesn’t separate their contributions or record GPU utilization, so it can’t tell us whether this run was waiting on data.
Allocated device memory ranges from 885,530,112 to 980,466,688 bytes, a near-plateau on the panel’s zero-based scale. This sampled allocation leaves out temporary peaks. The run’s settings list two workers; its first epoch recorded one and epochs 2–8 recorded two. Every logged step names the first GPU, so the memory trace is for one device, not the total across two.
Patience and the recorded best
Patience is how many evaluations without a new best you allow before stopping. Our runs record a counter of evaluations since the last best, but no stopping rule. The main run’s counter reads 0,0,1,2,3,4,5,6; the Alaska run’s reads 0,0,1,2,3.
Count since the last best
For Alaska at epoch 5, 5−2=3 evaluations have passed since the best. A patience of three would stop training there. Both runs last saved a new best at epoch 2.
The main run’s epoch-2 score is 0.18292, to five decimals. Epoch 3 drops to 0.17505; epoch 4 rebounds partway to 0.17924, still under the peak. The counter never resets after epoch 2. Its best checkpoint stays at epoch 2 even though validation loss falls for six more epochs.
Lay every retained epoch on the same axes
The super-resolution model, which reconstructs a finer image, gives a quieter late-stage example. These are the five most recently retained epochs of a longer run, after its loss function changed, numbered 11–15. The model has 1,367,553 parameters; we kept no record of its earlier epochs.
The run’s settings record a change at epoch 11 from absolute-error loss to absolute error plus structural similarity, which compares local brightness, contrast, and structure. The recorded epoch loss goes from 0.01580 at 11 to 0.01575 at 15. Older lines are fainter; the transparency marks age, not confidence.
Where this shows up when you train
Choose the evaluation measure before you interpret progress. Then name the data split, the averaging rule, and the best-checkpoint flag. The widget starts at the Alaska run’s first evaluation. Scrub the visible history and choose “Stop here” before you reveal the later epochs and the recorded best. Repeat with the main run and compare.
See it move
Run this yourself
What you now know
- Batch loss, epoch loss, validation score, and learning rate describe different things.
- A moving loss band needs an interval and an averaging convention.
- The saved best follows the selected task score, which can stall while loss improves.
- Resource traces suggest checks but do not identify an unmeasured cause.
Where we’re headed
The main curve shapes are readable now. Next: connect them with internal model measurements at a shared progress point. Continue to the next chapter.