Part II · Ch. 13 — What Gets Tracked, and What It Looks Like

Part II · Chapter 13 of 29

What Gets Tracked, and What It Looks Like

Read the changing measurements of a training run


Read progress and the two losses

A training dashboard shows dozens of moving traces. Which one says whether the model is learning, and which one explains why it stopped improving? We’ll read the panels one at a time, each at the same point in training.

Training loss is the objective evaluated on training examples. Validation loss evaluates the same rule on a held-out development set. Generalization means performance on examples never used to fit the parameters. The gap between validation and training loss is worth watching, as long as both use the same objective and you know their averaging conventions.

The digit CNN’s training and validation cross-entropy in nats above validation accuracy as a fraction, all plotted against step. The loss panel also labels recorded epochs, and a vertical marker identifies step 2577.
Read loss and accuracy at the same progress point.

Start with the digit CNN’s step and epoch. A step counts an update; the epoch counts consumed training examples against training-set size. The upper axis gives the matching fractional epoch. Accuracy is correct labels divided by evaluated examples. It is a task metric, a task-specific performance measure, and it needn’t move whenever the loss does.

Read the digit CNN at step 3000

Training loss is 0.0082 and validation loss is 0.1284. The gap is 0.1284−0.0082=0.1202. Validation accuracy is 0.961. The best validation step is 2577, at epoch 67.8, with training loss 0.0140 and validation loss 0.1245. By step 3000, epoch 79.0, training loss is lower and validation loss is higher than that best value.

That comparison names a pattern worth investigating. One upward tick is not evidence of overfitting, because finite samples and changing parameters make curves fluctuate. Look at the raw trace over an interval before smoothing it. Smoothing can reveal a trend, but the raw values are what was measured.

Look inside the update

A gradient norm summarizes the magnitude of the loss sensitivities. A weight norm summarizes parameter magnitude. Both use the L2 rule: square the entries, add, take the square root. A global norm combines all entries; a per-parameter norm describes one named array. Our dashboard keeps each layer’s weights and biases as separate arrays.

Four panels against training step: the digit CNN’s per-layer gradient norms on linear and log scales, the small dense classifier’s update divided by its post-update weight norm, and the digit CNN’s learning rate.
Gradients describe sensitivity; the optimizer determines the actual update.

Let g name the gradient vector. A spike in its norm says the current calculation produced a larger sensitivity, and nothing about why. Check the affected parameter arrays, the batch, the objective, and the learning rate. At step zero the gradient norm reads zero only because no gradient has been computed yet.

$$ \|\mathbf{g}\|_2=\sqrt{\sum_i g_i^2} $$

In words: square the gradient entries, add them, and take the square root.

The update-to-weight ratio compares measured parameter movement with post-update parameter magnitude. It earns its keep when an optimizer rescales gradients, because the raw gradient norm can’t reconstruct the update. Let θ, theta, name all model parameters and k the update index. The small denominator floor keeps the division defined.

$$ R_k=\frac{\|\theta_k-\theta_{k-1}\|_2}{\max(\|\theta_k\|_2,10^{-12})} $$

In words: θ (theta) stands for all of the model’s parameters at once; divide their actual change’s magnitude by their post-update magnitude, with a small denominator floor.

An actual parameter-array ratio

Take the first update of the small dense classifier. For its first weight matrix the update norm is 0.02500 and the post-update norm is 2.0024, so the ratio is 0.02500/2.0024≈0.0125. The first bias gives 0.0123 and the second weight matrix 0.0092. The second bias gives exactly 1.0, because it started at zero and the new bias equals the update.

No universal healthy threshold exists for that ratio. A value of one for an initially zero bias needn’t signal a broken run. Check the denominator convention and the loss behavior beside it. The digit CNN’s learning rate is constant. The step schedule drawn for comparison, 1/2 at first and 1/4 after its midpoint, is a made-up example.

A single norm can hide many values

One magnitude hides how the entries are spread. A histogram counts values in ranges called bins. Two arrays can have similar norms while one holds many small entries and the other packs its magnitude into a few large ones. The distribution plus a few selected values tells those cases apart.

Six panels show first-convolution weight counts at three steps, offset versions of those histograms, activation counts from the digit CNN and the name transformer with mean, standard deviation and zero fraction, and inactive-unit fractions versus step.
Distributions and zero fractions expose information that a single norm hides.

The weight histograms use twelve fixed bins over [−3,3]. Values outside that range don’t enter the bin counts, so the viewer reports how many were left out. Overlay and offset views show distributions at several recorded steps. Offsetting a histogram vertically is a display choice; the shift is not extra entries.

Standard deviation measures spread around the mean: subtract the mean from every entry, square those differences, average them, take the square root. A zero fraction counts exact zeros divided by the number of values inspected. Activation statistics here come from the last validation forward pass, not from the training mini-batch behind an update.

Inspect [0,0,1,1]

The mean is 1/2. Every squared deviation from that mean is 1/4, so their average is 1/4 and the population standard deviation is 1/2. Two of four entries are zero, so the zero fraction is 1/2. The two quantities answer different questions, even where their values agree.

A dead unit gives zero for every inspected example. A high layer zero fraction on its own doesn’t show that every unit is dead, or that any unit is permanently inactive. For the digit CNN we recorded the outputs of both ReLU layers, and for the name transformer the ReLU inside its feed-forward block. The inactive-unit fraction at each recorded step is the share of units that gave zero for every validation example.

Track throughput, memory and saved states

Throughput asks how many examples were processed per second, and its timing boundary decides what that means. For the digit CNN the rate counts updates plus a full evaluation pass with the recording hooks attached. Its batch-wait time measures preparing batches in memory. A slow trace suggests where to look.

Four charts for the digit CNN plot instrumented samples per second, process RSS in bytes, batch preparation in milliseconds, and update plus evaluation in milliseconds against step. Saved states and the best validation score are marked; GPU utilization was not recorded.
Throughput, memory and timing for the digit CNN. GPU utilization was not recorded.

The digit CNN’s process RSS, its resident memory in RAM, ranges from 396 MB to 419 MB over the run. RSS is not VRAM. The timing panels show batch-wait time and compute time separately, the latter covering update and evaluation work. Comparing those boundaries tells you more than reading either number as generic “training time.”

Saved-state markers show the steps where parameters were saved. A best-so-far score comes from the cumulative minimum of validation loss. That best can fall between saved snapshots, so a “best” marker doesn’t promise a parameter file at that step. A complete resume checkpoint would also need optimizer state beyond these parameter snapshots.

Some panels are pictures with numbers

Some recorded quantities are arrays whose arrangement carries meaning. A kernel is a small shared image-weight grid. A feature map is its response across spatial locations. A hidden state carries numerical memory along a sequence. Attention weights are normalized mixing shares over positions. The architectures come later; their recorded values can be inspected now.

Six panels show a kernel by row and column, a spatial feature map, attention by query and key position, two raw hidden coordinates, a selected feature cell versus step, and confusion counts by true and predicted class.
Training measurements can be arrays and images as well as scalar curves.

Every panel shows which example it probes and at which step. A feature map from a fixed input example is not a weight kernel, even when both show up as small grids. Select a cell and you get a number and a trace over the available frames. The hidden-state panel plots two of the raw hidden coordinates; it is not a projection of all of them.

Read the evaluation grid

A confusion matrix counts true-class rows against predicted-class columns. The digit CNN’s final diagonal is [58,58,57,58,59,58,59,58,52,57], totaling 574 correct cases. Recall for a class is its diagonal count divided by its row sum. Precision is that count divided by its prediction-column sum. An empty denominator leaves the ratio undefined.

Off-diagonal cells locate specific confusions. They supply no probability thresholds, because counts don’t keep the assigned probability of each example. The digit CNN also saved its probabilities, which chapter 22 uses for threshold-based evaluation. Here the per-class curves show recall and precision against training progress.

Learning, stuck, overfitting or diverging?

Underfitting means a fitted model still misses important structure in its training examples. Overfitting means fitting details that don’t transfer to held-out data. Divergence means unstable increasing error or nonfinite computation. A nearly flat score is a plateau, and the shape alone doesn’t name its cause.

Four trials from the learning-rate and width sweep compare solid training-loss and dashed validation-loss curves against step. Each panel states the trial learning rate, hidden width, and training sample count.
A curve suggests checks; settings and further evidence determine the cause.

The four cases are real trials from our learning-rate and width sweep. One has learning rate 0.00001 and little change in either score. Another uses width two and improves both training and validation scores, so the narrow architecture by itself is no proof of underfitting. A wide model trained on forty examples ends with a validation score above its earlier best while training keeps improving.

See it move

Pick a symptom before you reveal the settings. Weigh the evidence for it against the evidence that the conclusion is too strong. The large-rate case ends at loss 6.08 and every value stays finite, so its loss never blew up to infinity. A good next check compares another rate, inspects gradients, or evaluates an earlier saved state.

Follow one measured dashboard quantity at a time, then use its recording time to interpret it.

The digit CNN’s rise in validation loss after its best step shows why development decisions need more than the last training-loss point. Look at sustained behavior and repeat comparisons on consistent splits. A dashboard focuses the investigation; it can’t turn a suggestive curve into proof of one cause.

Move one cursor and inspect every panel

The dashboard opens with loss, accuracy, and learning rate. Add panels as your questions need them. One cursor carries the same progress coordinate through every view, and each sparse array states the step of its latest preceding frame.

See it move

Load the spiral network, the digit CNN or the name transformer, compare two runs, or open the run.json from your own trainkit run. Step, epoch, and elapsed-time comparisons use each run’s own coordinate array. When one run is shorter than the other, its curve stops at its last recorded point.

Where this shows up when you train

A real task tracks metrics beyond classification loss. The dashboard also holds eight epochs from one of our weather models, the red-band super-resolution network of chapter 27, with 1,515,265 parameters. Its outputs can be compared with a bicubic interpolation baseline at each recorded epoch.

Three charts over epoch show the recorded training objective, PSNR in decibels, and SSIM score. The latter two compare the learned model with bicubic interpolation.
Eight epochs of a weather model, with task-specific metrics.

PSNR is peak signal-to-noise ratio: 10 log₁₀(peak²/MSE), in decibels. SSIM, structural similarity, compares local luminance, contrast, and structure. The model’s PSNR moves from 33.569 to 33.572 dB, and its SSIM from 0.8925 to 0.8939. The loss panel is the training objective, a separate number.

“Energy” needs units too. A sum of squared signal values is a different measurement from electrical energy in joules. Put the quantity, its calculation, its units, and its evaluation examples on the panel. Those habits let people compare runs without assuming that similarly named traces measure the same thing.

What you now know

  • A dashboard connects measurements at a common progress point.
  • Training loss alone cannot establish generalization.
  • Missing telemetry cannot be reconstructed by drawing a plausible curve.

Where we’re headed

Next: the same dashboard, applied while opening each model family, starting with a dense network. Continue to the next chapter.