Name the symptom before its cause
A loss curve looks wrong. The tempting response is to change several settings until it looks better. Slow that decision down enough to make the next run informative: name the symptom, state a hypothesis, choose one check, and decide what result would reject the hypothesis. A change smaller than the noise floor, the variation the measurement cannot distinguish, rejects nothing.
The evidence should include raw curves, the split, the settings, and the recorded status. Smoothing hides a spike, and a finite loss can still be very high. “Diverged” gets used loosely for poor learning, while nonfinite arithmetic is a specific claim. Don’t call a value NaN because it went up. The saved status and numeric entries tell those cases apart.
The eight cases cover numerical instability, zero gradients, a widening development gap, weak learning, majority-class behavior, outside-data failure, slow training, and disagreeing reruns. Some carry real curves from the sweep. Others are hand calculations or hypotheses with no recorded example, and each card says which.
When arithmetic or learning stalls
A flat loss can mean the weights aren’t moving, that they move too little to matter, or that the chosen metric barely responds to them. Gradients separate those possibilities. Check the parameter values as well as the plotted line. When a gradient is zero, ask which local factors produced it before treating zero as proof that the model found a useful minimum.
A zero product path
Let the hand prediction be w₁w₂x, with input x=1, target 1, and both weights zero. The prediction is 0 and the squared error is 1. The derivative with respect to w₁ contains w₂, and the derivative with respect to w₂ contains w₁, so both are zero. A coordinate-wise finite difference also returns zero here. Moving both weights together lowers the loss, so this zero-initialized saddle is not a useful solution.
Changing the initialization breaks that zero product path. Verify first that the new analytic gradients agree with numerical differences, then take an update. The change counts as a repair only if the loss and the held-out checks then improve. The hand model is a small counterexample to the assumption that a zero gradient means the job is done.
When a good-looking score is misleading
A model near majority-class accuracy might predict only the common class, or it might make a different mixture of mistakes. Aggregate accuracy can’t tell them apart. Read per-class confusion counts and label mappings first. We have no measured run that collapsed to the majority class, so the example is the hand case from evaluation: nine negatives, one positive, all-negative predictions, accuracy 9/10 and positive recall zero.
Excellent validation followed by poor external performance can come from leakage or from distribution shift. Those need different investigations, and one curve can’t separate them. Audit duplicate groups, temporal boundaries, and where preprocessing was fitted. Compare external inputs with the development population. A new independent evaluation tests the intended use, and tuning against it repeatedly turns it into development evidence too.
When it is slow or hard to reproduce
Slow training needs a timing boundary. Are you timing an arithmetic kernel, a complete update, full validation, input loading, or exported diagnostic frames? Those are different workloads. The digit CNN’s CPU run records its own timing as it trains. Its wait and compute times are useful measurements, but their names alone don’t say that a disk loader or a GPU sat idle.
In the digit CNN’s run, the wait time covers in-memory batch preparation: choosing indices, shuffling, slicing. The compute time covers the update plus full-split evaluation, excluding the batch wait and the frame export. A CPU run has no GPU utilization to record. Our GPU timing benchmark measures matrix compute separately, with and without the copy to the GPU. Those are the intervals we have. A loader-bound run, one starved by its data supply, and an idle GPU both remain untested hypotheses here.
When reruns disagree, compare inputs and procedure before comparing final accuracy. The data file, split indices, preprocessing, seed, parameter initialization, software version, and numerical operations can all matter. A seed promises nothing across platforms. A small early difference can change the whole path of optimization later, so the first differing state usually tells you more than the size of the final gap.
Where this shows up when you train
Choose a case and describe the raw evidence before you reveal a cause. Pick a check that could disprove the proposed explanation. For the zero-chain example, nonzero checked derivatives after an initialization change support the mechanism. For an overfitting hypothesis, a repeated held-out improvement under the predefined comparison matters more than a prettier training curve. Different symptoms need different acceptance evidence.
What you now know
- A symptom does not uniquely identify its cause.
- A repair needs a named check on appropriate data.
- A measurement that was never recorded cannot be filled in afterward.
Where we’re headed
Next: run trainkit, open its outputs, and explain the numbers in a model of your own. Continue the story.