Part IV · Ch. 23 — When It Goes Wrong

Part IV · Chapter 23 of 29

When It Goes Wrong

Connect symptoms to causes and checks


Name the symptom before its cause

A loss curve looks wrong. The tempting response is to change several settings until it looks better. Slow that decision down enough to make the next run informative: name the symptom, state a hypothesis, choose one check, and decide what result would reject the hypothesis. A change smaller than the noise floor, the variation the measurement cannot distinguish, rejects nothing.

The evidence should include raw curves, the split, the settings, and the recorded status. Smoothing hides a spike, and a finite loss can still be very high. “Diverged” gets used loosely for poor learning, while nonfinite arithmetic is a specific claim. Don’t call a value NaN because it went up. The saved status and numeric entries tell those cases apart.

Eight small cards, one per failure case. Cards with measured curves name the sweep trial they come from; the others are labeled as hand examples or not recorded.
A named failure is a hypothesis until its evidence and check agree.

The eight cases cover numerical instability, zero gradients, a widening development gap, weak learning, majority-class behavior, outside-data failure, slow training, and disagreeing reruns. Some carry real curves from the sweep. Others are hand calculations or hypotheses with no recorded example, and each card says which.

When arithmetic or learning stalls

A flat loss can mean the weights aren’t moving, that they move too little to matter, or that the chosen metric barely responds to them. Gradients separate those possibilities. Check the parameter values as well as the plotted line. When a gradient is zero, ask which local factors produced it before treating zero as proof that the model found a useful minimum.

A zero product path

Let the hand prediction be w₁w₂x, with input x=1, target 1, and both weights zero. The prediction is 0 and the squared error is 1. The derivative with respect to w₁ contains w₂, and the derivative with respect to w₂ contains w₁, so both are zero. A coordinate-wise finite difference also returns zero here. Moving both weights together lowers the loss, so this zero-initialized saddle is not a useful solution.

Left: the zero-product toy, with both local derivatives at zero and the loss stuck at one. Right: two measured sweep curves, a narrow model learning weakly and a wide model on little data, with their actual endpoints.
A flat curve can have several causes; gradients and settings narrow the explanation.

Changing the initialization breaks that zero product path. Verify first that the new analytic gradients agree with numerical differences, then take an update. The change counts as a repair only if the loss and the held-out checks then improve. The hand model is a small counterexample to the assumption that a zero gradient means the job is done.

When a good-looking score is misleading

A model near majority-class accuracy might predict only the common class, or it might make a different mixture of mistakes. Aggregate accuracy can’t tell them apart. Read per-class confusion counts and label mappings first. We have no measured run that collapsed to the majority class, so the example is the hand case from evaluation: nine negatives, one positive, all-negative predictions, accuracy 9/10 and positive recall zero.

Excellent validation followed by poor external performance can come from leakage or from distribution shift. Those need different investigations, and one curve can’t separate them. Audit duplicate groups, temporal boundaries, and where preprocessing was fitted. Compare external inputs with the development population. A new independent evaluation tests the intended use, and tuning against it repeatedly turns it into development evidence too.

When it is slow or hard to reproduce

Slow training needs a timing boundary. Are you timing an arithmetic kernel, a complete update, full validation, input loading, or exported diagnostic frames? Those are different workloads. The digit CNN’s CPU run records its own timing as it trains. Its wait and compute times are useful measurements, but their names alone don’t say that a disk loader or a GPU sat idle.

Our GPU timing benchmark’s compute curves, with and without the copy to the GPU, beside the digit CNN’s CPU timing: batch wait and compute time per update. The CPU run has no GPU utilization to show.
Available timing evidence can narrow a bottleneck without proving an unmeasured cause.

In the digit CNN’s run, the wait time covers in-memory batch preparation: choosing indices, shuffling, slicing. The compute time covers the update plus full-split evaluation, excluding the batch wait and the frame export. A CPU run has no GPU utilization to record. Our GPU timing benchmark measures matrix compute separately, with and without the copy to the GPU. Those are the intervals we have. A loader-bound run, one starved by its data supply, and an idle GPU both remain untested hypotheses here.

When reruns disagree, compare inputs and procedure before comparing final accuracy. The data file, split indices, preprocessing, seed, parameter initialization, software version, and numerical operations can all matter. A seed promises nothing across platforms. A small early difference can change the whole path of optimization later, so the first differing state usually tells you more than the size of the final gap.

Where this shows up when you train

Choose a case and describe the raw evidence before you reveal a cause. Pick a check that could disprove the proposed explanation. For the zero-chain example, nonzero checked derivatives after an initialization change support the mechanism. For an overfitting hypothesis, a repeated held-out improvement under the predefined comparison matters more than a prettier training curve. Different symptoms need different acceptance evidence.

What you now know

  • A symptom does not uniquely identify its cause.
  • A repair needs a named check on appropriate data.
  • A measurement that was never recorded cannot be filled in afterward.

Where we’re headed

Next: run trainkit, open its outputs, and explain the numbers in a model of your own. Continue the story.