Part IV · Ch. 21 — Designing the Run

Part IV · Chapter 21 of 29

Designing the Run

Choose the objective, model, batch, and learning rate


Choose the task before the model

We have a dataset and a way to turn it into tensors. The next choices are what the run should optimize and how we decide when it has done enough. I want those written down before a promising curve tempts us to change the question. A training recipe is a set of testable decisions, not a pile of lucky knobs.

A hyperparameter is a choice outside the parameter updates, such as width or learning rate. Capacity is the range of patterns a model can represent. A baseline is a simple comparison rule. Choose a baseline and a metric on the intended split before you choose a model size. A network that can’t beat a useful baseline hasn’t earned its extra complexity on that task.

A table with one row per recorded run: task, data shape, model parameter shapes, objective, settings and baseline, with each row’s assumptions highlighted.
Start with a recipe whose data and objective match the task.

The recorded families give you starting points. Dense layers combine table features; convolution shares rules over image grids; recurrent updates carry state through ordered streams; attention compares token positions; graph layers gather along explicit edges; autoencoders reconstruct inputs. Match classification labels to cross-entropy, and numerical or reconstruction targets to squared error, where those objectives fit the task. Match the baseline too: majority class, training mean, or persistence of the last observed value.

Look for a useful rate range

A learning-rate finder is a short run that raises the rate to find a promising range. That is a different experiment from a set of independent trials, each at one fixed rate. Our sweep holds fixed-rate trials; our range test is one continuous run with the rate rising. Both can be plotted against an axis called “rate,” so read the procedure along with the axis labels.

Left: final losses from the sweep’s separate fixed-rate trials. Right: the hand quadratic’s range test, showing each sequential update of w and the loss after it. The range test’s loss against its rising learning rate over 200 updates is plotted with its suggested step marked. Each panel has its own title.
A rate sweep and a range test answer related questions using different experiments.

The range test applies 200 increasing rates, from 1e-5 to 1, to the spiral network and its data split, using Adam. Its rule picks the rate where the loss falls fastest against log10(rate), using the unsmoothed full-training loss after each update and excluding the endpoints. It suggests about 0.37, at step 183. That is a rule of thumb for this run; a different model or dataset needs its own test.

An increasing-rate test by hand

Let L(w)=(w−1)², so the gradient is 2(w−1), starting at w=0 and loss 1. Use rates [0.1,0.2,1,2] in that order. First the gradient is −2, so w becomes 0−0.1(−2)=0.2 and loss becomes 0.64. Next the gradient is −1.6, so w becomes 0.2−0.2(−1.6)=0.52 and loss becomes 0.2304. These are sequential steps on one changing weight, not four independent starts.

The larger rates

At w=0.52 the gradient is −0.96. Rate one gives 0.52−1(−0.96)=1.48, with after-step loss 0.2304 again. We crossed the minimum without lowering the loss. The gradient is now 0.96; rate two gives 1.48−2(0.96)=−0.44, whose loss is (−1.44)²=2.0736. That is nine times the previous loss. The resulting weights are [0.2,0.52,1.48,−0.44].

Compare like with like

A fair fixed-rate comparison keeps the other relevant choices in view. The sweep has 84 trials: two families, seven rates, three widths, and two requested data sizes. The requested size is not always the size you get. The small dense classifier’s larger request, for one, gets only 48 training rows. Calling it a 300-row experiment would spoil its comparison with the spiral network before anyone opened the curves.

Parallel coordinates with one line per sweep trial, crossing axes for family, learning rate on a log scale, width, requested and actual training size, validation size, and final validation loss. Selected trials are highlighted and the rest are muted.
A sweep is interpretable only when the actual data sizes and settings remain visible.

Each line crosses a trial’s family, rate, width, requested size, actual training size, validation size, and final validation loss. Those axes use different scales, so a vertical alignment implies no equal units. Filter one family and one actual size before you compare rates or widths. A line that looks unusually steep between two axes can come from axis scaling rather than an unusual training event.

Final validation loss against learning rate for one family and one actual data size, one panel per width. Failed trials are marked with their status and have no final value.
Compare held-out results within a controlled slice of the sweep.

A sorted endpoint table finds candidates; the raw curves show how they got there. Keep a failed trial’s recorded status and whatever endpoint exists, and never invent a final value. Training-loss minima on their own can reward memorization. Validation supplies the development criterion, and repeated selection against it is still development. These coupled small trials establish no causal ranking of parameter importance.

Choose constraints before seeing the final score

Regularization covers choices that discourage fitting details that don’t transfer. Dropout randomly masks intermediate activations during training. A seed initializes pseudorandom choices such as masks, shuffling, or weights. Reproducibility means recording enough procedure, data, and environment information to repeat the result. The seed is one part of that record.

Inverted dropout

Take hidden values (1,2), a keep fraction of 1/2, and a sampled mask (1,0). Inverted dropout keeps the first value and divides it by the keep fraction, dropping the second: (1×1/(1/2),2×0/(1/2))=(2,0). Inference uses the full (1,2). The scaling preserves the expected contribution over masks, not this particular mask’s contribution.

After development, evaluate the final test once under the agreed rule. If the result makes you change the model, that test has entered development and no longer plays its untouched role. Keep the whole decision record, unsuccessful choices included. Reporting only the winning seed or best endpoint hides how much selection happened, and it makes a repeat look inexplicably worse.

Where this shows up when you train

The Run Designer opens on the spiral network’s slice of the sweep: planar binary classification, width sixteen, 300 requested rows. Narrow the rate range with the plot’s brush or the numeric controls. The companion table and the raw curves update together. Then edit the hand rate sequence and inspect each new weight and loss. Those hand calculations are separate from the measured trial endpoints.

What you now know

  • The task and evaluation criterion determine what to optimize.
  • Sweep comparisons must hold relevant settings and actual data size in view.
  • A seed and a stopping rule are parts of the run’s reproducible record.

Where we’re headed

Next: deciding whether the trained model did anything useful beyond the baseline. Continue the story.