Choose the task before the model
We have a dataset and a way to turn it into tensors. The next choices are what the run should optimize and how we decide when it has done enough. I want those written down before a promising curve tempts us to change the question. A training recipe is a set of testable decisions, not a pile of lucky knobs.
A hyperparameter is a choice outside the parameter updates, such as width or learning rate. Capacity is the range of patterns a model can represent. A baseline is a simple comparison rule. Choose a baseline and a metric on the intended split before you choose a model size. A network that can’t beat a useful baseline hasn’t earned its extra complexity on that task.
The recorded families give you starting points. Dense layers combine table features; convolution shares rules over image grids; recurrent updates carry state through ordered streams; attention compares token positions; graph layers gather along explicit edges; autoencoders reconstruct inputs. Match classification labels to cross-entropy, and numerical or reconstruction targets to squared error, where those objectives fit the task. Match the baseline too: majority class, training mean, or persistence of the last observed value.
Look for a useful rate range
A learning-rate finder is a short run that raises the rate to find a promising range. That is a different experiment from a set of independent trials, each at one fixed rate. Our sweep holds fixed-rate trials; our range test is one continuous run with the rate rising. Both can be plotted against an axis called “rate,” so read the procedure along with the axis labels.
The range test applies 200 increasing rates, from 1e-5 to 1, to the spiral network and its data split, using Adam. Its rule picks the rate where the loss falls fastest against log10(rate), using the unsmoothed full-training loss after each update and excluding the endpoints. It suggests about 0.37, at step 183. That is a rule of thumb for this run; a different model or dataset needs its own test.
An increasing-rate test by hand
Let L(w)=(w−1)², so the gradient is 2(w−1), starting at w=0 and loss 1. Use rates [0.1,0.2,1,2] in that order. First the gradient is −2, so w becomes 0−0.1(−2)=0.2 and loss becomes 0.64. Next the gradient is −1.6, so w becomes 0.2−0.2(−1.6)=0.52 and loss becomes 0.2304. These are sequential steps on one changing weight, not four independent starts.
The larger rates
At w=0.52 the gradient is −0.96. Rate one gives 0.52−1(−0.96)=1.48, with after-step loss 0.2304 again. We crossed the minimum without lowering the loss. The gradient is now 0.96; rate two gives 1.48−2(0.96)=−0.44, whose loss is (−1.44)²=2.0736. That is nine times the previous loss. The resulting weights are [0.2,0.52,1.48,−0.44].
Compare like with like
A fair fixed-rate comparison keeps the other relevant choices in view. The sweep has 84 trials: two families, seven rates, three widths, and two requested data sizes. The requested size is not always the size you get. The small dense classifier’s larger request, for one, gets only 48 training rows. Calling it a 300-row experiment would spoil its comparison with the spiral network before anyone opened the curves.
Each line crosses a trial’s family, rate, width, requested size, actual training size, validation size, and final validation loss. Those axes use different scales, so a vertical alignment implies no equal units. Filter one family and one actual size before you compare rates or widths. A line that looks unusually steep between two axes can come from axis scaling rather than an unusual training event.
A sorted endpoint table finds candidates; the raw curves show how they got there. Keep a failed trial’s recorded status and whatever endpoint exists, and never invent a final value. Training-loss minima on their own can reward memorization. Validation supplies the development criterion, and repeated selection against it is still development. These coupled small trials establish no causal ranking of parameter importance.
Choose constraints before seeing the final score
Regularization covers choices that discourage fitting details that don’t transfer. Dropout randomly masks intermediate activations during training. A seed initializes pseudorandom choices such as masks, shuffling, or weights. Reproducibility means recording enough procedure, data, and environment information to repeat the result. The seed is one part of that record.
Inverted dropout
Take hidden values (1,2), a keep fraction of 1/2, and a sampled mask (1,0). Inverted dropout keeps the first value and divides it by the keep fraction, dropping the second: (1×1/(1/2),2×0/(1/2))=(2,0). Inference uses the full (1,2). The scaling preserves the expected contribution over masks, not this particular mask’s contribution.
After development, evaluate the final test once under the agreed rule. If the result makes you change the model, that test has entered development and no longer plays its untouched role. Keep the whole decision record, unsuccessful choices included. Reporting only the winning seed or best endpoint hides how much selection happened, and it makes a repeat look inexplicably worse.
Where this shows up when you train
The Run Designer opens on the spiral network’s slice of the sweep: planar binary classification, width sixteen, 300 requested rows. Narrow the rate range with the plot’s brush or the numeric controls. The companion table and the raw curves update together. Then edit the hand rate sequence and inspect each new weight and loss. Those hand calculations are separate from the measured trial endpoints.
What you now know
- The task and evaluation criterion determine what to optimize.
- Sweep comparisons must hold relevant settings and actual data size in view.
- A seed and a stopping rule are parts of the run’s reproducible record.
Where we’re headed
Next: deciding whether the trained model did anything useful beyond the baseline. Continue the story.