One update has several stages
One update changed the fixed model’s prediction. What turns that calculation into a training run? Repeat the stages over examples, track progress, and record enough state to explain each step. The arithmetic inside a stage stays the same even as the surrounding loop grows.
A mini-batch is a subset of examples used for one update. A step is one optimizer update. An epoch is one training-set’s worth of consumed examples. Those counters measure different things: a large dataset cut into small batches takes many steps to finish one epoch. The last batch can be shorter and still counts.
Shuffle means changing the order of training examples, usually between passes. Stochastic gradient descent estimates the gradient from sampled examples instead of evaluating the whole training set for every update. Full-batch descent uses all the training examples. The choice changes the work per step and the variation in the gradient estimates.
The diagram keeps the fixed model’s parameter shapes attached to the loop: (3,2), (3), (2,3), and (2). A batch enters the forward pass, the loss reduces predictions to a scalar, the backward pass computes matching gradient arrays, and an update changes the parameters. The next batch then uses the new values.
Five examples with batch size two
Successive batches consume 2, 2, and 1 examples. After the first update the epoch count is 2/5. After the second it is 4/5. After the third it is 5/5=1. Three steps, one complete pass through the five examples. Counting batches as epochs would overstate the progress.
A batch loss and a whole-split evaluation loss are not the same number. The batch may hold easier or harder examples than the full set, and its loss may be measured before the update rather than after. A useful run record says which examples and which parameter state each reported quantity used. A chart labeled only “loss” leaves all of that open.
The gradient does not determine the step alone
An optimizer turns gradients and retained state into updates. Plain gradient descent needs the current gradient and the learning rate. Momentum keeps part of earlier directions. Adam, adaptive moment estimation, keeps averages of gradients and of their squares. Give all three the same gradient and they produce different parameter changes.
Use one illustrative weight w=1 and gradient g=1. With learning rate η, eta, equal to 1/2, plain descent gives w_new=1−(1/2)×1=1/2.
For momentum, let the old velocity v=1 and retention m=1/2. We use the unnormalized convention: new velocity equals retained old velocity plus the current gradient. So v_new=(1/2)×1+1=1.5, and w_new=1−(1/2)×1.5=1/4. The retained state made this step larger than plain descent’s.
$$ v_{k}=m v_{k-1}+g_k,\qquad w_k=w_{k-1}-\eta v_k $$In words: retain part of the previous velocity, add the current gradient, then step using the new velocity.
The subscript k counts updates. The previous velocity is v with subscript k−1, and g_k is the new gradient. Other conventions scale the incoming gradient as well, so the word momentum on its own doesn’t pin down the arithmetic. A reproducible description names the update rule and its retention setting.
Adam’s first update
Adam keeps a mean a and a mean square s of gradients. Set both retention factors to 1/2 and both initial states to zero. At the first g=1, a=s=1/2. Divide each by 1−1/2 to correct the zero initialization, giving a-hat=s-hat=1. With η=1/2 and a teaching denominator floor δ=0, the step is (1/2)×1/√1=1/2.
Let r retain part of the gradient mean and q retain part of the mean square. The square on g_k changes what the second average stores: magnitude without sign, while the first average can let positive and negative gradients cancel. The two states get different names because they play different roles in the final division.
$$ a_k=r a_{k-1}+(1-r)g_k,\quad s_k=q s_{k-1}+(1-q)g_k^2 $$In words: average the gradient and its square separately.
The hat marks the averages after correcting for their initial zeros. The denominator holds the square root of the corrected mean square plus a small positive floor δ, pronounced delta, in real software. The hand example can use zero because its denominator is already nonzero; an implementation keeps the floor.
$$ \hat a_k=\frac{a_k}{1-r^k},\quad\hat s_k=\frac{s_k}{1-q^k},\quad w_k=w_{k-1}-\eta\frac{\hat a_k}{\sqrt{\hat s_k}+\delta} $$In words: correct the initial zero averages, then scale the average gradient by the square root of its average square plus a small numerical floor.
Adam’s update magnitude is generally not the learning rate times the raw gradient norm. Retained averages and coordinatewise scaling get in between. That matters when you read an update-to-weight ratio: take it from the measured parameter change, not from an expression that assumes plain gradient descent.
Rate, decay, clipping and checkpoints
A learning-rate schedule changes η as training progresses. Warm-up raises the rate over an opening interval. A schedule can also stay constant, drop at chosen points, or follow a smooth curve. Those are chosen rules, and a smooth-looking loss curve doesn’t say which one was used.
Weight decay shrinks parameters toward zero under a stated rule. Gradient clipping caps gradient magnitude before the optimizer sees it. A checkpoint saves model values at a named step. Continuing the same optimizer trajectory also needs its retained state and other run state, and the parameter snapshots saved with this book’s runs leave that state out, so they can’t resume a run exactly.
The recorded trace in the figure is our small dense classifier run’s constant learning rate; the step, warm-up, and cosine panels are illustrations. The markers show where that run saved its parameters, and only those states can be reopened.
Clipping and decay with small numbers
Gradient (3,4) has norm √(9+16)=5. A norm cap of 1 scales it by 1/5 to (0.6,0.8), keeping its direction and cutting its magnitude. For decoupled decay alone, w=1, η=1/2, and decay coefficient 1/2 give w_new=(1−1/4)×1=3/4.
Clipping is not evidence that the unmodified gradient was wrong. It is a deliberate restriction on the update input. Decay likewise changes the optimization recipe and leaves the forward pass alone. A run’s settings should record whether either was enabled, and whether stored gradient norms were taken before or after clipping.
Replay a recorded loop
The replay below uses our small dense classifier run, which trains full-batch. Its batch is its whole training set, so every update consumes one epoch’s worth of examples.
See it move
Start paused at initialization. Step once and compare a parameter’s old and new values. The gradients shown for an update are pre-update batch gradients, while the displayed loss and parameter state are post-update. That timing tells you which gradient produced each change.
Select a sample in the logits view and look at its input and probability shares. Then move the transport cursor. The scalar loss, the parameter snapshot, and the available sample values follow the recorded progress. Panels saved less often show the step they come from.
The run loader also accepts a local trainkit run.json. The file stays in the browser and is checked before use. Missing fields show as not recorded, and measured zeros show as zeros.
Where this shows up when you train
The loop’s stages repeat across model families. Their input structures and forward calculations differ, and they all still need an objective, derivatives, an update rule, and progress records. Seeing that shared structure makes a convolutional network or a sequence model a new forward recipe inside a familiar loop rather than a separate ritual.
A run is more than a final weight file. Its trace says how the parameters changed, which examples were evaluated, and which states were saved. Next: read the shapes of those traces, and compare a falling loss with the task score and the checkpoint that was kept.
What you now know
- A step is one update, while an epoch measures examples consumed.
- Momentum and Adam retain state that affects later updates.
- Recorded predictions and weights must be read with their step conventions.
Where we’re headed
Next: what these measurements look like across recorded training runs. Continue to the next chapter.