Teaching the Network

The two problems we left open

We can build a network that bends a boundary, but two questions remain: how does each buried weight find downhill, and how do we keep the network from memorizing the examples?

We keep the gradient-descent rule from fitting a line. The distinction to hold on to through this chapter is between making the training loss fall and making a useful model.

How blame reaches a buried weight

The chain rule multiplies local slopes along a calculation. For its slow build and nudge-check, see the sister site's backpropagation chapter; we use its same numbers.

Input $x=2$ meets $w_1=1$, giving $h=w_1x$; then $w_2=2$ gives $\hat y=w_2h$, “y-hat,” our prediction; target $y=3$.

In words: the input stays at $h = 2$ through the first weight, doubles again into $\hat{y} = 4$, and since the target was $3$ the prediction is one too high, for a squared loss of $1$.

The curly $\partial$, read “partial,” means a slope while other weights stay fixed. Follow blame backward:

1. Loss: $\partial L/\partial\hat y=2(\hat y-y)=2$, using the toolkit’s square-slope rule.
2. Outer weight: $\partial\hat y/\partial w_2=h=2$; multiply $2\times2=4$.
3. Hidden value: $\partial\hat y/\partial h=w_2=2$; blame arrives as $2\times2=4$.
4. Inner weight: $\partial h/\partial w_1=x=2$; multiply $4\times2=8$.
5. Collect: in weight order $(w_1,w_2)$, the gradient is $[8,4]$.

In words: the loss is twice as sensitive to the inner weight as to the outer one, so the gradient, the vector of both slopes, is eight over four.

Let $\theta$ (“theta”) collect the weights and $\eta$ (“eta”) be the learning rate. Apply the update rule $\theta\leftarrow\theta-\eta\nabla L$ with $\eta=0.05$:

In words: nudge each weight a little against its own blame, re-run the network, and the loss falls from $1$ to about $0.71$ in one measured step.

Check: $2\times0.6=1.2=h$; $1.2\times1.8=2.16=\hat y$; the miss is $2.16-3=-0.84$, whose square is about $0.71$.

One forward pass stores values; one backward pass computes all weight slopes from them.

Better ways downhill

On the toolkit's bowl, plain downhill steps work well; narrow valleys and flat plateaus make the walk harder.

Picture a ball rolling downhill rather than a hiker re-deciding at every footstep. The ball crosses a plateau because it arrives carrying speed, and it stops zigzagging across a narrow valley because the sideways pushes alternate and cancel while the gentle downhill push points the same way every step and builds up.

1. SidewaysAlternating pushes cancel.
2. DownhillConsistent pushes accumulate.
3. PlateauArrive carrying speed.

For momentum, $\beta$ (“beta”) is the momentum coefficient, usually about $0.9$: nine-tenths of the previous velocity survives each step. Its memory spans roughly ten steps, smoothing wobble while still responding to change.

In words: build up speed in directions the slope keeps pointing, and let disagreeing wiggles cancel out. Then step along that accumulated velocity rather than the raw gradient.

Adam scales each weight's step in proportion to (average gradient) divided by the square root of (average squared gradient). Large squared gradients reduce the scale; smaller ones increase it. The learning rate $\eta$ still matters, and tuned SGD with momentum can win.

Memorizing instead of learning

Picture seven noisy points around a gentle trend. A straight line underfits: it misses the bend even on training data. A degree-3 curve allows up to two bends and follows the trend; a degree-9 curve allows up to eight and can pass through all seven points while swinging wildly between them. That curve overfits: it fits noise as though it were signal.

Generalization means performance on new data, which is our goal. A network's power to fit complicated shapes also gives it the capacity to memorize.

Hold out examples that never update the weights. Plot their loss alongside training loss over epochs, meaning complete passes through the training set. At first both fall; later, training loss keeps falling while held-out loss rises.

In the figure, validation loss bottoms near epoch 27 at about $0.30$, then rises toward $0.56$ while training loss approaches $0.05$. Early stopping keeps the model from the best validation checkpoint rather than chasing the lowest training loss.

Three piles of data

Before training, separate three jobs:

Each validation-guided choice leaks information into the design, making that score more optimistic. The untouched test set checks the whole selection process. Tune against it once and it stops measuring anything.

A common split is 80/10/10. What matters more is that all three piles represent the world the model will face, with no disguised duplicates crossing the split. With scarce data, cross-validation rotates the held-out slice and averages results.

Discouraging memorization

A regularizer discourages fitting noise, trading some training accuracy for better performance on new data.

For L2, $\lambda$ (“lambda”) controls the penalty strength and $\mathbf w$ collects the weights:

In words: pay the usual loss, plus a fee that grows with how big the weights get, so the network keeps them small unless a big weight earns its keep. The double bars are the vector length from the toolkit, so $\lVert \mathbf{w} \rVert^2$ is that length squared: every weight squared and added up.

Raising $\lambda$ makes the fee bite harder, favoring the gentle degree-3 curve over violent wiggles. Too much regularization underfits, so choose its strength using validation performance. The most reliable cure for overfitting is a bigger, more varied training set that represents the task, and every regularizer above is partly a substitute for data you did not have. More data makes accidental patterns harder to rely on; it does not make memorization impossible.

The whole training loop

The full training loop for every neural network in this book fits in three steps. The order matters: split before fitting, use validation for development, and save the untouched test set for the final grade.

1. FitForward → loss → backward → step; repeat on training batches.
2. ChooseUse validation to choose settings and the stopping point.
3. GradeTest once, after those choices are settled.

You have hand-graded a two-weight network ($\nabla L = [8, 4]$, loss $1 \to 0.71$), watched a wiggly curve memorize seven points, and seen the discipline of three piles and one final test.

Where you'll meet this