Teaching the Network

The two problems we left open

Last chapter we built a network of plain straight-line neurons that, stacked a few layers deep, can bend into any shape, and we watched gradient descent nudge one toward a good boundary. But we cheated twice, and both cheats have to be paid off before anything real can be trained. This chapter pays them.

The first debt is mechanical. A real network has thousands, millions, sometimes billions of weights buried inside its hidden layers, and when it gets a training point wrong, every one of them needs to know which way to turn. Gradient descent from fitting a line runs on the gradient — one slope per weight — and we never said how to get all those slopes without re-running the whole network once per weight, which at real scale is impossible. The second debt is subtler and more important: a network powerful enough to fit any shape is powerful enough to fit the wrong shape — to trace every training point exactly, noise and all, and then fail on everything it has never seen.

So the chapter has two halves. First the mechanics: backpropagation, which grades every buried weight for the price of two passes, then better ways to take the downhill step — momentum and Adam. Then the wisdom: overfitting, the train/validation/test split, and the regularizers that keep a network honest. The first half is arithmetic; the second is judgement — and it is where most of the real craft of machine learning lives.

How blame reaches a buried weight

The engine is the chain rule, built in full — slowly, with a nudge-check — on the sister site; for the patient version, read the LLM site's backpropagation chapter. In one sentence: when machines feed machines, a small wiggle in an early weight reaches the output multiplied by the slope of every stage in between. So to find how the loss responds to a buried weight, multiply the local slopes along the path from that weight down to the loss.

Work it on the smallest honest network there is, using the exact numbers the sister site uses — so a reader crossing between the two sites meets one consistent example. Two weights in series: an input $x = 2$ flows into a first weight $w_1$, giving a middle value $h = w_1 x$; that $h$ flows into a second weight $w_2$, giving the prediction $\hat{y} = w_2 h$ — "y-hat, the model's guess". The target is $y = 3$. Set $w_1 = 1$ and $w_2 = 2$.

Now the payoff, said with force: nobody re-ran the network to get those two slopes. One forward pass computed and stored every value; one backward pass, reusing exactly those stored values, computed every weight's blame — everyone upstream reuses the blame that has already flowed to the value below them. Two passes grade all the weights. Two, or two billion. That reuse is the entire reason training a deep network is affordable; without it you would re-run the whole model once per weight, which at real scale is never. Forward, grade, backward, step — that four-beat cycle, repeated a few million times, is training.

Better ways downhill

Plain gradient descent takes one step straight down the local slope, every step, with no memory of where it has been. On the smooth bowls of the math toolkit that is fine. On the crumpled loss landscapes of real deep networks it struggles in two ways this section fixes: it zigzags helplessly across long narrow valleys, and it crawls to a near-stop across vast flat plateaus.

First, an upgrade that changes what "the gradient" even means. Grading the loss over the full training set at every step is unaffordable, so each step grades a small random batch of examples instead. A batch's gradient is only a noisy estimate of the true one — a compass that jitters around downhill rather than pointing at it exactly. That is stochastic gradient descent, or SGD, and it is what everyone actually runs. The jitter is not just tolerated but useful, rattling the walker out of shallow dead-ends the way a little shaking settles sand into a tighter pack.

Second — momentum. Instead of stepping on the raw slope, keep a running average of recent gradients — a velocity $\mathbf{v}$ — and step on that:

In words: build up speed in directions the slope keeps pointing, and let disagreeing wiggles cancel out — then step along that accumulated velocity, not the raw gradient.

The new symbol $\beta$ — "beta" — is the momentum coefficient, usually about $0.9$: how much of the old velocity carries into the next step. Why $0.9$ and not something else — because keeping nine-tenths of your speed each step means it takes about ten steps of steady pushing to turn the walk around, which is long enough to smooth out random wobble and short enough to still respond when the ground genuinely changes. Picture a ball rolling downhill instead of a hiker re-deciding at every footstep. It powers through plateaus because it arrives carrying speed, and it stops zigzagging across a narrow valley because the sideways slaps alternate and cancel while the gentle downhill push points the same way every step and accumulates.

Third — Adam, the one you will actually meet in code. Its idea in a sentence: give every weight its own learning rate, automatically. A weight whose gradient has been consistently large gets smaller, calmer steps; one whose gradient has been tiny and rare gets larger, bolder ones — so no single global $\eta$ has to suit every weight at once. It does this by tracking, per weight, both a running average of the gradient (that is momentum again) and a running average of its square — a measure of how jumpy that weight has been — and dividing each step by the square root of the latter:

step $\;\propto\;$ (average gradient) $/\;\sqrt{\text{average squared gradient}}$

That is Adam. A thrashing weight has a large average square, so it gets divided down into small steps; a barely-moving weight is divided by almost nothing and is free to take a big one. It usually "just works" with very little tuning, which is exactly why it dominates.

Land it with a fair summary, no hype. SGD is the honest baseline; momentum smooths and accelerates it; Adam adapts per weight and is the sensible default for most deep networks — yet plain SGD with momentum still wins some of the largest, most carefully tuned training runs on Earth. The optimizer is a knob, not a magic wand. Every one of them is still just walking downhill on the same loss.

The real enemy: memorizing

Now the harder half, and it opens with a scene. Imagine seven scattered data points that clearly come from a gentle underlying trend plus a little measurement noise. A straight line through them misses the bend entirely — too simple to capture the real pattern, with high error even on the data it trained on. We say it underfits. A gentle degree-3 curve — degree being the dial for how much a curve is allowed to bend, a degree-3 curve getting up to two bends — follows the trend and shrugs off the scatter, a good fit. But a wild degree-9 curve, allowed eight bends' worth of freedom, can be forced through all seven points exactly, wiggling violently between them and shooting off near the edges. It overfits: perfect on the training points, nonsense on any new one.

Here is the distinction the whole rest of machine learning turns on. Getting the training data right is not the goal. Generalization — doing well on data it has never seen — is. A model can have near-zero training error and be worthless, because it memorized the exam instead of learning the subject. And here is the trap specific to neural networks: last chapter's universal-approximation power cuts both ways. A big enough network can memorize its entire training set perfectly, noise and all — it is the degree-9 curve with a million dials instead of nine.

To manage this you first have to see it, so make memorizing measurable. Split your data: train on one part, and keep a held-out part the model never learns from, used only to check how it does on the unseen. Now plot both losses as training proceeds, measured in epochs — one epoch is one complete pass over the training set. The training loss falls and keeps falling — the model gets better and better at the questions it can see. The held-out loss falls too at first — real pattern being learned — then bottoms out and starts rising, as the model begins memorizing training noise that does not transfer. That rising curve is overfitting made visible, and the growing gap between the two curves is exactly the memorization.

That gives us the sweet spot and the cheapest fix. The moment the held-out loss stops falling is the best this model will ever generalize; training past it makes things strictly worse. So stop there. Early stopping is exactly that — watch the held-out curve and quit at its minimum. It is the cheapest and most-used regularizer there is.

Three piles of data

Turn that honest-measurement habit into the discipline every practitioner follows: before doing anything else, split the data into three piles. The training set is what the model actually fits its weights to. The validation set is what you check during development — to pick the learning rate, choose the network size, decide when to early-stop. And the test set is locked in a drawer and looked at exactly once, at the very end, for the true grade.

Why three and not two? The reason is subtle and almost universally botched. Every time you use the validation set to make a choice, you leak a little information about it into the model. Tune long enough against validation and you start overfitting to it — your validation score becomes optimistic, a grade you have quietly studied for. The test set catches exactly that: because you never once used it to make a decision, its number is trustworthy. State the rule plainly, and tape it to the wall: the moment you make any choice based on the test set, it stops being a test set.

Two practical notes. A common split is something like 80/10/10, but the real constraint is not the ratio: all three piles must look like the world the model will actually face — the same kinds of examples, with no training point sneaking into the test pile in disguise. And when data is genuinely scarce, cross-validation — rotating which slice is held out and averaging the results — squeezes more signal from the same points.

Keeping a network honest

Early stopping was our first regularizer. There is a whole family of them, and they share one idea: make memorizing harder, so the network is nudged toward simple, general patterns instead of intricate, memorized ones. A regularizer deliberately hobbles the model's ability to memorize, and that small handicap usually buys a large gain on new data.

First, weight decay, also called L2. Add a penalty to the loss for large weights: instead of minimizing $L$ alone, minimize $L$ plus a small multiple of the total squared size of the weights:

In words: pay the usual loss, plus a fee that grows with how big the weights get — so the network keeps them small unless a big weight really earns its keep. The double bars are the vector length from the toolkit, so $\lVert \mathbf{w} \rVert^2$ is that length squared: every weight squared and added up.

The new symbol $\lambda$ — "lambda" — is the strength dial: crank it up and the fee bites harder. Why does this help? Small weights make smooth, gentle functions — the tame degree-3 curve. Big weights are what make the violent wiggles of overfitting possible. Weight decay is a thumb on the scale toward smoothness.

Second, dropout — the one regularizer unique to neural networks, and delightfully strange. During training, at every step, randomly switch off a fraction of the neurons (say half), setting their outputs to zero for that step only. The network must still predict correctly with a random half of it missing, so it cannot lean on any single neuron as a memorized lookup — it is forced to spread each pattern redundantly across many. At test time you switch every neuron back on. The effect is like cheaply training a huge ensemble of slightly different networks that share one set of weights — robustness for almost no cost.

Third, gather the two you already met: early stopping (quit at the validation minimum) and, quietly the most powerful of all, more data. Here is the deepest, least glamorous truth in the chapter: the single most reliable cure for overfitting is a bigger, more varied training set. A model cannot memorize its way through a dataset too large to memorize — it is forced to find the actual pattern. Every regularizer above is, in part, a substitute for data you did not have.

Close on the balance, with no false precision. Regularize too little and the network memorizes; regularize too much and you handcuff it into underfitting — the straight line again, this time by choice. The right amount is found the way everything else here is: by watching the validation curve. There is no formula, only measurement.

Where you'll meet this

You now own the full training loop, and it is a ritual you can recite for any neural network in the rest of this site. Split the data three ways. Forward a batch through the network. Grade it with a loss. Backpropagate the blame to every weight in two passes. Step with an optimizer — SGD, momentum, or Adam. Watch the validation curve. Regularize and early-stop so the network generalizes instead of memorizing. Every model in Part III — the convolutional networks, the recurrent networks, the graph networks, and the transformers on the sister site — is trained by this exact loop. Only the network's shape changes.

Follow the thread to the sister site: pretraining a large language model is this same loop run at industrial scale — the same backprop, the same Adam-family optimizer, the same falling loss — poured over trillions of tokens (the LLM site's training-run chapter shows it happening). The reader who understands overfitting on seven points understands, in miniature, why frontier models are trained on more data than any human could read in a thousand lifetimes: enough that memorizing it is not an option.

And name honestly what is still ahead. We now know how to train a network and keep it honest — but we have been feeding it plain flat lists of numbers, every input treated as unrelated to its neighbors. Real data has shape: an image is a grid, a sentence is a sequence, a contact network is a graph. Pour any of those into a plain stack of neurons and you throw their structure away before the network ever sees it. The next part of the book matches the network's wiring to the data's shape — and the gains are enormous.

Close warm. You have hand-graded a two-weight network ($\nabla L = [8, 4]$, loss $1 \to 0.71$), watched a wiggly curve memorize seven points, and learned the discipline — three piles, one honest test — that separates a model that works from one that only looks like it works. That discipline, more than any architecture, is what makes machine learning a science instead of a magic trick.