Rolling Downhill

The mountain is real now

Chapter 5 taught a walk on a made-up hill. Chapter 12 quietly built a real one: every complete setting of the model's dials $\theta$ — theta, the whole collection of weights from Chapter 3 — is a place, and every place has a height, the exam score $L(\theta)$. Nobody drew this landscape and nobody can see it whole; it exists the way a mountain range exists at night. Training is the art of walking it downhill by feel.

The object has a name: the loss landscape. One dial gives you a curve. Two dials give you a rolling surface over a plane. Billions of dials give you a landscape in billions of dimensions that no one will ever picture — and, as Chapter 5 promised, the arithmetic does not care how many dimensions there are.

Here is the plan. We will walk a one-dial landscape end to end by hand, meet the terrain features that real training fights — valleys inside valleys, ridges, spikes — and close with the analogy this whole site was secretly built around. Because finding low ground on a rugged landscape is, more or less, what my entire field does for a living.

A one-dial mountain

Let me be honest about the data before we start: there is none. To keep every number checkable, our machine has a single dial $w$ and a loss curve I made up for teaching — small enough to hand-check, lumpy enough to be honest:

In words: the loss at dial-setting $w$. Plug in a number, get a height.

We also need the tilt of the ground — the slope of that curve at each $w$. Chapter 5 gave us the one pattern we need: $x^2$ has slope $2x$. The same shrinking-secant game gives $x^4$ a slope of $4x^3$, and a straight-line term like $0.2w$ has the constant slope $0.2$ (a line's tilt never changes). Two bits of bookkeeping let us put the pieces together, and both are exactly what you would guess.

First, slopes add. A curve built by adding pieces has, at every point, the sum of the pieces' slopes — each term tilts the total on its own, and they do not interfere. Second, a constant out front scales the slope. The term $0.1w^4$ climbs one-tenth as fast as $w^4$ does, so its slope is one-tenth as big: $0.1 \times 4w^3 = 0.4w^3$. The minus sign on $-w^2$ is that same rule with a constant of $-1$, so that term's slope is $-2w$. And the bare $+3$ lifts the whole curve upward without tilting it anywhere, so it contributes no slope at all. Add the four pieces:

In words: the tilt of the ground at dial-setting $w$ — positive means the loss climbs to the right, negative means it falls to the right.

Do not take that on faith — check it the way Chapter 5 taught. At $w = 3$ the formula claims a tilt of $0.4 \cdot 27 - 6 + 0.2 = 5.0$. Now nudge the dial by hand: $L(3) = 2.70$ and $L(3.01) = 2.7504$, so the ground rose $0.0504$ over a run of $0.01$, a measured tilt of about $5.04$. Shrink the nudge further and that reading walks in toward $5.0$ exactly. The formula told the truth.

Now look at the terrain before we take a single step — picture before symbols.

The figure shows two valleys. A shallower one on the right, its floor near $w \approx 2.18$ at height $L \approx 0.94$. A deeper one on the left, its floor near $w \approx -2.28$ at height $L \approx 0.05$. Between them sits a ridge crest near $w \approx 0.10$ at height $L \approx 3.01$. You can verify any single point by plugging in — the crest, for instance: $L(0.1) = 0.00001 - 0.01 + 0.02 + 3 = 3.01$.

The two valleys deserve their names. The deep one is the global minimum — the best the whole landscape has to offer. The shallow one is a local minimum — locally a success, globally a consolation prize. Chapter 5's parabola had one bottom, so the distinction never came up. Real landscapes are made of little else.

Walking it

The walking rule is Chapter 5's, with fresh letters — same rule, real mountain:

In words: nudge the dial a small step against the tilt of the loss. Here $\eta$ — eta, the learning rate — sets how big that step is.

Let us walk, starting on the right-hand slope at $w = 3$, with $\eta = 0.1$. Step 1, in full:

Success — the walk found a valley and settled in. And now the rug: our walker is standing in the wrong valley. Across the ridge, the floor at $w \approx -2.28$ scores $0.05$, nearly twenty times better, and our walker will never find it. Why? Gradient descent only ever steps downhill, and from $w = 3$ every downhill road leads right. Worse: the walker is already below the ridge crest — its loss $0.94$ is under the crest's $3.01$ — so no sequence of honest downhill steps can ever climb back over. The fog never lifts, as Chapter 5 warned, and now the fog is hiding something that matters.

It could have gone differently. Start the identical walk at $w = -1$ instead. One step, in full: the slope there is $0.4 \cdot (-1) - 2 \cdot (-1) + 0.2 = -0.4 + 2 + 0.2 = 1.8$; the step is $-0.1 \times 1.8 = -0.18$; so $w \to -1.18$ — moving left, away from zero, downhill into the deep valley. Summarized, it glides to $w \approx -2.28$, $L \approx 0.05$. Same rule, same $\eta$, same landscape — a different starting square, and a twenty-fold better ending. Where you start matters. And real training starts $\theta$ at random values, which is worth one raised eyebrow before the next section explains why it is not fatal.

The learning-rate craft, on real terrain

Chapter 5 showed the $\eta$ pathologies on a parabola, where the worst that could happen was bouncing in place. On lumpy terrain the stakes climb — and one of the outcomes genuinely surprises. Let us walk the gallery, all starts from $w = 3$; the widget below lets you replay every one.

$\eta = 0.01$, the crawl. After one hundred steps the walker sits at $w \approx 2.20$ — still edging toward the local floor. Nothing failed. But at scale this is the pathology that costs the most money: Chapter 15's runs price every step in GPU-time, and a crawl finishes the walk long after the budget is gone.

$\eta = 1.0$, the rampage. The walker ricochets between the slopes — $w$: $3 \to -2.0 \to -3.0 \to 1.6 \to 2.96 \to -1.71 \to -3.33 \to 4.6 \to -25.4 \to \dots$ — and explodes off the landscape, the loss lurching $2.7 \to 0.2 \to 1.5 \to 1.4 \to 2.5 \to 0.6 \to 3.6 \to 27.6 \to$ thousands. Two lessons inside one disaster. First, the loss does not fall steadily — it spikes up and down, and when Chapter 15 shows real loss curves with spikes, this is what a spike is. Second, one oversized step can undo a hundred good ones.

$\eta = 0.6$, the shock. The first step is $-0.6 \times 5.0 = -3.0$, vaulting the walker from $w = 3$ clear over the ridge to $w = 0.0$ — and from there it rolls left: $0 \to -0.12 \to -0.38 \to -0.95 \to -2.0 \to \dots$ into the deep valley our careful $\eta = 0.1$ walker could never reach. A reckless step size escaped the trap. But watch the ending: near $w \approx -2.28$ the deep valley's walls are too steep for a step that big, and the walker bounces around the floor without ever settling. Big steps explore; small steps land; no single $\eta$ does both.

That last sentence is the whole point, so let us name the fix. Real training runs change $\eta$ over time — a learning-rate schedule. Start careful, stride boldly through the early terrain, then shrink the steps so the walk can settle. You are about to perform this exact maneuver by hand in the widget below: vault with $0.6$, then land with $0.1$.

Two dials, then billions

Lift to two dials to reactivate Chapter 5's gradient. Now the loss is $L(w_1, w_2)$, and the tilt is two readings stacked into one arrow — the gradient $\nabla L$. The walk subtracts $\eta$ times that arrow, component by component. No new arithmetic — this is exactly Chapter 5's $(1,1) \to (0.5, 0)$ move, on lumpier ground.

Now the honest jump. A real model has billions of dials, so the walk happens in a space with billions of directions. Two things are worth knowing, and I will hedge both as what researchers observe, not as theorems. First: in enormous dimensions, dead-stop traps are rarer than the two-dimensional cartoons suggest. At most flat-ish points, some of the billions of directions still tilt downhill — such a point is a saddle, not a true bottom — so the walk usually finds a way to keep leaking downhill. Second: rugged landscapes hold astronomically many valleys that are all roughly equally good, so training twice from different random starts lands in different valleys of similar quality. That is why "a good valley, not the best one" is a livable creed rather than a disappointment.

In words: nudge every one of the billions of dials a small step against its own slope of the loss — the same rule, run per dial, all at once.

And one deflating sentence to close the loop with Chapter 5: through all of this, the algorithm never got smarter. It is still exactly $w - \eta \cdot \text{slope}$, run per dial, in fog. Everything this chapter added was terrain.

The annealing mountain

Now the analogy this site was built around, and it gets a full section because it is genuinely the same mathematics. Materials science has an energy landscape. Every arrangement of the atoms in a solid has an energy; nature relentlessly nudges systems toward lower-energy arrangements; the map from arrangement to energy is a loss landscape with different labels. Metal cooling after a weld, water freezing into ice, a jostled pile of gravel settling — all of them are walkers on rugged terrain, seeking low ground.

The local-minimum story is a materials engineer's daily bread, and it is told most cleanly with glass and quartz. Cool molten silica fast and the atoms get frozen mid-jumble: glass — a material permanently stuck in a local minimum, beautiful and slightly wrong. Cool the same silica slowly and the atoms have time to find the deep, ordered arrangement: quartz crystal, the global basin. Identical ingredients; the difference is entirely the schedule of the cooling. Metallurgists and glassmakers have a word for that slow, staged cooling: annealing.

The map is term for term: atomic arrangement $\leftrightarrow$ the dial settings $\theta$; energy $\leftrightarrow$ the loss $L(\theta)$; thermal jiggling $\leftrightarrow$ big early steps and noise; the cooling schedule $\leftrightarrow$ the learning-rate schedule; glass $\leftrightarrow$ a run that quenched into a poor valley; crystal $\leftrightarrow$ a run that annealed into a deep one. The $\eta = 0.6$-then-$0.1$ maneuver from the last section is annealing, performed by hand.

One forward seed, one sentence: Chapter 15 adds a genuine source of jiggle to the walk — the gradient there is measured on random handfuls of data, so it shivers — and that shiver, like thermal noise, helps shake walkers out of shallow dips. The full story of that noise belongs to the training run.

See it move

The descent-stepper returns from Chapter 5, on new terrain. The curve is now a red loss landscape with two valleys, and the walker is a dial-setting $w$. Everything this chapter claimed is replayable: drag the walker, pick a learning rate, and press Step or Auto ▸. Switching $\eta$ mid-run is allowed and encouraged — it is the annealing exercise.

Why the LLM cares

State the equation of the whole enterprise once more, now with every symbol owned:

In words: step every dial against its slope of the loss. Chapter 12 built $L$; this chapter walked it.

The $\eta$ craft — schedules, spikes, escapes — is a real job description on real training teams, because at scale a bad learning-rate day costs more than a car. But this chapter has been living on an enormous IOU. Every step needs $\nabla L$: the tilt of the loss with respect to every one of billions of dials, refreshed at every single step. Measuring one tilt the shrinking-secant way means re-running the whole model. Measuring billions that way means billions of re-runs per step. At that price, deep learning would not exist.

There is a trick — one of the great tricks in the history of computing — that gets every one of those tilts for roughly the cost of two runs. It even has a wonderfully accusatory name. And you now hold the full loop conceptually: score the guess (Chapter 12), feel the tilt (next chapter), step downhill (this one), repeat. Everything after that is scale.

Where we're headed

One promise is still unpaid, and it is enormous. Every step of the walk consumed a slope reading — and a real model needs billions of them, one per dial, recomputed at every single step. Measure them the shrinking-secant way and each reading costs a full run of the model: billions of runs per step, forever. At that price, none of this would exist. The actual trick gets every one of those billions of slopes for roughly the cost of two runs — one pass forward through the machine, one pass backward carrying blame. It is the reason deep learning is possible at all, and you already own every mathematical part of it. Next chapter: backpropagation.