Slopes and Descent

Standing on a hillside, in fog

Picture yourself somewhere on a vast hillside at night. The fog is so thick you cannot see three steps ahead — no summit, no horizon, and certainly no valley. Yet you need to reach the valley floor. What can you actually use? One thing: the tilt of the ground under your boots. You can feel which way is downhill. So you take a step that way, stop, feel the ground again, and step again. Repeat.

That fog-walk is the whole of this chapter, and it has exactly two jobs. First, make "feel the tilt" precise — that is the derivative. Second, make "step downhill" precise — that is an update rule with a step size we will call the learning rate $\eta$. Here is the stake: this same fog-walk, repeated a few million times, is what the word training means in Part III. Every model you have ever used learned by walking downhill in fog. Let us learn to walk.

Slope: the line case first

Before curves, a warm-up on the friendliest shape there is. Back in Chapter 4 we met the line $f(x) = 2x + 1$. "Feeling the tilt" of a line has a plain name: slope — rise over run, meaning how much the output changes per unit of input. To talk about a change, we need one new symbol: $\Delta$, capital delta, read "the change in." So $\Delta f$ is the change in the output and $\Delta x$ is the change in the input. That is the only new symbol in this section, and we will use it sparingly.

Two facts to carry forward. The sign of a slope tells you direction: a positive slope climbs as you go left-to-right, a negative slope falls. The size tells you steepness: bigger number, harder climb. And here is the special thing about a line — it has one slope for its entire length. Measure it between any two points you like and you always get the same 2. That sameness is precisely what curves do not have, which is the problem the next section exists to solve.

The derivative: slope where you stand

Bring back Chapter 4's parabola. From here on this chapter calls it $f(x) = x^2$ — same curve, fresh letter. On a curve, "the slope" is not one number: it depends on where you stand. Near the bottom the ground is gentle; out on the arms it is steep. So we have to ask a local question: what is the slope exactly at $x = 1$, the point $(1, 1)$?

We cannot use two far-apart points the way we did for the line — that would average the tilt over a stretch of curve. Instead we measure between $(1,1)$ and a second point, then slide that second point closer and closer, watching what the measurement settles toward.

Now we name it. The derivative is that settled slope at a single point. We write it $\frac{df}{dx}$, read "the derivative of $f$ with respect to $x$." It is a slope measurement and nothing more exotic than that: subtraction and division, refined until the two points touch.

And there is a shortcut, which we will state as a pattern rather than prove. For $f(x) = x^2$, the derivative at any position $x$ is:

In words: at position $x$, the parabola's local slope is twice $x$.

Check it against what we just measured: at $x = 1$ the pattern gives $2 \cdot 1 = 2$ ✓. Check one more point with the same shrinking trick: at $x = 3$, the secant from $(3, 9)$ to $(3.1, 9.61)$ gives $(9.61 - 9)/0.1 = 6.1$, settling toward $2 \cdot 3 = 6$ ✓.

One more curve deserves its slope reading, because Chapter 4 left an IOU here. It claimed that $e^x$ climbs at a rate exactly equal to its own current height, and promised to make that precise once we had the derivative. We have it now, so measure $e^x$ the same shrinking way at $x = 1$. The height there is $e^1 \approx 2.72$. Step forward by $0.01$: a calculator gives $e^{1.01} \approx 2.7456$, so the secant slope is $(2.7456 - 2.7183)/0.01 \approx 2.73$ — the curve's own height back again, give or take the rounding crumb, and the crumb shrinks as the step does. That is the teaser made precise: at every point on $e^x$, the steepness equals the height. No other base does that, and that one property is the whole reason $e$ is mathematics' favorite.

Read the derivative as a compass across the whole curve. At $x = -2$ it is $2 \cdot (-2) = -4$: the ground falls to the right, so downhill is to the right. At $x = 3$ it is $+6$: downhill is to the left. And at $x = 0$ it is $0$ — flat. Flat is not boring; flat is where valleys bottom out. "Derivative equals zero" is the mathematical spelling of "you have arrived."

The downhill rule

Now turn the compass into a walk. If the derivative is positive, the ground climbs to the right, so downhill means decrease $x$. If the derivative is negative, downhill means increase $x$. Both cases collapse into one instruction: step opposite the derivative. Written out:

In words: new position equals old position, minus the learning rate times the local slope.

The Greek letter $\eta$ — eta — is the learning rate: the number that sets how big each step is. The minus sign is what makes the walk go down — we move against the tilt, not with it.

Notice a built-in grace: nobody shrank the steps on purpose. Near the bottom the slope is small, so $\eta$ times the slope is small — careful little steps right where you want them, for free. Far from the bottom the slope is large, so the steps are bold. The walk is cautious exactly where caution matters.

So what happens if the step is too big? Watch three pathologies on the same start, $x = 4$.

• $\eta = 1$ — the endless bounce. The rule becomes $x \leftarrow x - 1 \cdot (2x) = -x$. The walker jumps $4 \to -4 \to 4 \to -4$ forever: each stride flies clear across the valley to the equal-height point opposite. It never lands.
• $\eta = 1.1$ — worse. Now $x \leftarrow -1.2x$, giving $4 \to -4.8 \to 5.76 \to -6.91 \to \dots$. The swings grow; the walk explodes off the hill.
• $\eta = 0.01$ — too small. Now $x \leftarrow 0.98x$, so after ten whole steps you have crawled from $4$ to only about $3.27$. Perfectly safe — and you will be walking for a very long time.

Picking $\eta$ is a genuine craft. Chapter 13 wrestles with it on real, lumpy loss landscapes, where the choice has real consequences.

Many dimensions: the gradient

Real models do not have one dial — they have billions. So take the smallest honest step up from one input to many: a machine with two inputs, $f(x, y) = x^2 + 2y^2$. Its graph is a bowl-shaped valley sitting over the plane. We stand at the point $(x, y) = (1, 1)$, where $f = 1 + 2 = 3$.

How do we feel the tilt when there are two directions to walk? We freeze one at a time. The partial derivative $\frac{\partial f}{\partial x}$ — read "the partial derivative of $f$ with respect to $x$," where the curly $\partial$ signals that the other inputs are held still — is the slope you feel walking purely in the $x$ direction.

Now stack those two slope readings into a single object — the payoff of Chapter 1, where we learned that a list of numbers is an arrow:

In words: the gradient of $f$ is the arrow whose components are the two slope readings.

We call this arrow the gradient, written $\nabla f$ and read "the gradient of $f$." Here is the fact that makes it useful (the figure makes it credible without proof): this arrow points in the steepest uphill direction, and it stands perpendicular to the level rings of the valley. So the steepest downhill direction is exactly $-\nabla f$ — the same arrow, flipped.

The downhill rule survives intact: subtract $\eta$ times the gradient from the position, component by component. With $\eta = 0.25$:

Now the scale-up. Two inputs made a 2-component gradient. A model with two billion dials makes a 2-billion-component gradient — one slope reading per dial, all stacked into one unimaginably long arrow. You cannot picture it, and you do not need to: the arithmetic per component is exactly what you just did.

One notation handshake before we move on: this chapter wrote $\nabla f$ for a generic machine $f$. When the machine being descended is the loss $L$ (Part III), the same arrow is written $\nabla L$ — same object, specific mountain.

See it move

Now take the controls yourself. The descent-stepper below puts you in the walker's boots on $f(x) = x^2$. Drag the dot anywhere on the hill and feel the tangent tilt with it; pick a learning rate; then press Step for one update or Auto ▸ to watch the walk unfold. A fading trail records where you have been, and the status line spells out the arithmetic of every step.

Why the LLM cares

Here is the reveal, stated plainly: training a neural network is this chapter. The model's parameters $\theta$ — theta, the collective name for every weight in every matrix from Chapter 3 — are the coordinates, the walker's position. A single number called the loss $L$ — "how wrong were we?", built properly in Chapter 12 — is the height. And learning is fog-walking that landscape downhill.

Which means the one line that trains everything is a line you can already read:

In words: nudge every parameter a small step against its own slope of the loss.

Every component of $\theta$ — all billions of them — gets exactly the treatment our $x$ and $y$ just got. That is the entire idea. What is left is Part III filling in the pieces, one chapter each:

• Chapter 12 builds the mountain: what the loss $L$ actually measures, and why it is written with a natural log.
• Chapter 13 does this walk on real, lumpy landscapes, where the $\eta$ pathologies you just saw have real costs.
• Chapter 14 answers the question this chapter quietly raised — how could anyone possibly compute two billion partial derivatives quickly? That trick is backpropagation, and it is why deep learning works at all.

So take a breath and notice what just happened: you have personally executed the algorithm that spent millions of GPU-hours creating every model you have ever chatted with. Same rule. Bigger hill.

Where we're headed

One tool remains in Part I. When an LLM writes, it does not pick the next word — it spreads belief across thousands of candidates, weighs them, and then rolls a very opinionated die. Spreading belief and choosing from it is the mathematics of probability, and it is the native language of everything the model says. That is Chapter 6 — the last stop before we open the machine itself.