The mountain is real now
A loss landscape assigns height $L(\theta)$ to parameters $\theta$, “theta.” Gradient descent uses the local tilt. Let's move beyond a single smooth bowl.
A one-dial mountain
This invented loss has one parameter w and two valleys.
In words: the loss at dial-setting $w$. Plug in a number, get a height.
Slopes of added terms add; a constant multiplier scales the slope. The derivative of w⁴ is 4w³; the constant 3 contributes zero. So:
$$ \frac{dL}{dw} = 0.4w^3 - 2w + 0.2 $$In words: the tilt of the ground at dial-setting $w$. Positive means the loss climbs to the right, negative means it falls to the right.
Check the slope before trusting the walk
At w=3, the derivative is $0.4\times27-6+0.2=5.0$. Direct evaluation gives L(3)=2.70 and L(3.01)≈2.7504. The difference divided by 0.01 is about 5.04, approaching 5 as the nudge shrinks.
A global minimum is lowest overall. A local minimum is lowest nearby.
| Feature | Position | Loss |
|---|---|---|
| Deep valley | About −2.28 | About 0.05 |
| Shallow valley | About 2.18 | About 0.94 |
| Ridge | About 0.10 | About 3.01 |
Walking it
The learning rate η, “eta,” is 0.1 for this walk.
$$ w_{\text{new}} = w - \eta \, \frac{dL}{dw} $$In words: nudge the dial a small step against the tilt of the loss. The learning rate $\eta$, "eta," sets how big that step is.
Worked example
Step 1, from $w = 3$:
- slope $= 0.4 \cdot 27 - 6 + 0.2 = 10.8 - 6 + 0.2 = 5.0$
- step $= -0.1 \times 5.0 = -0.5$
- position: $3 \to 2.5$; loss: $L = 2.70 \to 1.16$
Step 2, from $w = 2.5$:
- slope $= 0.4 \cdot 15.625 - 5 + 0.2 = 6.25 - 5 + 0.2 = 1.45$
- step $= -0.1 \times 1.45 = -0.145$
- position: $2.5 \to 2.355$; loss: $L = 1.16 \to 1.00$
The pattern is set, so we summarize the walk in a table:
| step | $w$ | slope | $L$ |
|---|---|---|---|
| 0 | $3.00$ | $5.00$ | $2.70$ |
| 1 | $2.50$ | $1.45$ | $1.16$ |
| 2 | $2.36$ | $0.71$ | $1.00$ |
| 3 | $2.28$ | $0.40$ | $0.96$ |
| 4 | $2.24$ | $0.23$ | $0.95$ |
In words: each step is smaller than the last, because the ground flattens as we near the floor. The walk settles onto the valley floor at $w \approx 2.18$, $L \approx 0.94$.
This small-step trajectory from w=3 settles in the shallow valley. Starting at −1 instead gives slope $-0.4+2+0.2=1.8$, step −0.18, and next position −1.18. It heads toward the deeper valley. Starting position matters.
The gradient is a local downhill direction; a finite step can cross a ridge or even increase loss.
The learning-rate craft, on real terrain
| η from w=3 | What happens |
|---|---|
| 0.01 | Still near w=2.20 after 100 steps: a costly crawl. |
| 0.1 | Settles near the shallow floor. |
| 0.6 | First step: 3−0.6×5=0. Then 0 → −0.12 → −0.38 → −0.95 → about −2; later swings fail to settle. |
| 1.0 | 3 → −2 → −3 → 1.6 → 2.96 and growing instability; loss can spike dramatically. |
A learning-rate schedule changes η over time. The schedule can start gently, take larger steps, then shrink them. In the widget, switch from 0.6 to 0.1 to settle after the initial crossing.
Two dials, then billions
The gradient ∇L stacks slopes. Subtract η times each component from its matching parameter.
In words: nudge every one of the billions of dials a small step against its own slope of the loss. It is the same rule, run once per dial, all at once.
A saddle differs from a minimum: some directions still lead down. Large networks often have many useful solutions and saddle-like regions, but this is not a guarantee of equal outcomes across random starts. We seek a useful solution under a compute budget.
Schedules and optimizer memory
Annealing names gradual cooling or its algorithmic analog. The related training idea is to reduce the step size as training progresses. A fixed rate keeps steps the same size; a decaying rate makes later steps smaller.
| Rule | What it keeps | How it helps |
|---|---|---|
| Plain descent | Current gradient. | A direct local step. |
| Momentum | A running direction. | Can smooth alternating steps across a narrow valley. |
| Adam | Gradient and squared-gradient averages. | Adapts the scale coordinate by coordinate. |
An optimizer implements that choice. Data sampling adds noise too; it may help exploration but does not guarantee escape from a poor region. Fine-tuning starts this process from learned weights rather than a random initialization.
See it move
Replay the toy landscape. Change one setting at a time; reset before comparing starting positions.
Why the LLM cares
$$ \theta_{\text{new}} = \theta - \eta \, \nabla L $$In words: step every dial against its slope of the loss. Chapter 12 built $L$; this chapter walked it.
Each step needs the slope with respect to every parameter. Finite differences would require many full reruns. Backpropagation computes those derivatives together by reusing the forward computation.
Practical optimizers add memory and scaling to this basic descent rule. Their efficiency still depends on getting the gradient affordably.
What you now know
- Loss assigns a height to every parameter setting.
- Local and global minima need not coincide.
- Initialization and finite step size affect which region a run reaches.
- A large step can increase loss or diverge.
- Schedules and optimizer state change how gradients become updates.
- Training aims for useful low loss, not a certified global minimum.
Where we're headed
We know how to use slopes. Next we'll compute them by passing derivatives backward through the model.