Standing on a hillside, in fog
Imagine finding a valley in thick fog. You can feel the ground's tilt but cannot see the whole hill, so you take a downhill step, measure again, and repeat. The rest of this chapter is that rule, written as arithmetic.
Slope: the line case first
Slope is rise over run. The symbol $\Delta$, “delta,” means “change in.” For the line $f(x)=2x+1$, every input step of one raises the output by two.
Worked example
Walk the line from $x = 1$ to $x = 3$. The output goes from $f(1) = 3$ to $f(3) = 7$. Slope is the change in output divided by the change in input:
$$ \text{slope} = \frac{\Delta f}{\Delta x} = \frac{7 - 3}{3 - 1} = \frac{4}{2} = 2 $$In words: the output rose 4 while the input rose 2, so the line climbs 2 per unit, and it does so everywhere.
Positive slope rises to the right; negative slope falls. Its magnitude tells us how steeply. A straight line has the same slope everywhere.
The derivative: slope where you stand
Now use $f(x)=x^2$. A secant measures an average slope between two points. Slide the second point toward the first:
Worked example
Anchor at $(1, 1)$. Pick a second point on the curve and compute rise over run; then bring it in.
| second point | rise / run | measured slope |
|---|---|---|
| $(2,\ 4)$ | $(4-1)/(2-1)$ | $3$ |
| $(1.5,\ 2.25)$ | $(2.25-1)/0.5$ | $2.5$ |
| $(1.1,\ 1.21)$ | $(1.21-1)/0.1$ | $2.1$ |
| $(1.01,\ 1.0201)$ | $(1.0201-1)/0.01$ | $2.01$ |
In words: as the second point creeps toward $x = 1$, the readings settle toward 2. That settling value is the slope at $x = 1$.
The limiting slope is the derivative, written $df/dx$, “the derivative of f with respect to x.” A tangent shows that slope. We approach zero separation; we never divide by zero.
$$ \frac{df}{dx} = 2x $$In words: at position $x$, the parabola's local slope is twice $x$.
At x=3, the secant to 3.1 gives $(9.61-9)/0.1=6.1$, approaching 6. For $e^x$ at x=1, $(2.7456-2.7183)/0.01\approx2.73$, approaching its height $e\approx2.72$. That is the property promised last chapter.
| Point (x, f(x)) | Slope | Downhill direction |
|---|---|---|
| (−2, 4) | −4 | Right |
| (0, 0) | 0 | At the minimum |
| (3, 9) | +6 | Left |
On our bowl, zero slope marks the bottom. On other curves it can mark a peak or a flat passing point, so zero slope alone does not prove we found a minimum.
The downhill rule
Gradient descent steps opposite the slope. The learning rate $\eta$, “eta,” sets the step scale.
$$ x_{\text{new}} = x - \eta \, \frac{df}{dx} $$In words: new position equals old position, minus the learning rate times the local slope.
Worked example
Descend $f(x) = x^2$ starting at $x = 4$, with $\eta = 0.25$. Step 1, in full:
- slope $= 2 \cdot 4 = 8$
- step $= -0.25 \cdot 8 = -2$
- position: $4 \to 2$; height: $f = 16 \to 4$
Step 2, also in full:
- slope $= 2 \cdot 2 = 4$
- step $= -0.25 \cdot 4 = -1$
- position: $2 \to 1$; height: $f = 4 \to 1$
From here the pattern is clear, so we summarize: each step halves $x$. The walk reads $4,\ 2,\ 1,\ 0.5,\ 0.25,\ \dots$, closing on the valley at $0$.
| η | Update | Behavior |
|---|---|---|
| 0.01 | x becomes 0.98x. | After ten steps, about 3.27: slow. |
| 0.25 | x becomes 0.5x. | 4 → 2 → 1 → 0.5. |
| 1 | x becomes −x. | 4 → −4 → 4: endless bouncing. |
| 1.1 | x becomes −1.2x. | 4 → −4.8 → 5.76 → −6.91: growing swings. |
At a fixed learning rate, small slopes produce small steps. A large rate can still leap past the low ground.
Many dimensions: the gradient
Use the bowl $f(x,y)=x^2+2y^2$. At (1,1), its height is 3. A partial derivative measures one coordinate direction at a time. Read $\partial f/\partial x$ as “the partial derivative of f with respect to x.”
Worked example
Compute each partial two ways at $(1, 1)$.
With respect to $x$: hold $y$ frozen. Then the $2y^2$ part is a constant along that walk, and we already know $x^2$ has slope $2x$, so $\frac{\partial f}{\partial x} = 2x = 2$. Nudge check: $f(1.1, 1) = 1.21 + 2 = 3.21$, a rise of $0.21$ over a run of $0.1$ → slope $\approx 2.1$ ✓.
With respect to $y$: hold $x$ frozen. The $2y^2$ part is $y^2$ made twice as steep, so $\frac{\partial f}{\partial y} = 4y = 4$. Nudge check: $f(1, 1.1) = 1 + 2.42 = 3.42$ → slope $\approx 4.2$ ✓.
$$ \frac{\partial f}{\partial x} = 2x = 2 \qquad \frac{\partial f}{\partial y} = 4y = 4 \qquad \text{at } (x, y) = (1, 1) $$In words: walking in the $x$ direction the ground tilts up at slope 2; walking in the $y$ direction it tilts up more steeply, at slope 4.
The gradient stacks those slopes into a vector. The symbol $\nabla f$ reads “the gradient of f.”
![A bird's-eye view of the bowl x squared plus 2 y squared as nested muted level rings. At the point (1,1) on the middle ring, an arrow labeled gradient [2,4] points perpendicular to the ring, straight uphill; an arrow points the exact opposite way, the step direction.](media/slopes-and-descent/gradient-compass.svg)
In words: the gradient of $f$ is the arrow whose components are the two slope readings.
The gradient points steepest uphill, perpendicular to a level curve. Its negative points downhill. With η=0.25, subtract the scaled gradient:
Worked example
From $(1, 1)$ with gradient $(2, 4)$:
$$ (1,\ 1) - 0.25 \cdot (2,\ 4) = (1 - 0.5,\ 1 - 1) = (0.5,\ 0) $$In words: take a quarter of each gradient component and subtract it from the matching coordinate, $1 - 0.5$ across and $1 - 1$ up, landing at $(0.5, 0)$. The height drops from $3$ to $f(0.5, 0) = 0.25$. One more step, summarized: the gradient at $(0.5, 0)$ is $(1, 0)$, giving $(0.25, 0)$ and height $0.0625$. Two steps, height $3 \to 0.0625$.
| Inputs | Slope information | Update |
|---|---|---|
| One | One derivative. | Subtract η times that slope. |
| Two | Two partial derivatives. | Subtract the matching component from each coordinate. |
| Billions | One gradient entry per parameter. | The same arithmetic in every slot. |
For a general function we write ∇f; when the function is a loss L, we write ∇L.
See it move
Drag the starting point, choose η, then press Step or Auto. Read the arithmetic alongside the trail.
Why the LLM cares
A model's parameters $\theta$, “theta,” are the position. Its loss L is the height. Plain gradient descent updates them as follows:
$$ \theta_{\text{new}} = \theta - \eta \, \nabla L $$In words: nudge every parameter a small step against its own slope of the loss.
Part III builds the loss, explores practical update rules, and shows how one backward pass computes all those slopes.
What you now know
- A derivative measures local slope.
- Shrinking secants approach the tangent's slope.
- Gradient descent subtracts learning rate times gradient.
- An excessive learning rate can cause bouncing or divergence.
- The gradient holds one slope per input and points uphill.
- Zero slope alone does not guarantee a minimum.
Where we're headed
One foundation remains: probability. We'll spread belief across possible next words and learn how a model chooses one.