Part II · Ch. 9 — Downhill: the Gradient

Part II · Chapter 9 of 29

Downhill: the Gradient

Fit a line, one visible update at a time


Which direction lowers the loss?

We have a loss, a fixed model, and a target. Which weight should move, and which way? Measure the local effect of a small change, collect those effects into an arrow, and take one checkable update. Then recompute the loss rather than assuming the update helped.

A derivative is the local rate of change of an output with one input. A partial derivative varies one parameter and holds the others fixed. A gradient collects those partial derivatives into a list with one entry per parameter. In a two-weight picture that list draws as an arrow in weight space.

Left: loss contours on weight-1 and weight-2 axes with opposite uphill-gradient and downhill arrows. Right: L = (w − 1) squared versus w; losses 1.21 and 0.81 give a central slope estimate of −2.
The gradient points uphill locally; the update moves in the opposite direction.

The gradient points uphill locally: it gives the direction of steepest increase under the usual Euclidean distance convention. Its negative points downhill. Neither arrow knows anything about distant points on the surface. The direction describes the neighborhood of the current weights, so it changes as the weights move.

A local minimum is no higher than nearby points. A saddle has rising and falling directions. A plateau has small slopes across a region. Those shapes explain why a local update slows down or behaves differently along different directions. They don’t turn one two-dimensional picture into a map of a larger model.

Estimate a slope by two nudges

For the teaching slice L(w)=(w−1)², evaluate near w=0. L(0.1)=0.81 and L(−0.1)=1.21. The centered finite difference is [0.81−1.21]/0.2=−2. The negative sign says a small increase in w lowers the loss here. This slice is a different objective from the two-weight surface in the last chapter.

A finite difference estimates a slope from evaluated changes instead of symbolic differentiation. The centered version looks on both sides of the point. It is good for checking an analytic calculation, though rounding makes very tiny nudges unreliable. We use it later as an independent check on every parameter in the small model.

Follow loss back to one output weight

Back to the fixed model’s original parameters, input x=(1,0), and target class 1. The hidden values are (t,t,0), where t=0.761594. Let q=p₀=0.318300. The two probabilities sum to one, so the difference between class 1’s probability and its target value is p₁−1=−q.

From loss to one output weight

For softmax followed by cross-entropy, the loss derivatives with respect to logits are dL/dz=(p₀,p₁−1)=(q,−q). The first logit depends on W₂₁₁ through hidden value h₁=t. So dz₀/dW₂₁₁=t, and dL/dW₂₁₁=q×t=0.242416. The matching weight in the second output row has gradient −q×t.

The 2 by 3 output-weight gradient has rows (0.242416, 0.242416, 0) and (−0.242416, −0.242416, 0). The bias gradients are (0.3183, −0.3183); the zero hidden input produces the zero third column.
Each output weight receives the score error multiplied by its hidden input.

The full W₂ gradient is [[0.242416,0.242416,0],[−0.242416,−0.242416,0]]. The b₂ gradient is [0.318300,−0.318300]. The two third-column entries are zero because the hidden input h₃ is zero. Nothing was left out; zero is their computed derivative for this example.

Let c select a class and j select a hidden input. The output score’s error signal is p_c−y_c, and its multiplier with respect to that weight is h_j. Multiply them and you have one entry of the weight gradient. The bias derivative carries the same score signal, because the bias enters its score with multiplier one.

$$ \frac{\partial L}{\partial W_{2,cj}}=(p_c-y_c)h_j $$

In words: the output score’s error signal times the hidden input gives that weight’s slope.

Changing one logit changes every probability through softmax’s shared denominator. The compact p−y result already accounts for that coupling once cross-entropy is differentiated. Treating the probabilities as independent outputs of separate score functions would drop part of the calculation. The derivative follows the recipe that produced the loss.

Take a step and check the result

The learning rate η, pronounced eta, sets the scale of the update. Gradient descent subtracts the learning rate times the gradient. For this demonstration take η=0.1 and change only the output layer. The hidden weights and biases stay at the original initialization, so the last-layer gradients act alone.

Update the output layer together

W₂ becomes [[0.975758,−1.024242,0],[0.024242,1.024242,1]]. The new b₂ is [−0.031830,0.031830]. Every new entry uses its own original value and its own gradient. For instance, 1−0.1×0.242416=0.975758.

Let w stand for one parameter and let the partial derivative name its slope. The left arrow means replace the stored value with the expression on the right. For a full update, apply the same rule to every selected parameter, using gradients computed at the same original model state.

$$ w\leftarrow w-\eta\frac{\partial L}{\partial w} $$

In words: subtract the learning rate times the slope from the current weight.

Check the prediction after the step

The hidden values are unchanged. The new logits are (−0.068754,0.830349). Softmax gives p₁=0.710765, and the class-1 loss becomes 0.341413 nats. The old loss was 0.383166 nats. The probability on the true class rose and the loss fell.

A small step that helped doesn’t prove a larger step would work better. The derivative describes local behavior. Move far enough and you can cross the valley and climb the other side, or land in a region with a different slope. So it helps to look at paths from different update settings on the same objective.

Three optimizer paths overlay loss contours with weight 1 horizontally and weight 2 vertically. Each panel labels its optimizer, learning rate, and how the run ended.
Different step rules and rates produce different paths on the same objective.

These paths come from training the two-weight linear model of the last chapter three times. The starting weights and the objective are the same each time; the optimizer and the learning rate differ.

See it move

Scrub the main path first. Then pick one of the comparisons. The sandbox computes its own path from the same inputs and targets, rounded, so its numbers can differ slightly from the original run. Change its learning rate and momentum and watch the path change.

The nudge readout shows the centered finite-difference calculation at the selected point beside the analytic gradient. Check both coordinate directions. The contour and rotated surface use the same stored loss grid, so rotating changes your view and not the values. Keep the uphill gradient and the downhill update direction apart.

Run a line fit yourself

Now train a model small enough to print every parameter and derivative. This one fits y=ax+b to twelve points using two parameters, a and b. The data and the program come in the trainkit download described in Train Your Own Model. It fits on the same points it is scored on, so it demonstrates optimization and says nothing about held-out performance.

A gradient measures local change and a checked update can reduce loss.
python3 -m trainkit fit trainkit/samples/fit.csv --output fit-output

Our run finished at a=1.9855, b=1.0000, and loss 0.000083 after sixty steps. Read the printed changes as applications of the update rule: a slope, a chosen rate, and a new value whose prediction you can check.

A smaller in-sample loss says the fit improved on those points, not that it behaves well anywhere else.

Where this shows up when you train

A finite-difference nudge is a good diagnostic when you implement a new calculation. Try several nudge sizes and compare signs and magnitudes with the analytic derivative. Too small a nudge disappears into rounding; too large a nudge measures behavior over a wide interval instead of a local slope.

Only the last layer has moved so far. The hidden layer has parameters too, and their effect reaches the loss through several operations. Next: carry the same error signals backward through those operations, adding all the returning paths, until each of the seventeen parameters has its own derivative.

What you now know

  • A gradient is a list of local slopes.
  • Gradient descent subtracts a scaled gradient.
  • Recomputing the loss checks what the update did.

Where we’re headed

The output layer is accounted for. Next: the same error signal through the hidden activation to every earlier parameter. Continue to the next chapter.