Use the stored forward values
The last layer’s gradients were short to calculate because its weights touch the logits directly. How does the loss reach a weight farther back? Reuse the stored forward values and trace every returning contribution. The arithmetic gives one derivative beside every parameter, including the ones whose derivatives come out zero.
Restore the original model from chapter 7. We are not starting from the output layer that chapter 9 already updated. The input is x=(1,0), the target is class 1, and the hidden values are (t,t,0). Resetting the starting state makes this a whole-model update, comparable with the earlier last-layer-only one.
A computational graph connects operations by their data dependencies. The chain rule multiplies local derivatives along a path and adds contributions where paths meet. Backpropagation applies that rule backward through the graph. It reuses forward values already computed, so every operation has what it needs to calculate its local slope.
Treat the returning number as a sensitivity: how much the final loss would change for a small change in that intermediate value. A multiplication, an addition, or an activation turns that sensitivity into contributions for its own inputs. Nothing is sending a desired replacement activation backward. The calculation asks how the existing computation responds to change.
A chain before a branch
Take a teaching chain whose local multipliers are 2, 1, and 3. Starting from incoming derivative 1 at the end, the return calculation is 1×3×1×2=6. Each link scales the sensitivity it receives. Walking the graph in reverse doesn’t reverse the meaning of the forward calculation.
A branch adds one more step. If an intermediate value reaches the loss by two outgoing paths, changing it disturbs both. Its total derivative has to include both effects. Drop one outgoing connection and you compute the gradient of a different graph, however consistent the remaining arithmetic looks.
Put one gradient beside every parameter
The derivatives line up with the original parameter arrays. The W₁ gradient has three rows and two columns, like W₁. The b₁ gradient has three entries, like b₁. The same correspondence holds for W₂ and b₂.
The complete gradient inventory
To four decimal places, the W₁ gradients are [[0.1337,0],[−0.2674,0],[−0.3183,0]]. The b₁ gradients are [0.1337,−0.2674,−0.3183]. The W₂ gradients are [[0.2424,0.2424,0],[−0.2424,−0.2424,0]]. The b₂ gradients are [0.3183,−0.3183].
An upstream gradient is the sensitivity arriving from later operations. A local gradient is one operation’s derivative with respect to an input. Multiply them and you get the contribution passed toward that input. The directional names follow the computation graph. No data move backward through time.
Autograd is software that records operations and applies the same chain rule. It saves you expanding every derivative by hand, and it differentiates the graph you wrote. A mistaken input, a missing dependency, or a wrong loss definition gives a perfectly consistent derivative of the wrong calculation.
See it move
Finish the forward pass, then select Trace backward. Choose the middle hidden unit’s input weight and look at its two outgoing paths. Their products add before the tanh slope and the input multiplier go on. Select a zero-gradient entry and find which factor caused the zero. Both should agree with the full table above.
The tracer also nudges the selected parameter on either side of its current value and recomputes the loss. That centered finite difference is an independent numerical check. Its last few digits can differ, because the nudge and the floating-point arithmetic are finite, and the sign and scale should agree away from difficult numerical limits.
Move the parameters together
Every derivative above came from the original parameter state. Now take learning rate η, eta, equal to 0.1 and update all the parameters from that state at once. Don’t update a hidden weight, recompute a downstream derivative on the changed model, and call the result one simultaneous step. That mixes two different starting states.
Apply every parameter update
To four decimal places, W₁ becomes [[0.9866,0],[0.0267,1],[1.0318,−1]], and b₁ becomes [−0.0134,1.0267,−0.9682]. W₂ becomes [[0.9758,−1.0242,0],[0.0242,1.0242,1]], and b₂ becomes [−0.0318,0.0318]. Each new value is its original value minus 0.1 times its own derivative.
The new preactivation is about (0.9733,1.0535,0.0637). Applying tanh gives hidden values of about (0.7501,0.7832,0.0636). The third hidden value is nonzero now, because its input weight and bias moved. That change can reach the output through its existing outgoing connections.
Recompute the complete pass
The new logits are about (−0.1020,0.9157). The class-1 probability is about 0.7345 and its loss is about 0.3085 nats. Compare that with the original 0.3832 and the last-layer-only result 0.3414. All three start from the same example, so the comparison is fair.
Backpropagation and the optimizer do different jobs here. Backpropagation computes the derivative table. The optimizer uses that table, the learning rate, and whatever state it keeps to choose new parameter values. You can swap the optimizer without touching the chain rule that produced the gradient.
Where this shows up when you train
For a new layer or loss, a small numerical example is the debugging tool worth reaching for. Keep inputs, targets, and initial parameters fixed. Compare the forward values first, then the gradient arrays, then the simultaneous update. That order separates a wrong forward recipe from a wrong derivative or a wrong update.
The model now has a complete, inspectable learning step: forward values, loss, returning sensitivities, updated parameters. Repeating that step over batches brings in progress counters, data ordering, and optimizer memory. Those are the pieces a long training trace needs before it can be read.
What you now know
- The chain rule multiplies local slopes along each path.
- Contributions add when several paths reach a value.
- Backpropagation computes the derivatives, and the optimizer changes parameters.
Where we’re headed
One complete update is done. Next: many updates, organized into batches and epochs. Continue to the next chapter.