Part II · Ch. 10 — Backwards: How the Gradient Reaches Every Layer

Part II · Chapter 10 of 29

Backwards: How the Gradient Reaches Every Layer

Trace the chain rule through the model


Use the stored forward values

The last layer’s gradients were short to calculate because its weights touch the logits directly. How does the loss reach a weight farther back? Reuse the stored forward values and trace every returning contribution. The arithmetic gives one derivative beside every parameter, including the ones whose derivatives come out zero.

Restore the original model from chapter 7. We are not starting from the output layer that chapter 9 already updated. The input is x=(1,0), the target is class 1, and the hidden values are (t,t,0). Resetting the starting state makes this a whole-model update, comparable with the earlier last-layer-only one.

Forward arrows carry x, pre-activation a, hidden h, logits z, and loss L. Backward arrows carry (q, −q), sum shared paths, multiply by 1 − h squared, and multiply by the input.
Backward arithmetic reuses the values stored by the forward pass.

A computational graph connects operations by their data dependencies. The chain rule multiplies local derivatives along a path and adds contributions where paths meet. Backpropagation applies that rule backward through the graph. It reuses forward values already computed, so every operation has what it needs to calculate its local slope.

Treat the returning number as a sensitivity: how much the final loss would change for a small change in that intermediate value. A multiplication, an addition, or an activation turns that sensitivity into contributions for its own inputs. Nothing is sending a desired replacement activation backward. The calculation asks how the existing computation responds to change.

A chain before a branch

Take a teaching chain whose local multipliers are 2, 1, and 3. Starting from incoming derivative 1 at the end, the return calculation is 1×3×1×2=6. Each link scales the sensitivity it receives. Walking the graph in reverse doesn’t reverse the meaning of the forward calculation.

A branch adds one more step. If an intermediate value reaches the loss by two outgoing paths, changing it disturbs both. Its total derivative has to include both effects. Drop one outgoing connection and you compute the gradient of a different graph, however consistent the remaining arithmetic looks.

Bring both output contributions back

Use t=tanh(1)≈0.7616 and q=p₀≈0.3183. The logit derivatives are (q,−q). To return through the output matrix, collect every outgoing contribution to each hidden unit. The weights that multiplied hidden values on the way forward now multiply the returning score sensitivities.

Add both paths at each hidden unit

For hidden unit one, 1q+0(−q)=q. For unit two, −1q+1(−q)=−2q. For unit three, 0q+1(−q)=−q. So the hidden gradient is (q,−2q,−q)≈(0.3183,−0.6366,−0.3183). The middle unit gets two nonzero contributions and has to add them.

Let h_j name hidden unit j, W₂,cj its outgoing weight to class c, and p_c−y_c the returning logit sensitivity. Summing over c collects all the class paths. The result has one entry per hidden unit, because those are the intermediate values whose sensitivities we want.

$$ \frac{\partial L}{\partial h_j}=\sum_c W_{2,cj}(p_c-y_c) $$

In words: add the loss contributions returning through every outgoing connection.

Tanh’s slope at a hidden output h is 1−h². For hidden values (t,t,0), those slopes are about (0.4200,0.4200,1). Multiply coordinatewise by the hidden gradient to get the preactivation gradient: about (0.1337,−0.2674,−0.3183). The third hidden value is zero, and tanh’s slope there is one.

A seventeen-row table pairs each parameter with its current weight and gradient. The second column of W1 and third column of W2 have zero gradients for this example.
Every parameter’s gradient for this example, zeros included.

Let a_j name the hidden preactivation, the weighted sum and bias before tanh. Its incoming weight for input coordinate i appears in the product W₁,ji x_i, so the local multiplier is x_i. Each input-weight gradient is the preactivation gradient times the input that feeds it. Its bias gradient uses multiplier one.

$$ \frac{\partial L}{\partial W_{1,ji}}=\frac{\partial L}{\partial a_j}(x_i),\qquad a_j=\sum_iW_{1,ji}x_i+b_{1,j} $$

In words: multiply the hidden preactivation’s slope by the input that feeds this weight.

For x=(1,0), every second-column input-weight gradient is zero. That happens because the second input is zero, not because the second column is missing from the model. A different example can produce nonzero gradients there. A zero gradient on one example doesn’t show that a parameter never matters.

Put one gradient beside every parameter

The derivatives line up with the original parameter arrays. The W₁ gradient has three rows and two columns, like W₁. The b₁ gradient has three entries, like b₁. The same correspondence holds for W₂ and b₂.

The complete gradient inventory

To four decimal places, the W₁ gradients are [[0.1337,0],[−0.2674,0],[−0.3183,0]]. The b₁ gradients are [0.1337,−0.2674,−0.3183]. The W₂ gradients are [[0.2424,0.2424,0],[−0.2424,−0.2424,0]]. The b₂ gradients are [0.3183,−0.3183].

An upstream gradient is the sensitivity arriving from later operations. A local gradient is one operation’s derivative with respect to an input. Multiply them and you get the contribution passed toward that input. The directional names follow the computation graph. No data move backward through time.

Autograd is software that records operations and applies the same chain rule. It saves you expanding every derivative by hand, and it differentiates the graph you wrote. A mistaken input, a missing dependency, or a wrong loss definition gives a perfectly consistent derivative of the wrong calculation.

See it move

Finish the forward pass, then select Trace backward. Choose the middle hidden unit’s input weight and look at its two outgoing paths. Their products add before the tanh slope and the input multiplier go on. Select a zero-gradient entry and find which factor caused the zero. Both should agree with the full table above.

The tracer also nudges the selected parameter on either side of its current value and recomputes the loss. That centered finite difference is an independent numerical check. Its last few digits can differ, because the nudge and the floating-point arithmetic are finite, and the sign and scale should agree away from difficult numerical limits.

Move the parameters together

Every derivative above came from the original parameter state. Now take learning rate η, eta, equal to 0.1 and update all the parameters from that state at once. Don’t update a hidden weight, recompute a downstream derivative on the changed model, and call the result one simultaneous step. That mixes two different starting states.

For all seventeen parameters, table columns show the original value, the change −0.1 times its slope, and the updated value. Recomputing after all updates gives a loss of about 0.3085 nats.
All updates use the same original parameter state.

Apply every parameter update

To four decimal places, W₁ becomes [[0.9866,0],[0.0267,1],[1.0318,−1]], and b₁ becomes [−0.0134,1.0267,−0.9682]. W₂ becomes [[0.9758,−1.0242,0],[0.0242,1.0242,1]], and b₂ becomes [−0.0318,0.0318]. Each new value is its original value minus 0.1 times its own derivative.

The new preactivation is about (0.9733,1.0535,0.0637). Applying tanh gives hidden values of about (0.7501,0.7832,0.0636). The third hidden value is nonzero now, because its input weight and bias moved. That change can reach the output through its existing outgoing connections.

Recompute the complete pass

The new logits are about (−0.1020,0.9157). The class-1 probability is about 0.7345 and its loss is about 0.3085 nats. Compare that with the original 0.3832 and the last-layer-only result 0.3414. All three start from the same example, so the comparison is fair.

The chain rule reaches every parameter by reusing one forward computation.

Backpropagation and the optimizer do different jobs here. Backpropagation computes the derivative table. The optimizer uses that table, the learning rate, and whatever state it keeps to choose new parameter values. You can swap the optimizer without touching the chain rule that produced the gradient.

Where this shows up when you train

For a new layer or loss, a small numerical example is the debugging tool worth reaching for. Keep inputs, targets, and initial parameters fixed. Compare the forward values first, then the gradient arrays, then the simultaneous update. That order separates a wrong forward recipe from a wrong derivative or a wrong update.

The model now has a complete, inspectable learning step: forward values, loss, returning sensitivities, updated parameters. Repeating that step over batches brings in progress counters, data ordering, and optimizer memory. Those are the pieces a long training trace needs before it can be read.

What you now know

  • The chain rule multiplies local slopes along each path.
  • Contributions add when several paths reach a value.
  • Backpropagation computes the derivatives, and the optimizer changes parameters.

Where we’re headed

One complete update is done. Next: many updates, organized into batches and epochs. Continue to the next chapter.