Backwards: How the Gradient Reaches Every Layer

Use the stored forward values

The last layer’s gradients were short to calculate because its weights touch the logits directly. How does the loss reach a weight farther back? We will reuse the stored forward values and trace every returning contribution. The arithmetic will give one derivative beside every parameter, including the entries whose derivatives are zero.

Restore the original model from chapter 7. We are not starting from the output layer already updated in chapter 9. The input is still x=(1,0), the target is class 1, and the hidden values are (t,t,0). Resetting the starting state makes this a complete-model update that can be compared with the earlier last-layer-only update.

A computational graph connects individual operations by their data dependencies. The chain rule multiplies local derivatives along a path and adds contributions where paths meet. Backpropagation applies this rule backward through the graph. It uses forward values that have already been computed, so each operation has the information needed to calculate its local slope.

Think of the returning number as a sensitivity: how much the final loss would change for a small change in that intermediate value. A multiplication, addition, or activation converts that sensitivity into contributions for its own inputs. We are not sending a desired replacement activation backward. We are calculating how the existing computation responds to changes.

A branch adds another step to this picture. If one intermediate value influences the loss by two outgoing paths, changing it perturbs both paths. Its total derivative must include both effects. Dropping one outgoing connection would compute the gradient of a different graph, even if the remaining arithmetic looked internally consistent.

Bring both output contributions back

Use t=tanh(1)=0.761594155956 and q=p₀=0.318300257805. The logit derivatives are (q,−q). To return through the output matrix, collect every outgoing contribution to each hidden unit. The same weights that multiplied hidden values in the forward pass now multiply the returning score sensitivities.

Let h_j name hidden unit j, W₂,cj its outgoing weight to class c, and p_c−y_c the returning logit sensitivity. Summing over c collects all class paths. The result has one entry per hidden unit because that is the set of intermediate values whose sensitivities we are now computing.

In words: add the loss contributions returning through every outgoing connection.

Tanh’s slope at a hidden output h is 1−h². For hidden values (t,t,0), those slopes are (0.419974341614,0.419974341614,1). Multiply coordinatewise by the hidden gradient to obtain the preactivation gradient: (0.133677941207,−0.267355882415,−0.318300257805). The third hidden value is zero, but tanh’s slope there is one.

Let a_j name the hidden preactivation, the weighted sum and bias before tanh. Its incoming weight for input coordinate i appears in the product W₁,ji x_i. The local multiplier is therefore x_i. Each input-weight gradient is the preactivation gradient multiplied by the corresponding input. Its bias gradient uses multiplier one.

In words: multiply the hidden preactivation’s slope by the input that feeds this weight.

For x=(1,0), every second-column input-weight gradient is zero. This happens because the second input is zero, not because the second column is absent from the model. A different example can produce nonzero gradients there. A zero gradient on one example does not establish that a parameter never matters.

Put one gradient beside every parameter

We can now align derivatives with the original parameter arrays. The W₁ gradient has three rows and two columns, just like W₁. The b₁ gradient has three entries, just like b₁. The same shape correspondence holds for W₂ and b₂. The backward pass does not invent a new parameter layout.

An upstream gradient is the sensitivity arriving from later operations. A local gradient is one operation’s derivative with respect to an input. Multiplying them produces the contribution passed toward that input. The directional names follow the computation graph; they do not mean that data are physically moving backward through time.

Autograd is software that records operations and applies this same chain rule. It relieves us of manually expanding every derivative, but it still differentiates the graph we actually wrote. A mistaken input, missing dependency, or wrong loss definition can produce a perfectly consistent derivative of the wrong calculation.

See it move

Complete the forward pass, then select Trace backward. Choose the middle hidden unit’s input weight and inspect its two outgoing paths. Their products add before the tanh slope and input multiplier are applied. Select a zero-gradient entry and check which factor caused the zero. Both actions should agree with the full table above.

The tracer also perturbs the selected parameter on either side of its current value and recomputes the loss. That centered finite difference supplies an independent numerical check. Its last few digits can differ because the nudge and floating-point arithmetic are finite, but the sign and scale should agree away from problematic numerical limits.

Move the parameters together

We have calculated every derivative at the original parameter state. Now choose learning rate η, eta, equal to 0.1 and update all parameters from that state together. Do not update a hidden weight, recompute a downstream derivative with the changed model, and call the result one simultaneous gradient step. That would mix different starting states.

The new preactivation is (0.973264411759,1.053471176483,0.063660051561). Applying tanh gives hidden values (0.750135303643,0.783152219713,0.063574194449). The third hidden value is now nonzero because its input weight and bias moved. That change can affect the output through its existing outgoing connections.

Backpropagation and the optimizer perform different jobs in this sequence. Backpropagation computes the derivative table. The optimizer uses that table, the learning rate, and possibly retained state to choose new parameter values. We can change the optimizer without changing the chain rule that computed the gradient.

Where this shows up when you train

For a new layer or loss, a small numerical example is a valuable debugging tool. Keep inputs, targets, and initial parameters fixed. Compare the forward values first, then the gradient arrays, then the simultaneous update. This separates a wrong forward recipe from a wrong derivative or a wrong update implementation.

The model now has a complete, inspectable learning step: forward values, loss, returning sensitivities, and updated parameters. Repeating that step over batches introduces progress counters, data ordering, and optimizer memory. Those are the next pieces we need before a long training trace becomes readable.

Where we’re headed

We have one complete update. Next we will organize many updates into batches and epochs. Continue to the next chapter.