Blame Flows Backward

The impossible bill

Each training step needs a loss slope for every parameter. Nudging each weight separately would require a full rerun per weight. Backpropagation gets all those slopes with a forward pass and a backward pass. We'll perform both with a pencil.

Machines in a chain

Recall $u=2x+1$, then $z=u^2$. At x=3, u=7 and z=49. The first machine's slope is 2; the second's at u=7 is 14.

Local slopes multiply backward.

The chain rule multiplies those sensitivities along the chain.

Trace a small input change

$$ \frac{dz}{dx} = \frac{dz}{du} \cdot \frac{du}{dx} = 14 \times 2 = 28 $$

In words: the slope through a chain is the product of the local slopes along the way.

A small change first doubles, then is magnified about fourteenfold. The combined slope is 28.

A network you can hold

Our toy network has input x, two weights w₁ and w₂, a middle value h, and prediction $\hat y$, “y-hat.” This time it predicts a number, so use squared error instead of next-token cross-entropy.

Two weights map x=2 to h=2 and prediction 4; target 3 gives squared loss 1. — The forward pass gives prediction 4, loss 1.

$$ h = w_1 x \qquad \hat{y} = w_2 h \qquad L = (\hat{y} - y)^2 $$

In words: multiply the input by the first dial to get the middle value, multiply that by the second dial to get the guess, then score the guess by how far it lands from the target $y$, squared.

A forward pass uses x=2, w₁=1, w₂=2, target y=3.

$$ h = 2, \quad \hat{y} = 4, \quad L = (4 - 3)^2 = 1 $$

In words: two becomes two, then four. That is one too high against the target of three, so the loss is one.

Changing the loss changes the first derivative entering the backward chain. The method of passing derivatives through the rest of the network stays the same.

The backward pass starts at the loss. “Blame” means sensitivity, not an instruction giving each weight its correct value.

Each branch multiplies incoming blame.

One local derivative per backward hop
Destination	Calculation	Derivative
Prediction	2 × (4−3)	2
w₂	Incoming 2 × h=2	4
h	Incoming 2 × w₂=2	4
w₁	Incoming 4 × x=2	8

Collect the weight slopes

$$ \frac{\partial L}{\partial w_2} = \frac{dL}{d\hat{y}} \cdot \frac{\partial \hat{y}}{\partial w_2} = 2 \times 2 = 4 \qquad \frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial h} \cdot \frac{\partial h}{\partial w_1} = 4 \times 2 = 8 $$

In words: each dial's blame is the blame arriving from downstream times how strongly that dial moves its own output.

$$ \nabla L = \begin{bmatrix} 8 \\ 4 \end{bmatrix} $$

In words: the loss is twice as sensitive to the inner dial $w_1$ as to the outer one $w_2$.

The component order is w₁, then w₂. The doubling pattern comes from our chosen values; different local slopes need not double anything.

Use the gradient, then check the loss

The learning rate η, “eta,” is 0.05. Update both weights using derivatives from the same forward pass:

$$ w_1 \leftarrow 1 - 0.05 \times 8 = 0.6 \qquad w_2 \leftarrow 2 - 0.05 \times 4 = 1.8 $$

In words: move each dial a little way in the downhill direction its blame points.

Now h=0.6×2=1.2 and prediction=1.8×1.2=2.16. The miss is −0.84, so loss is 0.7056, about 0.71, down from 1.

Watch (1:30): what to notice. The red blame pulse starts at the loss and travels right to left, dropping $4$ into $w_2$ and $8$ into $w_1$ as it passes. Then one step turns the dials and the loss falls $1 \to 0.71$. Two dials or two billion, it is the same two passes.

Why two passes are enough

A computation graph records what depends on what. Backprop computes each intermediate derivative once and shares it upstream. If several paths reach the same value, their derivative contributions add.

Two panels. Left, 'the nudge way': a stack of bars each reading 'wiggle dial N → re-run everything', for dials 1, 2, 3, an ellipsis, and dial 2,000,000,000, with a total 'bill: 2,000,000,000 full runs — per step'. Right, 'the backprop way': two bars, one -accented 'one forward pass — keep the sticky notes' and one -accented 'one backward pass — blame visits every dial once', with a total 'bill: ≈ 2 runs — at ANY size'. Between them: 'same gradient, either way'. — Save forward values for the backward pass.

Compare the work needed for a scalar loss
Method	Work	Reuse
Finite differences	A baseline plus roughly one forward evaluation per dial.	Repeatedly recomputes shared downstream work.
Backpropagation	One forward and one backward traversal.	Accumulates and shares local derivatives.

Two passes are not constant-cost: both become more expensive with a larger model. The advantage is a small multiple of forward work, instead of multiplying that bill by the number of weights again.

Nonlinear activations contribute their local derivatives like any other operation. The backward pass needs forward values, such as the h=2 we used above, so training stores activations or recomputes some to trade time for memory.

When blame fades or blows up

A vanishing gradient leaves early weights with a weak update signal. An exploding gradient can destabilize the update.

Two bar-chart panels of blame magnitude versus stages crossed, ticks 0 to 10, bars. Left, 'local slopes 0.5 — blame vanishes': bars fall from 1 at stage 0 to about 0.001 at stage 10, labeled '≈ 0.001', with a note 'early layers hear whispers'. Right, 'local slopes 1.5 — blame explodes': bars climb from 1 to about 58, labeled '≈ 58', with a note 'early layers get screamed at'. The two panels use different y-scales. — Small slopes fade; large slopes amplify.

Ten factors make a big difference

Ten factors of 0.5 give $0.5^{10}=1/1024\approx0.001$. Ten factors of 1.5 give $1.5^{10}\approx58$. The same chain rule can shrink or amplify a signal.

Initialization sets useful starting scales. Residual connections provide direct derivative paths. Normalization helps control activation scale, though it does not guarantee every local slope is near one.

Why the LLM cares

Forward: compute token predictions.
Grade: evaluate the observed tokens' loss.
Backward: accumulate derivatives for every trainable parameter.
Update: let the optimizer change those parameters.

The same derivative machinery reaches attention projections, MLPs, and embeddings. Backprop computes the gradient; the optimizer decides how to use it.

What you now know

Local derivatives multiply along a chain.
Contributions add where several paths meet.
Our toy's gradient is [8,4].
One update changes the weights to 0.6 and 1.8, lowering loss to about 0.71.
Reusing intermediate derivatives makes backprop efficient.
Long products can cause vanishing or exploding gradients.

Where we're headed

We have the complete update cycle. Next we'll repeat it over a training run and learn to read its progress.