Blame Flows Backward

The impossible bill

Chapter 13 ended on a debt it could not pay. The walk downhill needs the slope of the loss with respect to every dial — billions of numbers — refreshed at every single step. And the only slope-measuring tool we own is Chapter 5's honest nudge: wiggle one dial, re-run the entire model, watch how the loss moved. One nudge per dial means one full model run per dial — billions of complete runs for a single step. That is not slow. That is never.

So here is the claim, stated baldly so the payoff has a target to hit: there is a way to get every one of those billions of slopes for roughly the price of two runs — one forward, one backward. It is called backpropagation, it is arithmetic you already own, and it is the single trick that makes deep learning economically possible. Without it, none of the models you have ever used could have been trained.

We build to it in four moves. First a new tool — the chain rule, the last piece of calculus this site will ever ask of you. Then a network small enough to hold in one hand. Then the backward pass running over it, by pencil. Then the reason two passes suffice no matter how many dials there are.

Machines in a chain

Bring back the two machines from Chapter 4, chained one into the next — fresh letters for a fresh diagram. The first machine is the line $u = 2x + 1$; its output $u$ feeds the second machine, the parabola $z = u^2$. At $x = 3$ we get $u = 7$ and then $z = 49$ — exactly the $g(f(3)) = 49$ you computed back in Chapter 4, now drawn as a pipeline.

Now the question that unlocks everything. If $x$ wiggles a little, how much does $z$ wiggle at the far end? Walk the wiggle through. The first machine is a line with slope $2$, so a wiggle in $x$ comes out of it doubled — $\frac{du}{dx} = 2$. That doubled wiggle now enters the parabola, whose slope at $u = 7$ is $2u = 14$ (Chapter 5's $x^2 \to 2x$ pattern, read at the value actually passing through). So the arriving wiggle gets multiplied by $14$. Total magnification: $2 \times 14 = 28$.

That product has a name — the chain rule:

In words: the slope through a chain is the product of the local slopes along the way.

Nothing mystical arrived. Each factor is an ordinary Chapter 5 slope, measured at the value that is actually flowing through that machine — the $14$ is the parabola's steepness right at $u = 7$, the $2$ is the line's steepness everywhere. Multiply them and you have the whole chain's sensitivity.

Now reframe those same numbers as this chapter's central metaphor: run the arrow backward. Suppose $z$ at the end of the line came out wrong. How should the blame for that error be split among the machines that produced it? The $\times 14$ belongs to the parabola's steepness, the $\times 2$ to the line's. Blame enters at the output and gets multiplied by each local slope as it travels back toward the input. Sensitivity read forward and blame read backward are the very same product — one picture, two directions.

A network you can hold

Now the smallest thing that can honestly be called a network. One input $x$. A first dial $w_1$ that produces a middle value $h = w_1 \cdot x$. A second dial $w_2$ that turns $h$ into the prediction $\hat{y} = w_2 \cdot h$ — that is $\hat{y}$, "y-hat", the model's guess. And a grading station at the end. Two layers, two learnable dials, nothing hidden.

We need a ruler at that grading station, and here we make a deliberate switch. Chapter 12's ruler was $-\ln p$, built for probability bets. But this toy predicts a plain number, so we grade it with a plain ruler: the squared miss,

In words: multiply the input by the first dial to get the middle value, multiply that by the second dial to get the guess, then score the guess by how far it lands from the target $y$, squared.

I am choosing the squared-miss ruler purely so the arithmetic fits on one page — and I want to be honest about why that is allowed. The flow of blame is identical whichever ruler sits at the end of the chain; swapping rulers only changes the very first number blame starts with. And this ruler's slope is Chapter 5's pattern in disguise: the slope of $(\text{miss})^2$ with respect to the miss is $2 \cdot \text{miss}$, the same $x^2 \to 2x$ fact, applied to the miss instead of to $x$. No new differentiation rule enters this chapter.

Run the network left to right — the forward pass — with the chapter's numbers: $x = 2$, $w_1 = 1$, $w_2 = 2$, and target $y = 3$.

In words: two becomes two, becomes four — one too high against the target of three, so the exam fee is one.

Chapter 13's walk wants two numbers from this picture: $\frac{\partial L}{\partial w_1}$ and $\frac{\partial L}{\partial w_2}$ — how much the loss tilts if each dial alone is turned a little, the others held still. (That curly $\partial$ is the partial derivative: the tilt from one dial with everything else frozen.) Getting both is the whole job of the next section.

Blame flows backward

Start at the scene of the crime — the loss — and walk left, one hop at a time, multiplying local slopes as we go. This is the chain rule with a job to do.

Hop 0 — blame at the prediction. The ruler's slope is $2 \cdot \text{miss}$, and the miss is $\hat{y} - y = 4 - 3 = 1$, so $\frac{dL}{d\hat{y}} = 2 \cdot 1 = 2$. Read it plainly: if $\hat{y}$ rose a little, the loss would rise at two-per-unit. The prediction is carrying blame of $2$.

Hop 1a — into dial $w_2$. With $h$ frozen at $2$, the machine $\hat{y} = w_2 \cdot h$ is a straight line in $w_2$ with slope $h = 2$ — a line's slope is its multiplier (Chapter 4). Multiply the incoming blame by that local slope and the second dial gets its share.

Hop 1b — through to the middle value $h$. Same station, other lever: with $w_2$ frozen, $\hat{y} = w_2 \cdot h$ has slope $w_2 = 2$ in $h$, so $\frac{\partial L}{\partial h} = 2 \times 2 = 4$. The blame signal, now grown to $4$, keeps traveling left. Notice the subtle double move at every station: a share drops off into the local dial, and the signal itself passes through, rescaled, toward the input.

Hop 2 — into dial $w_1$. Here $h = w_1 \cdot x$ with $x = 2$, so $\frac{\partial h}{\partial w_1} = 2$, and the first dial's blame is $4 \times 2 = 8$. Collect the two dial slopes:

In words: each dial's blame is the blame arriving from downstream times how strongly that dial moves its own output.

Stack the two into the gradient the walk asked for:

In words: the loss is twice as sensitive to the inner dial $w_1$ as to the outer one $w_2$.

You may have noticed a tidy pattern — blame $2$ at the output, $4$ in the middle, $8$ at the first dial, doubling at every hop. That is a coincidence of this toy, where every local slope happens to equal $2$; it is not a law. Turn any of the numbers and the doubling vanishes. Do not expect it in general.

Now close the loop with Chapter 13 and actually take the step. With the learning rate $\eta = 0.05$ — eta, the size of each step — nudge each dial against its own slope:

In words: move each dial a little way in the downhill direction its blame points.

Re-run the forward pass with the new dials: $h = 0.6 \times 2 = 1.2$, then $\hat{y} = 1.8 \times 1.2 = 2.16$, so the miss is $2.16 - 3 = -0.84$ and $L = (-0.84)^2 = 0.7056 \approx 0.71$. The loss fell from $1$ to $0.71$ in one measured step. The full learning cycle — forward, grade, backward, step — has now happened once, entirely by hand, and you did it.

Why two passes are enough

The engine of the whole trick is one word: reuse. Look again at what the backward pass did at the middle value $h$ — it computed $\frac{\partial L}{\partial h} = 4$ exactly once, and $w_1$ consumed it. Now imagine ten dials feeding into $h$ instead of one. Every single one of them needs that same $4$ as the first factor of its own blame. Backprop computes each intermediate blame once and hands it to everyone upstream. Nudging, by contrast, would re-run the whole model ten separate times and rediscover that $4$ from scratch, ten times over, never once realizing it was the same number.

Count the bill both ways. On our toy, nudging costs one forward run per dial — two runs — plus a baseline; backprop costs one forward pass and one backward pass. At toy scale it is a wash. But scale it up to a two-billion-dial model and the two prices tear apart:

Nudging a two-billion-dial model would demand two billion full forward runs per step. Backprop demands one forward pass and one backward pass — about the cost of two runs, always, no matter how many dials. The backward pass visits each connection exactly once, just as the forward pass does. That is the whole miracle: the cost stops depending on the number of dials.

Give it its proper name: backpropagation — "propagating the blame backward". And let me demystify it in one sentence, because the word "blame" invites too much: it is not a learning rule, not an intelligence, not a brain mechanism. It is bookkeeping for the chain rule, carried out in an order clever enough never to repeat work. Nothing is punished; nothing is at fault. Blame is just the name we give a partial derivative on its way home.

What about the curves? Real networks put squashing functions between layers ($\sigma$ and cousins, Chapter 4). One calm beat handles them: a curve is one more machine in the chain, so it contributes one more factor — its local slope at the value flowing through it — to every blame that passes. Nothing new happens; the products just grow one factor longer. (The exact shape of $\sigma$'s slope we leave to the sister site.)

One practical consequence earns its own sentence, because it explains real hardware bills. The backward pass needs the forward values — remember that $h = 2$ was a factor in $w_2$'s blame. So the machine must keep its intermediate values from the forward trip, like sticky notes stuck on every station. At billion-dial scale those sticky notes are exactly why training devours memory.

When blame fades — or blows up

Push the chain-of-factors picture to its honest failure mode. Blame crossing a deep network is multiplied by one local slope per stage — a long product. Suppose each stage's factor is $0.5$. After ten stages the blame is $0.5^{10} = 1/1024 \approx 0.001$ of what it started as. (You met $2^{10} = 1{,}024$ in Chapter 4; this is that same fact stood on its head.) The early layers receive only whispers, and so they barely learn. The name for this is the vanishing gradient problem.

The mirror failure is just as real. Let each factor be $1.5$ instead. After ten stages the blame has grown to $1.5^{10} \approx 58\times$ what it started as — it arrives not as a whisper but as a scream, and the walk (Chapter 13's rampage) explodes. Exploding gradients are the backstage cause of those loss spikes.

The field answered with machinery you have already seen. Keep each layer's outputs in a tame range so the local slopes stay near $1$ — that is layer norm, one of the transformer's parts from Chapter 11. Give blame a highway that skips the multiplications entirely — that is the residual stream, Chapter 11's conveyor belt: blame rides it backward just as information rides it forward. And start the weights at carefully chosen sizes so the early products neither shrink nor swell — that is initialization. Here is the quiet payoff of the whole chapter: you can now see why the transformer diagram looks the way it does. Half its plumbing exists so that blame can survive the return trip.

Why the LLM cares

Assemble the full training heartbeat, every organ now named — a four-line ritual you can recite:

• Forward. The context flows through embeddings, attention, and MLPs to produce next-token probabilities (Chapters 7–11).
• Grade. The true next token's probability becomes a loss, $-\ln p$ (Chapter 12).
• Backward. Blame flows from that single number back through every layer, every attention head, every matrix, handing each of the billions of weights its personal slope (this chapter).
• Step. Every weight moves $\eta$ against its slope (Chapter 13).

That heartbeat, repeated, is all that training is.

Now one sentence of vertigo about scale. The same backward sweep you just ran across two dials runs across billions — straight through the very $\mathbf{W}_Q$, $\mathbf{W}_K$, $\mathbf{W}_V$ matrices you multiplied by hand in Chapter 10 — in about the time of two forward passes, and it happens millions of times over a single training run.

Which plants the seed of the next chapter precisely. One heartbeat grades one batch of text. What does it look like when this loop runs on trillions of tokens, across thousands of machines, for months — and how would you even know it is working? The answer is one number, falling. You already know which number.

"Blame flows backward" sounded like a metaphor when this chapter began. It is now an algorithm you have executed with a pencil: $2$, then $4$, then $8$, then a step, then a smaller loss. Every model you have ever talked to learned in exactly this way.

Where we're headed

You now hold the complete engine: a machine that guesses (Part II), a ruler that grades the guess (Chapter 12), a walk that improves the dials (Chapter 13), and the two-pass trick that makes the walk affordable (this chapter). What is left is not another mechanism — it is scale, and scale changes the texture of everything. Run this heartbeat on trillions of tokens, across thousands of machines, for months, and new questions take over: what do you feed it? How do you batch the exam? What does the falling number look like week by week — and how did anyone know, before spending a hundred million dollars, that it would keep falling? Next chapter, we watch an actual training run.