Part III · Ch. 11 — The Residual Stream

Part III · Chapter 11 of 16

The Residual Stream

One vector, read and written by every layer


Where does the vector go next?

A transformer layer does not replace the vector it is handed. It adds to it.

Chapter 10 left something hanging. A head took the position of the at $(1.000, 1.000)$, looked back at drink, and produced $(3.413, 0.196)$. Nobody said what the model does with that answer. It does not throw the old position away and move in. It adds the head's output to what was already sitting there, and the running total is what the next block reads.

So a position inside a model is not a sequence of replacements. It is one vector that gets added to, over and over, by every block in the stack. By the end of this chapter you will have added thirteen rows of a table with a pencil and landed on the model's final answer. One of those thirteen rows moves the answer backward, and you will have found it.

Every block reads the whole thing and adds to it

The running vector at one position has a name. It is the residual stream, and it starts life as the token's embedding and ends as whatever the model hands to the output layer. Each block's contribution is a write: a vector of the same shape as the stream, added in.

Think of a running total on a receipt. Every line adds something, the total at the bottom is the sum of the lines, and a refund is a line with a minus sign on it. Nothing on a receipt erases an earlier line, and a refund line leaves the original charge printed above it.

A tall rounded lane labeled at the bottom 'embedding 1.000, 1.000' and at the top 'final position 7.000, 1.000', with an upward arrow beside it labeled 'depth'. Six boxes sit to the right of the lane, labeled layer 1 through layer 6, each reading 'attention, then MLP'. Each box has a gray arrow running from the lane to the box, labeled on the first one 'reads all of it', and an amber arrow running from the box back to the lane, labeled 'adds its write'. No arrow passes across the lane to anything else.
One lane, six blocks, nothing crossing. Every block reads the whole stream and adds its write back into the same lane. The lane is the only place the blocks meet.

Written down, one block's step is this:

$$\mathbf{h}^{(\ell+1)} = \mathbf{h}^{(\ell)} + \mathrm{Block}_\ell\!\left(\mathbf{h}^{(\ell)}\right)$$

In words: the stream after block $\ell$ is the stream before block $\ell$ plus whatever that block computed from it. The old vector is still a term in the sum after the block has run.

The equation says something easy to skip over. The block's input and its output are the same size, and they are added, so a block cannot change the subject. It can only push the stream a little way in some direction.

One layer, computed

Chapter 10's toy has exactly one layer, so we can run it end to end on paper. Follow the second token, the, through the sequence drink the. Its embedding is $(1.000, 1.000)$. The head's output from chapter 10 is $(3.413, 0.196)$. Then comes the block every architecture diagram draws as a box labeled MLP, which in this toy is $\mathbf{W}_2\,\mathrm{ReLU}(\mathbf{W}_1\mathbf{h} + \mathbf{b}_1)$ with $\mathbf{W}_1 = \begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix}$, $\mathbf{b}_1 = (-1, -1)$ and $\mathbf{W}_2 = 0.5\,\mathbf{I}$.

Worked example: one layer, one position

Step 1, the attention write. Add the head's output to the embedding: $1.000 + 3.413 = 4.413$ across, $1.000 + 0.196 = 1.196$ up. The stream now stands at $(4.413, 1.196)$, length $4.572$.

Step 2, into the MLP. The first matrix hands each neuron the difference between the two coordinates, one way around for the first neuron and the other way around for the second, and the bias takes $1.000$ off both. Working from the unrounded position rather than the three decimals in the table, the pre-activations are $(2.218, -4.218)$.

Step 3, the ReLU. ReLU keeps positive numbers and sends negative ones to zero, so $(2.218, -4.218)$ becomes $(2.218, 0.000)$.

Step 4, the MLP write. The second matrix halves what is left: $0.5 \times 2.218 = 1.109$, and the second coordinate stays at zero. The MLP's write is $(1.109, 0.000)$, so the stream ends the layer at $5.522$ across and $1.196$ up, length $5.650$.

$$\mathbf{h}^{(2)} = \mathbf{e} + \mathbf{w}_{\text{attn}} + \mathbf{w}_{\text{MLP}}$$

In words: the position at the end of the layer is the embedding plus the attention write plus the MLP write. Check it by adding the three rows of the table below down each column and comparing the answer to the last row.

One layer of the toy, both sequences. The mirror is exact: swap the two coordinates of every entry in the left column and you get the right one.
StepWrite, drink thePosition afterLengthWrite, feed thePosition after
embedding(1.000, 1.000)(1.000, 1.000)1.414(1.000, 1.000)(1.000, 1.000)
attention 1(3.413, 0.196)(4.413, 1.196)4.572(0.196, 3.413)(1.196, 4.413)
MLP 1(1.109, 0.000)(5.522, 1.196)5.650(0.000, 1.109)(1.196, 5.522)

The cosine between the two end positions is $0.414$. Chapter 10 measured $0.114$ between the raw head outputs and $0.505$ once the embedding was added back. One layer of adding has moved that number again, and the word the is now in two places that share less than half their heading.

Only one of the two neurons can fire

Look at the two rows of $\mathbf{W}_1$ again: $[1, -1]$ and $[-1, 1]$. They are negatives of each other, so the two pre-activations are always negatives of each other before the bias goes on, and the bias is the same on both. One of them is positive and the other is more negative, every time. The ReLU then zeroes one and keeps the other.

This toy MLP is a switch with two settings. Feed it a position where the first coordinate leads and it writes along the first axis; feed it one where the second leads and it writes along the second. That is a small fact about two hand-chosen rows, and it lets you predict the MLP's write before you compute it. Real MLPs have thousands of neurons and no such symmetry, and which ones fire is the interesting part rather than a foregone conclusion.

The stream is a path

One layer is not a stack. To see what depth does we need more writes than the toy has, so the rest of this chapter uses a recorded trace. It is twelve writes, two per layer for six layers, that I wrote for this chapter with the shape a real stack's writes have, rounded to one decimal per component so you can add them by hand. They are not the toy's numbers and they are not measured from a trained model. The claims below about real models come from the papers cited, not from this table.

Drawn in the plane, the thirteen positions make one continuous path from the embedding to the answer.

A plane with axis 1 of the stream across and axis 2 up. Thirteen dots joined by mint arrows trace a path from step 0, ringed in amber at 1 comma 1, rightward and slightly up to step 3 near 2.5 comma 1.6, then down and right through steps 4 to 7 at 5.7 comma 1. One arrow, from step 7 to step 8, is drawn in red and runs back to the left to 5.1 comma 1.4, labeled 'step 8, MLP 4: the only step back'. The path then resumes rightward through steps 9 to 12 and ends ringed in mint at 7 comma 1, labeled 'step 12, the final position'.
Thirteen positions, one path. Every arrow is one block's write. Twelve of them carry the position further along toward the answer, and exactly one of them goes back.

Two things are visible in the picture before any arithmetic. The path is mostly one direction, which is why the cosine between each position and the final one climbs from $0.800$ at the embedding to $1.000$ by the last few steps. And it doubles back once, at step 8, which is the subject of a later section on this page.

Add the column

This is the table, and it is also the exercise. Add the thirteen entries in the write column, one coordinate at a time, and compare your answer to the final row.

The twelve-write trace for the drink-context sequence. The mirror sequence swaps the two coordinates of every write.
StepWritePosition afterLengthCosine with the final position
embedding(1.000, 1.000)(1.000, 1.000)1.4140.800
attention 1(0.400, 0.300)(1.400, 1.300)1.9100.822
MLP 1(0.200, 0.100)(1.600, 1.400)2.1260.838
attention 2(0.900, 0.200)(2.500, 1.600)2.9680.910
MLP 2(0.300, −0.100)(2.800, 1.500)3.1760.939
attention 3(1.600, −0.200)(4.400, 1.300)4.5880.989
MLP 3(0.500, −0.300)(4.900, 1.000)5.0010.998
attention 4(0.800, 0.000)(5.700, 1.000)5.7870.999
MLP 4(−0.600, 0.400)(5.100, 1.400)5.2890.992
attention 5(0.700, −0.200)(5.800, 1.200)5.9230.998
MLP 5(0.400, −0.100)(6.200, 1.100)6.2970.999
attention 6(0.500, −0.100)(6.700, 1.000)6.7741.000
MLP 6(0.300, 0.000)(7.000, 1.000)7.0711.000

The thirteen writes add to $(7.000, 1.000)$, which is the last row of the position column. That is not a coincidence you have to take on faith. It follows from the equation in the second section, applied twelve times in a row:

$$\mathbf{h}^{(L)} = \mathbf{e} + \sum_{\ell=1}^{L} \mathrm{Block}_\ell\!\left(\mathbf{h}^{(\ell)}\right)$$

In words: the position at the top of the stack is the embedding plus the sum of every block's write. No block appears anywhere except as one term in that sum, which is why adding the column works and why the order of the rows does not change the total.

Who moved it

The sum tells you where the position ended. It does not tell you which block got it there. For that, measure each write against the direction the answer finally pointed in. The final position $(7.000, 1.000)$ has length $7.071$, so its unit direction is $(0.990, 0.141)$, and a write's contribution is chapter 1's shadow:

$$\mathbf{w} \cdot \hat{\mathbf{h}}^{(L)}$$

In words: take one block's write, dot it with the unit vector pointing along the final position, and you get how far that write pushed the stream along the direction the answer ended up in. Because every write is a term in one sum, these thirteen shadows add up to the final length, $7.071$, with nothing left over.

A horizontal bar chart of thirteen rows, from embedding at the top down to MLP 6 at the bottom, measuring each write's shadow on the final direction 0.990, 0.141. Most bars are mint and point right: embedding 1.131, attention 1 0.438, MLP 1 0.212, attention 2 0.919, MLP 2 0.283, MLP 3 0.453, attention 4 0.792, attention 5 0.665, MLP 5 0.382, attention 6 0.481, MLP 6 0.297. The attention 3 bar is amber and longest at 1.556, annotated 'biggest single write, and it is not the last one'. The MLP 4 bar is red and points left to minus 0.537, annotated 'a write can be negative'. A line beneath reads 'the thirteen shadows sum to 7.071, the final length'.
Thirteen shadows on one direction. Each bar is how far that block pushed the position along the direction the answer ended up in, and the bars add to the final length.

The biggest single contribution is $1.556$, from attention 3, in the middle of the stack. The embedding comes second at $1.131$, and the last block of all contributes $0.297$, less than a twentieth of the total. Depth is not a ramp where the last step does the work. In this trace the work is spread across the middle, and the final blocks are making small corrections to something that was already mostly decided.

A layer can undo a layer

One bar in that chart points the wrong way. MLP 4 writes $(-0.600, 0.400)$, and its shadow on the final direction is $-0.537$. It is the only negative row in the table, and it arrives right after the stream reached $(5.700, 1.000)$, its furthest point along axis 1 so far. MLP 4 takes $0.600$ of that $5.700$ back, which is $10.526$ percent of it, and the stream's length drops from $5.787$ to $5.289$.

A line chart of the length of the stream against the number of writes added so far, from 0 to 12. The mint line climbs from 1.414 at depth 0 to 7.071 at depth 12. One segment, from depth 7 to depth 8, is drawn in red and falls from 5.787 to 5.289, with the depth 8 point ringed in red and annotated 'the one dip: 5.787 down to 5.289'. Every other segment rises.
Length against depth, for this recorded trace. Twelve steps make the stream longer and one makes it shorter, which is the step whose shadow on the final direction is negative.

Growth with depth is the normal case in this trace and the dip is the exception. Whether that is a general property of stacks or a feature of these twelve rows is a fair question, and it has been measured. Stefan Heimersheim and Alex Turner ran dozens of prompts through GPT-2-family models in Residual stream norms grow exponentially over the forward pass (2023). They found the length of the residual stream growing by about $4.5$ percent per layer in GPT-2-XL. That is a measurement on one family of models with one normalization scheme, so it is evidence about those models and not a theorem about stacks.

What the negative row settles is a different question. If a block can write $-0.537$ against the direction the answer needs, then an earlier block's contribution can be partly canceled by a later one, and nothing in the architecture prevents it. A residual connection guarantees that every write is in the sum. It does not guarantee that any write survives in a form you can still read at the top.

Walk the path yourself

The interactive below starts on the thirteen rows of the table. The slider walks the path one write at a time and prints the position and its length at every step. Clicking a write in the list switches it off, and the path redraws without it, so you can ask what any one block was worth by taking it away.

Why the toy is one layer deep and not twelve

Run chapter 10's toy for a second layer and it falls apart, and the reason is worth seeing. Token 1 is the first word in the sequence, so it has nothing earlier to look at. A causal head lets it attend only to itself, which means the head hands back the token's own value and the stream doubles. After one layer the first position of drink the sits at $(11.500, 0.000)$, from an embedding of length $4.000$. Do that a few more times and the numbers run away.

Real stacks do not run away, because every block normalizes its input before reading it: the stream is rescaled to a standard length first, so a block sees a direction rather than a magnitude. That is layer norm, and it is taught in the LLM book's The Transformer along with multi-head attention and the rest of the architecture. This toy has no normalization, which is why it is one layer deep. A toy that scales would need the normalization, and the normalization would need a chapter of its own.

Where you'll meet this

  • "We read the stream at layer 14." Interpretability papers say this constantly, and it is a sentence about the sum. Stopping at layer 14 means adding the first 28 or so writes and looking at the total, which is a vector in the same space as the final one. The instrument works because the architecture adds.
  • Steering vectors. Adding a fixed direction to the stream at some layer, and watching the output change, is the technique in Turner and colleagues' Steering Language Models With Activation Engineering (2023). It attaches to the stream for the same reason a block does: one more term in a sum is a well-formed thing to add, and the blocks above it will read the result without complaint.
  • LoRA. A low-rank adapter, from Hu and colleagues' LoRA: Low-Rank Adaptation of Large Language Models (2021), trains a small extra write and adds it to what a frozen layer already produces. Fine-tuning a model this way is adding rows to the receipt rather than rewriting the ones already on it.
  • Debugging a long prompt. When a model's answer changes because of something far back in the context, the change reached the answer as a write from some head at some depth. The projection table on this page is the shape of the question you would ask: which write, and how much of it survived to the top.

The framing of the stream as a channel that blocks read from and write to comes from Elhage and colleagues' A Mathematical Framework for Transformer Circuits (2021). Their sentence for it: every component of a transformer communicates by reading and writing to subspaces of the residual stream.

What you now know

  • A transformer keeps one running vector per position, the residual stream, and every block reads all of it and adds a write of the same shape back into it.
  • One layer of the toy moves the word the from $(1.000, 1.000)$ to $(5.522, 1.196)$ in two writes, and the MLP's write is $(1.109, 0.000)$ because the ReLU zeroed one of the two pre-activations.
  • The position at the top of a stack is the embedding plus every write. That is why the thirteen rows of the trace add to $(7.000, 1.000)$, and why you can read the sum at any intermediate depth.
  • Projecting each write onto the final direction says who moved the position: attention 3 contributes $1.556$, more than the embedding's $1.131$, and the last block contributes $0.297$.
  • MLP 4's write projects $-0.537$, taking back $10.526$ percent of the axis-1 position that stood at $(5.700, 1.000)$, so a later block can undo an earlier one.

Where we're headed

Every write on this page has been a direction with a size, and we have been measuring them against one direction, the one the answer ended up pointing in. That was a convenience. The next chapter asks what a direction in this space is for. Do the things a model has learned live along particular directions, how many of them fit when there are more things to represent than dimensions to represent them in, and what happens to a reading when two of them overlap? The answer to the last question is $-0.500$, and it is not a bug.