Part III · Ch. 12 — Features as Directions

Part III · Chapter 12 of 16

Features as Directions

Superposition, probes, and the knob Part II handed you


Part II left you an offset

A feature is a direction. That one sentence is the working idea behind everything people do when they open a model up and look inside. It is a claim, and it holds in some places and not in others.

You already have the object. In Chapter 6 you subtracted one word's position from another and got an arrow that belonged to neither of them, then applied that arrow somewhere else. It was a direction with a length. The claim you were testing was that traveling along it changed one property of a word and left the rest alone.

Now the same arrow gets a harder job. The space is no longer a map of words. It is the running vector inside a model, and the direction stands for something the model is tracking about the text in front of it.

A feature is a direction, as a claim

Start with the statement, because it is a claim about how a model stores things rather than a fact about geometry. A feature direction is a unit arrow in the model's space standing for one property the model tracks. How much of that property is present is the reading you get by projecting the position onto it.

$$\text{reading} = \mathbf{h}\cdot\hat{\mathbf{f}}$$

In words: take the position $\mathbf{h}$ the model is holding, take the unit direction $\hat{\mathbf{f}}$ that stands for the feature, and multiply them together the way Chapter 1 multiplied two arrows. One number comes out. That number is the shadow of the position on the feature's direction, and it is what people mean when they say a model "represents" something.

The idea that properties sit along straight directions, so that they add and subtract instead of tangling, is the linear representation hypothesis. It is a hypothesis. A great deal of published work is built on top of it, and there are properties it does not describe, which is where Chapter 16 picks the argument back up.

Two features fit

Give the model two dimensions and two things to track. Put the first feature along the first axis, $\hat{\mathbf{f}}_{\text{drink}} = (1.000,\, 0.000)$, and the second along the second, $\hat{\mathbf{f}}_{\text{animal}} = (0.000,\, 1.000)$. Their dot product is 0.000, so they are perpendicular.

Chapter 11's position at the end of the stream was $(7.000,\, 1.000)$. Read it on the first feature and you get 7.000. Read it on the second and you get 1.000. Neither reading has anything to say about the other, because moving along one direction does not change the shadow on the other. Two features, two dimensions, no interference.

Two panels. On the left, a mint arrow along the horizontal axis labeled f_drink and an amber arrow along the vertical axis labeled f_animal, with a violet dot at 7.000 comma 1.000 and dashed guides dropping to each axis. Text reads 'two features, two dimensions', 'the pair agrees 0.000, so neither reading disturbs the other', 'reading on f_drink 7.000', 'reading on f_animal 1.000'. On the right, a unit circle with three arrows a hundred and twenty degrees apart: f_drink in mint at zero degrees reading 1.000, f_animal in amber reading minus 0.500, f_question in blue reading minus 0.500. A violet ring marks the point sitting on f_drink. Text reads 'three features, two dimensions', 'every pair agrees minus 0.500', and 'a point sitting on f_drink still reads on the other two, and it reads negative'.
Room for two, not for three. The readings in the right-hand panel are negative rather than zero, so the two features that are not present still register.

Three do not

Now ask the same two dimensions to carry a third feature. Two signals share one channel cleanly while they sit at different frequencies, and the moment a third is crammed in they start showing up in each other's readings. The same thing happens here, and the arithmetic is small enough to check.

The best you can do, up to a reflection or a relabeling, is spread the three directions evenly around the circle: $\hat{\mathbf{f}}_{\text{drink}}$ at 0 degrees, $\hat{\mathbf{f}}_{\text{animal}}$ at 120 degrees, which is $(-0.500,\, 0.866)$, and $\hat{\mathbf{f}}_{\text{question}}$ at 240 degrees, which is $(-0.500,\, -0.866)$. Every pair of them agrees −0.500. That arrangement is superposition: more features than dimensions, each with a direction of its own, and the interference accepted as the price.

Put a point at one unit of $\hat{\mathbf{f}}_{\text{drink}}$ and nothing else. Its three readings are 1.000, −0.500 and −0.500. The feature that is there reads correctly. The two features that are absent read −0.500 each, so a probe watching for either of them sees a signal nothing put in the vector.

The other failure is harder to spot. Add half a unit of $\hat{\mathbf{f}}_{\text{question}}$ to the same point and it lands at $(0.750,\, -0.433)$. Now the readings are 0.750, −0.750 and 0.000. The feature present at half strength reads nothing at all, because its half unit is canceled by the shadow the first feature casts backward along it. Superposition costs false readings and missed ones, and neither is a defect in the probe.

A unit circle with three faint feature arrows labeled f_drink in mint, f_animal in amber and f_question in blue. A solid mint arrow runs from the origin to f_drink, then a blue arrow continues from there to a violet dot at 0.750 comma minus 0.433. A caption reads 'one unit of f_drink, plus half a unit of f_question'. On the right a panel headed 'what the three probes read' lists f_drink 0.750, f_animal minus 0.750, and f_question 0.000, with the 0.000 ringed in mint. Below it: 'f_question is present at half strength and its own probe reads nothing at all'.
Present, and reading zero. The ringed line is the feature that is in the vector, and it is the one whose own probe reports nothing.

Those numbers are the worst case, and a model only takes this deal because the worst case is rare. Features in a real model are sparse: each one is off on almost every input, so two of them firing at once is uncommon. If only one of our three is active at a time, the whole cost of the arrangement is the −0.500 reading on the two absent features. If two fire together you get the 0.750 / −0.750 / 0.000 case above. Sparsity is what makes storing more features than dimensions a sensible trade, and it is the hinge of the argument in Elhage and colleagues' Toy Models of Superposition (2022).

Why a model does it anyway

Chapter 3 measured the fact this rests on. Perpendicular directions are capped at the number of dimensions: in 384 dimensions you get 384 of them and not one more. Nearly perpendicular directions are not capped anywhere near that number, and the gap between the two counts is the budget superposition spends.

Random unit directions in a space of each size: how much the worst pair agrees, and how much the average pair agrees.
DimensionsDirections drawnLargest |cos|Average |cos|
230.9910.635
161000.7750.198
641,0000.5790.100
25610,0000.3490.050
38410,0000.2960.041

The last row is the model Chapter 9 walked ten thousand real articles through. Ten thousand directions in 384 dimensions, the worst pair agreeing 0.296 and the average pair 0.041. Ten thousand features would not fit as perpendicular directions. They fit like this, with a little leakage on every pair, and that leakage is what a probe reads and cannot tell apart from the feature.

A line chart with dimensions on a logarithmic horizontal axis at 2, 16, 64, 256 and 384, and agreement measured as absolute cosine on the vertical axis. A mint line labeled 'largest of any pair' falls from 0.991 at two dimensions through 0.775, 0.579 and 0.349 to 0.296 at 384 dimensions. An amber line labeled 'average pair' falls from 0.635 through 0.198, 0.100 and 0.050 to 0.041. A note reads 'the line falls and never lands on zero'.
The budget, measured. The top line keeps falling as the space grows and never reaches zero, which is why the arrangement always costs something.

A probe is a direction plus a threshold

Chapter 11 left twenty-six recorded positions: thirteen from the drink context and thirteen from the feed context, one per write. Suppose you did not know which was which and wanted a test that could tell them apart. The standard tool is a linear probe, and building one by hand takes three steps.

Worked example: a probe from twenty-six states

Step 1, average each group. The thirteen drink-context positions average to $(4.238,\, 1.215)$. The thirteen feed-context positions average to $(1.215,\, 4.238)$, the same pair of numbers the other way around, because the two sequences are mirror images of each other.

Step 2, subtract. $(4.238,\, 1.215) - (1.215,\, 4.238) = (3.023,\, -3.023)$. Divide by its length and the direction is $(0.707,\, -0.707)$.

$$\hat{\mathbf{w}} = \frac{\bar{\mathbf{h}}_A - \bar{\mathbf{h}}_B}{\|\bar{\mathbf{h}}_A - \bar{\mathbf{h}}_B\|}$$

Step 3, pick where to cut. Project the midpoint of the two averages onto that direction and you get 0.000. Read above the threshold and call the state a drink context; read below it and call it a feed context.

In words: the probe's direction is the arrow from the average of one group to the average of the other, scaled to length one. That is Chapter 6's offset doing a new job. There it took you from one word to another; here it points from one class of internal states to the other, and the number it hands back is a vote rather than a landing place.

Run the twenty-six states along it. The drink-context readings are 0.000, 0.071, 0.141, 0.636, 0.919, 2.192, 2.758, 3.323, 2.616, 3.253, 3.606, 4.031 and 4.243, and the feed-context readings are the same list with every sign flipped. Twenty-four of the twenty-six land on the correct side of 0.000, and the tightest correct call clears the threshold by 0.071.

The state a probe cannot classify

Two states are left, and they are the same state. Position 0 is the embedding, before any block has written anything, and the embedding of the word the is $(1.000,\, 1.000)$ in both sequences because it is the same token. One vector, two labels. It reads 0.000 on the probe, which is the threshold, and no threshold anywhere on the line would do better, because the two things being told apart are in the same place.

A one-dimensional strip showing the reading along the probe direction from minus four to four. Twelve mint dots labeled 'thirteen drink-context states' sit above the line at positive readings and twelve amber dots labeled 'thirteen feed-context states' sit below it at the mirrored negative readings. A violet vertical line marks the threshold 0.000. A single white dot ringed in red sits on the threshold, annotated 'state 0 is the embedding, the same vector in both sequences, so it reads 0.000 and no threshold can place it'. Another annotation marks the closest correct call at margin 0.071, and a note reads '24 of the 26 states land on the right side'.
One point that no line can place. The ringed dot on the threshold fails not because the probe is poor but because what the probe is looking for has not been written yet.

That one point is worth more than the twenty-four that worked. When a probe fails it means one of two things, and they are very different. Either the property is not written into the vector along a straight direction, or it is not written into the vector at all. At position 0 it is the second. Context has not arrived, and the single write from the attention block is what puts it there.

The probe is not the feature

Compare the probe's direction with the feature direction we started from. The probe is $(0.707,\, -0.707)$ and $\hat{\mathbf{f}}_{\text{drink}}$ is $(1.000,\, 0.000)$, and the cosine between them is 0.707. It is tempting to read that as a score, as though the probe had found most of the feature and missed a bit.

Don't. The two sequences in this toy are mirror images about the diagonal, so the average of one group and the average of the other are forced to be reflections of each other. Their difference has to point along $(0.707,\, -0.707)$ whatever the model is storing. The 0.707 is a fact about the symmetry of the exhibit, not a measurement of probe quality. Break the mirror and the number moves while nothing about the features has changed.

The general version of the point does not depend on our toy. A probe that scores well has shown the property is linearly decodable from those activations by somebody standing outside the model. Whether the model itself uses that direction is a separate question, and one the probe was never asked. Hewitt and Liang's Designing and Interpreting Probes with Control Tasks (2019) made this concrete: a probe strong enough to decode a real property is often strong enough to decode a random one, so the way to know what a score means is to run the same probe against shuffled labels and subtract.

Steering

A direction you can read is also a direction you can push along, which turns a measurement into a knob.

$$\mathbf{h}' = \mathbf{h} + \alpha\hat{\mathbf{w}}$$

In words: take the position the model is holding, add some multiple $\alpha$ of the direction, and hand the result back to the rest of the network as though the model had produced it. Positive $\alpha$ pushes toward one end of the direction, negative $\alpha$ pushes toward the other, and $\alpha = 0$ leaves everything alone.

Run it on the position Chapter 11 ended at. Adding 1.414 units of $(0.707,\, -0.707)$ to $(7.000,\, 1.000)$ lands on $(8.000,\, 0.000)$, which is where the model's first and second choices change places. You can work the flip point out before measuring anything. The position starts 1.000 above the horizontal axis, the direction's second coordinate is −0.707, and $1.000 \div 0.707 = 1.414$. Chapter 13 has the machinery that turns a position into probabilities, and it will measure what comes out.

Turner and colleagues' Activation Addition (2023) is this arithmetic on a real model: take the difference between two sets of internal states, add it back at run time, and the output moves the way the difference described. Everything this chapter said about probes applies to it, including that the direction you found separates your examples and is not necessarily one the model uses.

Sparse autoencoders, in three sentences

If features are crowded into too few dimensions, one response is to unpack them into more dimensions. A sparse autoencoder does that: it learns to rewrite each activation vector as a combination of a much larger set of directions, with a penalty forcing almost all of them to be off at once. Whether the directions it finds are the model's features or an artifact of the training setup is an open argument, and Bricken and colleagues' Towards Monosemanticity (2023) and Templeton and colleagues' Scaling Monosemanticity (2024) are where to go for the state of it.

Run it yourself

Three benches on one plane. The first crowds features into two dimensions and lets you drag a point while every probe reads it. The second plots Chapter 11's twenty-six states and hands you the probe direction and the threshold to move by hand, with a button that shuffles the labels so the control costs one click. The third adds a multiple of the probe direction to the final position and shows what the four-word vocabulary does about it.

Where you'll meet this

  • Anything called a probe, in any paper. The AI/ML book's Drawing Boundaries builds the classifier a probe is, one score and one threshold, with the sigmoid that turns the score into a confidence. A probe is that classifier pointed at a model's insides instead of at a dataset, and every caution there about what a boundary proves carries over.
  • Where the activations come from. The states this chapter probes are the numbers a model produces between its input and its prediction. The training book's The Forward Pass follows a pair of numbers through every multiplication and activation, which is the machinery that generates the twenty-six positions we treated as given.
  • Steering in production. When a vendor says a model's behavior was adjusted without retraining it, the arithmetic is usually this one: a direction found from a set of contrasting examples, added to the running vector at some layer with a coefficient somebody tuned. Knowing it is one vector addition tells you both why it is cheap and why it is brittle.

What you now know

  • A feature direction is a unit arrow in the model's space, and the strength of that feature at a position is the dot product of the position with the arrow.
  • Two features fit into two dimensions as perpendicular directions whose readings do not disturb each other, and that stops being available the moment there is a third feature.
  • Three directions in two dimensions can do no better than 120 degrees apart with every pair agreeing −0.500, and that arrangement is called superposition.
  • Superposition produces false readings on absent features and missed readings on present ones, and the point $(0.750,\, -0.433)$ shows both at once.
  • Models take the deal because features are sparse, so two rarely fire together, and because nearly perpendicular directions are not capped at the number of dimensions the way perpendicular ones are.
  • A linear probe is one direction and one threshold. The one built here reads 24 of 26 recorded states correctly, and the two it cannot read are the embedding, which is the same vector in both contexts.
  • A probe that works has found a direction separating your examples, which is not the same as a direction the model uses, and a shuffled-label control is what tells the two apart.

Two things you can now look up that you could not have named an hour ago: superposition, which is what people mean when they say a model has more features than neurons, and the fact that a probing result without a shuffled-label control is not yet evidence.

Where we're headed

We have a direction you can read and the same direction used as a knob, and we left the knob turned to 1.414 with a prediction about what should happen and no way to check it. The next chapter builds the last step of the model, the one that takes a position and turns it into a probability for every word in the vocabulary. It is a dot product, a softmax, and a map of the plane divided into regions by which word wins. Once it is built you can measure what steering did and read the answer in bits.