The Transformer

The machine, so far

Back in Chapter 9 we made a promise: there exists a machine that eats "the cat sat on the" and hands back $p(\text{next} \mid \text{context})$ — a probability for every token it knows. Ever since, we have treated that machine as a box. But we have not been idle: in Chapter 10 we built its beating heart, attention. This chapter assembles the rest of the box around that heart. And here is the promise for today, one I intend to keep in full: by the last paragraph the box will have no unexplained parts. Every component you meet is something you already own — matrices, adds, bends, softmax. Not one new kind of math is coming.

Start by taking honest inventory. One pass of attention is powerful, but on its own it still lacks four things a real language machine needs:

• Gap 1 — it can't see word order. Attention scores are built from dot products of queries and keys, and "the cat sat" and "sat cat the" produce the same dot products. The machine cannot yet tell who came first.
• Gap 2 — it asks only one kind of question. Our toy head in Chapter 10 asked "who did it?" But a sentence needs many questions asked at once — who did it, what came just before, which adjective modifies me.
• Gap 3 — after gathering, a token never gets to think. Attention lets a token collect information from its neighbors, but then nothing digests what it heard.
• Gap 4 — one pass of anything is shallow. Real understanding needs the whole arrangement repeated, deeply, without forgetting what came before.

The machine that closes all four gaps has a name. The transformer fixes them with exactly four ideas, and the next four sections take them in order: position stamps close Gap 1, multiple heads close Gap 2, MLP blocks close Gap 3, and a residual stream closes Gap 4. One honest diagram at a time — let's build it.

Stamping the seats

A confession first, then the fix. In Chapter 10 we quietly skipped a step. Attention scores travel with the words, not the seats: the dot product $\mathbf{q} \cdot \mathbf{k}$ contains nothing about who sat down first. Shuffle the sentence and every single dot product comes out identical. To a bare attention layer, "man bites dog" and "dog bites man" are the same news story (figure below). The only order-aware piece we met was the causal mask — the no-peeking-forward rule — and a crude "you may not look right" is nowhere near enough to carry grammar.

The fix is almost embarrassingly direct: before anything else happens, we add a position vector — a seat stamp — to each token's embedding. Seat 1 has its own stamp vector, seat 2 has another, and so on. The same word sitting in two different seats now enters the machine as two genuinely different vectors. That is Gap 1, closed.

One sentence of honest breadth, no formulas. Some models bake the stamps in as fixed wave patterns; others learn them like any other parameter; the newest models rotate the query and key arrows by a position-dependent angle instead (rotation you already own from Chapter 3). The engineering differs; the job is identical — get seat information into the vectors so attention can use it.

Many spotlights: multi-head attention

Chapter 10's attention asked one kind of question — our toy head asked "who did it?" But a sentence needs many questions asked simultaneously: who did it, what came just before, which adjective modifies me, is there a negation upstream. Each question is its own softmax over the tokens. The transformer's answer to Gap 2 is blunt: run several small attentions in parallel, and call each one a head.

Each head owns its own violet trio of matrices — its own query, key, and value lenses — so each head learns to ask a different question. The figure shows two heads working the same sentence. Head 1 is literally Chapter 10's grid; head 2 shows a different personality on the same words.

Head 1's shares are exactly the masked grid you computed in Chapter 10: the $\to 1.00$; cat $\to 0.02, 0.98$; sat $\to 0.09, 0.67, 0.24$. Head 2's shares ($1.00$; $0.80, 0.20$; $0.10, 0.75, 0.15$) are illustrative numbers, invented to show a "what came just before?" habit — each word putting most of its weight on its immediate predecessor. Same sentence, same mechanism, different learned lenses, different pattern.

How do the answers rejoin the stream? Each head produces its own small output vector per token (head 1's answer for sat was $\begin{bmatrix} 2.25 \\ 0.85 \end{bmatrix}$, from Chapter 10). The head outputs are laid end to end — two heads with 2 numbers each make one list of 4 — and one more learned violet matrix, the output matrix, mixes that combined list back down to the token's working width. No new math at all: concatenate, then matrix × vector, the move from Chapter 3.

Real-scale numbers, honestly sourced: GPT-3 runs 96 heads per layer, each working with 128-number queries and keys. That is 96 different questions asked about every token, in every layer, at once. Nobody tells head 41 what to ask — its question is whatever the violet lenses drifted into during training.

Private thinking: the MLP block

After attention, each token has gathered — sat now carries its cat. But gathering isn't digesting. The second block type in a transformer lets each token think about what it now holds, privately, with no cross-token traffic at all: the MLP block (also called the feed-forward block). That is Gap 3 closing.

Its anatomy is two matrices with a bend between them — and you were promised exactly this in Chapter 3's bridge: matrices alone stay flat, and two matrices in a row collapse into one, so the bend between them is what buys genuinely new behavior. The bend here is the ReLU — cruder than Chapter 4's smooth sigmoid: anything negative becomes 0, anything positive passes untouched. (Real models use a smoothed cousin called GELU; the shape story is identical.)

Why expand and then contract? The widening gives the block scratch space — room to test many little feature-questions at once ("does this vector look cat-ish and verb-adjacent?"), keep the ones that fired past the bend, and recombine them. Real proportions: the middle is typically 4× the stream width (GPT-3: a 12,288-wide stream expands to a 49,152-slot middle). And a fact worth saying out loud — roughly two-thirds of a big model's parameters live in these MLP blocks. Most of the machine, by weight, is private thinking.

The conveyor: residual stream and layer norm

Now the assembly principle that makes deep stacks workable — and the close of Gap 4. Neither block replaces a token's vector. Each block's answer is added onto it. The running vector that flows through the whole machine, collecting additions, is the residual stream. Blocks are editors scribbling margin notes, never authors rewriting the page.

(One honest forward note, no detail: this add-don't-replace design is also what will let blame flow backward through 96 layers when we train the thing — Part III cashes this in.)

The last small part. Before each block, a layer norm re-centers and re-scales the stream vector so its numbers sit in a standard range. Think volume knob, not thought: after many additions the stream's numbers could drift huge or tiny, and blocks behave predictably only on healthy-sized inputs. We'll skip its arithmetic deliberately — it is bookkeeping, and you lose nothing by reading it as "keep the numbers sane." (Our toy arithmetic above skipped it too.)

Assemble one full layer and the figure below appears: norm, then attention (talk to the other tokens), add to the stream; norm, then MLP (think alone), add to the stream. That unit — two detours off a conveyor — is one layer, and it is the only structural idea left between you and the whole machine.

The full stack

Stack that layer and you're done. Read the figure below from the bottom up and you have the whole journey: text becomes tokens (Chapter 7), tokens become embeddings plus seat stamps (Chapter 8 + this chapter), the vectors rise through layer after layer after layer, then one final norm, one final matrix, softmax, probabilities, and a sampled token (Chapter 9). Every stage is a chapter you have already lived.

Time to name the width symbol properly. The stream's width — how many numbers each token's vector carries — is called $d_{\text{model}}$ (say it "d-model, the width of the stream"). Our toys used 2 or 3; GPT-3 uses 12,288. And depth in real machines: GPT-2 small stacks 12 layers, GPT-3 stacks 96. Same layer design, photocopied — with its own learned matrices in every copy — then stacked.

The last stop deserves its own beat. After the final layer, the stream vector at the last position is multiplied by one final matrix — the unembedding matrix, which is $V$ rows tall (one row per token in the vocabulary, with $V \approx 50{,}000$ in GPT-class models) and $d_{\text{model}}$ columns wide. Each row dots the stream vector — Chapter 2's move, one last time — producing $V$ logits. Softmax turns them into $p(\text{next} \mid \text{context})$, and the die rolls. Every promise from Chapter 9 is now mechanically redeemed.

And now "$N$-billion parameters" becomes furniture instead of fog. Here is the complete list of where the numbers live: the embeddings ($V \times d_{\text{model}}$ numbers), each layer's attention lenses (the query, key, and value matrices plus the output matrix, per head) and its MLP pair ($\mathbf{W}_1$ and $\mathbf{W}_2$ — the bulk), and the unembedding matrix at the end. That's it. There is no other place a parameter can hide.

Then the generation loop, plainly: sample a token, append it to the context, run the whole stack again for the next one. Word by word, forever — that's the entire show, a loop you have now seen the inside of.

See it move

You have drawn the machine part by part; now watch it get built in motion. The video opens the box from Chapter 9 and rebuilds it from parts — seat stamps, the two detours of one layer, the stack rising, and the final softmax choosing "mat." Every number that moves on screen is one you have met.

Why the LLM cares

This section usually points forward — but this chapter is the LLM, so let's do the opposite and run the machine once, end to end, in a single breath. "The cat sat on the" arrives; the tokenizer chops it into tokens (7); each token becomes an embedding arrow (8) and picks up its seat stamp (this chapter); then, ninety-six times in a row, every token asks its questions and mixes its neighbors' values (10, via dot products from 2 and matrices from 3), and then thinks privately through two matrices and a bend (3, 4), each result added onto its stream (this chapter); the last token's final vector is dotted against every row of the unembedding matrix (2), softmax spreads the belief (6, 9), the die rolls (6), and "mat" appears. No step in that sentence is hand-waving anymore — each has a chapter number bolted to it.

Now notice what is conspicuously absent, because its absence is the deepest fact about LLMs. There are no grammar rules. No dictionary. No fact tables. No logic engine. Just the architecture you can now draw, run at widths and depths that are hard to picture. Everything an LLM "knows" is encoded in how the violet matrices transform vectors, and nowhere else.

Which sets up the obvious, enormous question for Part III. This chapter's machine, fresh off the drawing board, has every violet entry set to random noise — run it now and it produces confident gibberish. The entire difference between that noise-machine and the assistant you've talked to is the values of the numbers. Where do the right numbers come from? That is Part III — and you already hold its tools: a downhill compass (5) and a way to spread belief (6).