The machine, so far
Attention lets tokens exchange information. A transformer adds position information, parallel attention units, per-token processing, and a path through many layers. Let's assemble the decoder model behind our next-token predictor.
| Need | Component |
|---|---|
| Position information | Position embeddings or position-dependent rotations. |
| Several matching patterns | Multiple attention heads. |
| Process gathered information | An MLP at each position. |
| Build deep computations | Residual additions and normalization. |
Stamping the seats
Without position information, query-key scores travel with their tokens when we reorder the input. A causal mask restricts access by order, but explicit position information provides a richer signal.
A position embedding adds a seat-specific vector to a token's embedding.
Worked example
We stamp with toy integers, invented so you can check the sum by hand. Cat's embedding is $\begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}$. Sitting in seat 2, it picks up that seat's stamp $\begin{bmatrix} 2 \\ 0 \\ 0 \end{bmatrix}$, and the vector that enters the machine is the sum:
$$\mathbf{x}_{\text{cat, seat 2}} = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix} + \begin{bmatrix} 2 \\ 0 \\ 0 \end{bmatrix} = \begin{bmatrix} 2 \\ 1 \\ 0 \end{bmatrix}$$In words: the word's meaning-arrow plus the seat's location-arrow, added component by component like any two vectors from Chapter 1. The same cat sitting in seat 5, whose stamp is $\begin{bmatrix} 0 \\ 0 \\ 2 \end{bmatrix}$, enters as $\begin{bmatrix} 0 \\ 1 \\ 2 \end{bmatrix}$, a different arrow for the same word. Queries and keys computed from these vectors can now ask about position, so "what's the word right before me?" has an answer.
Some position vectors are learned; others use fixed wave patterns. Rotary position embeddings instead rotate queries and keys according to position. These are alternative implementations of the position signal.
Many spotlights: multi-head attention
A head computes one attention pattern. Multi-head attention lets several patterns coexist.
Head 1 reuses Chapter 10's masked shares. Head 2 is illustrative: its sat row is 0.10,0.75,0.15. Each head produces its own vector. We concatenate those outputs, then apply a learned output matrix to restore the stream width. Two two-entry outputs become a four-entry list before that projection.
As a historical scale example, GPT-3's largest model has 96 heads per layer, with 128-number keys and queries.
Private thinking: the MLP block
An MLP is a multilayer perceptron. The transformer's MLP block processes each position separately: expand, bend, project back.
Our bend is ReLU. Other models use smooth activations such as GELU or gated feed-forward blocks.
Worked example
Work every digit on a fresh two-number toy stream vector $\begin{bmatrix} 2 \\ 1 \end{bmatrix}$. First expand with $\mathbf{W}_1 = \begin{bmatrix} 1 & 1 \\ 0 & 2 \\ -2 & 1 \end{bmatrix}$ (2 numbers in, 3 out). Each row dots the input from Chapter 2: $1\cdot 2 + 1\cdot 1 = 3$, then $0\cdot 2 + 2\cdot 1 = 2$, then $(-2)\cdot 2 + 1\cdot 1 = -3$. The bend zeroes the negative. Then contract back with $\mathbf{W}_2 = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & -1 \end{bmatrix}$: its rows give $1\cdot 3 + 0\cdot 2 + 1\cdot 0 = 3$ and $0\cdot 3 + 1\cdot 2 + (-1)\cdot 0 = 2$.
$$\mathbf{W}_1 \begin{bmatrix} 2 \\ 1 \end{bmatrix} = \begin{bmatrix} 3 \\ 2 \\ -3 \end{bmatrix} \;\xrightarrow{\text{bend}}\; \begin{bmatrix} 3 \\ 2 \\ 0 \end{bmatrix} \;\xrightarrow{\;\mathbf{W}_2\;}\; \begin{bmatrix} 3 \\ 2 \end{bmatrix}$$In words: spread the vector into more slots, kill the negative ones, then mix what survives back down. The block's answer for this token is $\begin{bmatrix} 3 \\ 2 \end{bmatrix}$.
The intermediate width provides room for feature combinations. A common classic design expands to four times the stream width: GPT-3 uses 12,288 → 49,152 → 12,288. In that design, MLP matrices account for roughly two-thirds of the main layer weights.
The conveyor: residual stream and layer norm
The residual stream carries information through the stack. A residual connection updates that stream by addition.
Worked example (continued)
The stream carried $\begin{bmatrix} 2 \\ 1 \end{bmatrix}$ into the MLP block; the block computed $\begin{bmatrix} 3 \\ 2 \end{bmatrix}$; so the stream leaves as the sum:
$$\begin{bmatrix} 2 \\ 1 \end{bmatrix} + \begin{bmatrix} 3 \\ 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 3 \end{bmatrix}$$In words: add the block's answer onto the stream that fed it. The original $\begin{bmatrix} 2 \\ 1 \end{bmatrix}$ contributes directly to the sum.
Addition gives a direct path through the computation and later through its derivatives. It does not guarantee that earlier information remains recoverable: additions can cancel or alter it.
Layer norm centers and rescales a copy of the stream before a block in this pre-norm layout. We skip its arithmetic in the toy example. Other architectures use related normalizations.
A layer contains both detours. Each layer owns its own learned weights.
The full stack
The width $d_{\text{model}}$, “d-model,” counts entries in each stream vector. V is vocabulary size and n is the token count. The unembedding matrix has V rows and d-model columns.
| Stage | In → out | Shape |
|---|---|---|
| Embedding + position | IDs → stream vectors | n → n × d-model |
| Normalization | Stream → normalized block input | n × d-model, unchanged |
| Attention + output projection | Across-token mixing → stream update | n × d-model, unchanged |
| Residual addition | Stream + update → stream | Same shape on both sides |
| Norm + MLP + add | Per-position processing → stream | Widen internally, return to d-model |
| Final norm + unembedding | Last stream vector → logits | d-model → V |
| Softmax and selection | Logits → shares → one token | V → V → 1 |
GPT-2 small uses 12 layers; the largest GPT-3 uses 96. The figure uses L for layer count; Part III uses L for loss, a different quantity.
Parameters live in embeddings, attention projections and output maps, MLP matrices, normalization scales and offsets, and sometimes biases and learned position embeddings. Input and output embedding weights may be shared. Append the selected token and compute the next prediction.
See it move
Watch the stream acquire each block's update. Follow the last position to the vocabulary scores.
Why the LLM cares
We can now trace the forward computation: tokens become position-aware vectors, layers exchange and process information, and the final projection produces next-token scores.
The architecture supplies the recipe. The parameter values determine its behavior. Random initial values do not yet make a useful language model; learning must find values that predict well.
What you now know
- Position information lets attention use sequence location.
- Heads compute parallel attention patterns and combine their outputs.
- An MLP processes each position independently.
- Residual additions connect the stack; normalization prepares block inputs.
- The final projection produces one logit per vocabulary token.
- Each layer has learned parameters that training must set.
Where we're headed
The machine is assembled. Next we'll measure prediction error, giving training a number to reduce.