Part III · Ch. 17 — The Transformer

Part III · Chapter 17 of 29

The Transformer

Watch attention change while a model learns names


What shape does it eat?

A recurrent model passes information through a chain of states. What if each position could compare itself directly against earlier positions before deciding what to mix? Dot products and weighted averages are enough to build that. The model learns the vectors used for those comparisons along with the rest of its weights.

A transformer combines attention with transformations applied separately at each position. Its vocabulary is the set of available tokens, and V counts them. The name transformer in this chapter uses the same synthetic names and the same split as the name GRU in the previous chapter. Each character ID selects an embedding row. Position encoding adds information about where the token sits; the name transformer learns those position vectors and adds them to the token embeddings.

One saved token sequence decoded into its characters, each character’s learned embedding added to its learned position vector. Padding at the end is shown muted and labeled.
Tokens become vectors with position information before the attention block.

The recorded stream carries sixteen numbers per position. Its input has up to eight positions and its vocabulary has twenty-five tokens. The full parameter table holds 4,233 entries: embeddings, position vectors, attention projections, normalization scales and biases, dense layers, and the output head. Right-padding targets are ignored by the loss. Padding is not a zero embedding, and the model applies no separate padding mask inside attention.

Each position chooses a mixture

A query is a vector used to score information to retrieve. A key is the vector it gets compared with to make a score. A value is the vector mixed according to the resulting shares. A head is one learned query, key, and value transformation with its own attention calculation. The name transformer has two heads of eight coordinates each; our hand example uses two coordinates so every product is checkable.

The hand example’s query and key vectors, their nine dot products, the scaling, the causal mask and the three output mixtures, left to right. Values carry the symbol V_a.
Every attention share comes from a score, and every output is a weighted value sum.

Transpose swaps a matrix’s rows and columns without changing its entries. A matrix of row-shaped keys is transposed so query rows meet key columns in a matrix product. Divide each query-key dot product by the square root of the key width, then apply softmax across the allowed keys. The causal mask drops keys later than the query. Excluded cells don’t enter the denominator as small positive shares.

Compute all the scores and the mixture

Our three query vectors and three key vectors are both (1,0), (0,1), (1,1). The value vectors are (1,0), (0,2), (1,1). The second value differs from its key on purpose, to keep the two jobs distinct. A key sets a comparison score; a value supplies the information mixed after that comparison.

All nine comparisons

The first query gives [1×1+0×0,1×0+0×1,1×1+0×1]=[1,0,1]. The second gives [0×1+1×0,0×0+1×1,0×1+1×1]=[0,1,1]. The third gives [1×1+1×0,1×0+1×1,1×1+1×1]=[1,1,2]. Dividing by sqrt(2) gives rows [0.7071,0,0.7071], [0,0.7071,0.7071], and [0.7071,0.7071,1.4142].

Exponentials and denominators

Exponentiating gives [2.0281,1,2.0281], [1,2.0281,2.0281], and [2.0281,2.0281,4.1133]. Without masking, the row denominators are 5.0562, 5.0562, and 8.1695. Divide every exponential by its row total. The shares are [0.4011,0.1978,0.4011], [0.1978,0.4011,0.4011], and [0.2483,0.2483,0.5035].

In words: exponentiate each scaled score and divide by the total for that query row. Every unmasked mixing share is now computed; the next operation changes which keys may contribute.

Remove future positions, then mix values

With masking, the denominators become 2.0281, 3.0281, and 8.1695. The first row’s shares are [1,0,0], giving output (1,0). Let a=1/(1+exp(1/sqrt(2)))≈0.3302. The second shares are (a,1−a,0), giving (a,2(1−a))≈(0.3302,1.3395). Let b=1/(2+exp(1/sqrt(2)))≈0.2483. The last shares are (b,b,1−2b), giving (1−b,1)≈(0.7517,1).

For comparison, the unmasked output rows are (0.8022,0.7967), (0.5989,1.2033), and (0.7517,1). The last row is unchanged, because it has no future key to exclude. Bold Q and K collect queries and keys; bold V with subscript a collects attention values, distinct from vocabulary size V. The key width is d with subscript k, and bold M is the no-future mask:

$$\mathrm{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}_{\!a})=\mathrm{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_k}}+\mathbf{M}\right)\mathbf{V}_{\!a}$$

In words: score each query against the keys, scale the scores, apply the no-future mask, turn scores into shares, then mix the value vectors.

Read the actual attention during training

The trained attention arrays were saved for a fixed set of token sequences. Their axes are sequence, head, query position, and key position. Select a different head and you inspect a different learned comparison inside the same model. Keep the token labels beside the grid, so a bright cell points at positions you can name.

Attention shares for one token sequence and one head, on the same color scale, at the start and at the end of training, beside the training and validation loss curves.
The saved attention matrices show how this run’s mixing shares changed.

At step 3000, the name transformer’s training loss is 1.296 and its validation loss 1.320. The name GRU finished at a validation loss of 1.325 after 2000 steps. The two losses are comparable because the task is shared, but the runs differ in update budget and architecture, so they don’t say which family needs more data.

Attention learns scores that become inspectable mixing shares.

Where this shows up when you train

Attention is one branch of the name transformer’s computation. A residual adds a branch’s output to the incoming stream. In a hand example, stream (1,0) plus update (0,1) becomes (1,1). Layer normalization centers and rescales one token vector, then applies a learned scale and bias. The name transformer uses it before each branch. Its sixteen-coordinate stream keeps the same width across those additions.

The name transformer’s block: a shared residual stream, a normalized attention branch and a normalized dense branch, each added back to the stream, with tensor shapes labeled. A separate inset shows the hand addition (1,0)+(0,1)=(1,1).
The block combines mixing across positions with transformations within each position.

The first branch normalizes, calculates attention, joins the two heads, and projects their outputs before adding to the stream. The second normalizes, expands through a dense layer, applies ReLU, contracts back to stream width, and adds again. A final head produces vocabulary logits. So the operations after attention can change the prediction even when the attention shares look the same.

Hold the query row fixed and edit one value coordinate: the shares stay put while the weighted output moves.

What you now know

  • Attention scores produce row-normalized mixing shares.
  • Values carry the information combined by those shares.
  • A trained head’s pattern does not by itself establish a causal explanation.

Where we’re headed

A graph network mixes neighbors too, and its connections come from an observed edge list. Continue the story.