Attention · How LLMs Actually Work

One frozen arrow isn't enough

“River bank” and “bank account” start with the same embedding for bank, yet need different representations. Likewise, “it” in “the cat knocked over the vase because it was fragile” needs information about earlier words.

Attention lets a token use its context. We'll build one attention calculation for “the cat sat,” using dot products, matrices, and softmax.

Three roles: query, key, value

Each token gets a query, key, and value.

Three jobs for each token
Role	Question it answers	Use
Query	What information am I looking for?	Dot with each key.
Key	What can others match against?	Produce a relevance score.
Value	What information will I contribute?	Enter the weighted mixture.

Asking, matching, and contributing are different jobs, so they use different learned transformations.

The cast, fixed for the whole chapter

Each token gets a two-number query ■ q, key ■ k, and value ■ v, produced by a pretend mini-model. These same numbers appear in the text, all four figures, the video, and the widget.

Query, key and value for each token
token	query q	key k	value v
the	[2, 0]	[1, −1]	[0, 2]
cat	[0, 2]	[1, 1]	[3, 1]
sat	[1, 1]	[0, 1]	[1, 0]

Real models use lists hundreds of numbers long. Two numbers each keeps every step checkable by hand, and nothing else about the mechanism changes.

Scores, then shares

Sat's query is [1,1]. Dot it with each key, including its own:

$$\mathbf{q}_{\text{sat}} \cdot \mathbf{k}_{\text{cat}} = 1 \times 1 + 1 \times 1 = 2$$

In words: line "sat"'s question up against "cat"'s key, multiply matching slots, and add, for a score of 2. The other two go the same way: $\mathbf{q}_{\text{sat}} \cdot \mathbf{k}_{\text{the}} = 1 \times 1 + 1 \times (-1) = 0$ and $\mathbf{q}_{\text{sat}} \cdot \mathbf{k}_{\text{sat}} = 1 \times 0 + 1 \times 1 = 1$.

Scores 0,2,1 become positive shares through softmax. An attention weight is one of those shares.

$$(a_{\text{the}},\, a_{\text{cat}},\, a_{\text{sat}}) = \mathrm{softmax}(0,\, 2,\, 1) = \left(\tfrac{1.00}{11.11},\, \tfrac{7.39}{11.11},\, \tfrac{2.72}{11.11}\right) = (0.09,\, 0.67,\, 0.24)$$

In words: turn the three scores into three shares of one whole. We call each share $a_i$, "a" for attention weight. We avoid $w$, which this site reserves for learned weights, and these shares are computed on the fly. Said out loud: "sat" pays 67% of its attention to "cat," keeps 24% for itself, and gives 9% to "the." The shares sum to 1.00, which softmax guarantees. We round every share to two decimals in this chapter, so 0.665 is shown as 0.67.

The payoff: mixing the values

Weighted values add to [2.25, 0.85].

A weighted average combines the values. Here $\mathbf{o}_{\text{sat}}$ names sat's output.

$$\mathbf{o}_{\text{sat}} = 0.09\,\mathbf{v}_{\text{the}} + 0.67\,\mathbf{v}_{\text{cat}} + 0.24\,\mathbf{v}_{\text{sat}} = 0.09\begin{bmatrix} 0 \\ 2 \end{bmatrix} + 0.67\begin{bmatrix} 3 \\ 1 \end{bmatrix} + 0.24\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 2.25 \\ 0.85 \end{bmatrix}$$

In words: take 9% of the's payload, 67% of cat's, and 24% of sat's own, and add the three scaled arrows.

Worked example: the two slots

First slot: $0.09 \times 0 + 0.67 \times 3 + 0.24 \times 1 = 0 + 2.01 + 0.24 = 2.25$.

Second slot: $0.09 \times 2 + 0.67 \times 1 + 0.24 \times 0 = 0.18 + 0.67 + 0 = 0.85$.

Stack them: $\mathbf{o}_{\text{sat}} = [2.25, 0.85]$.

The result points near cat's value because that value supplied the largest share. We used rounded shares throughout; the displayed result follows those rounded numbers. A later residual addition will add this attention output back onto the vector that came in, so nothing the token already carried is thrown away.

Where q, k, and v come from

Our invented input vectors are $\mathbf{v}_{\text{the}}=[1,0,0]$, $\mathbf{v}_{\text{cat}}=[0,1,0]$, and $\mathbf{v}_{\text{sat}}=[1,1,1]$. Three learned matrices, W_Q, W_K, W_V, turn an input into its role vectors. The following calculation labels that input x to distinguish it from the value v.

The three lenses

Each grid is $2 \times 3$: it takes a three-number embedding and produces a two-number role vector. Real models compress the same way, a long embedding in and a shorter query out.

$\mathbf{W}_Q = \begin{bmatrix} 2 & 0 & -1 \\ 0 & 2 & -1 \end{bmatrix}$, $\mathbf{W}_K = \begin{bmatrix} 1 & 1 & -2 \\ -1 & 1 & 1 \end{bmatrix}$, $\mathbf{W}_V = \begin{bmatrix} 0 & 3 & -2 \\ 2 & 1 & -3 \end{bmatrix}$.

$$\mathbf{q}_{\text{sat}} = \mathbf{W}_Q\,\mathbf{x}_{\text{sat}} = \begin{bmatrix} 2 & 0 & -1 \\ 0 & 2 & -1 \end{bmatrix} \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$$

In words: row 1 dots $[2, 0, -1]$ with $[1, 1, 1]$ to give $2 + 0 - 1 = 1$; row 2 dots $[0, 2, -1]$ with the same input to give $0 + 2 - 1 = 1$. So $\mathbf{q}_{\text{sat}} = [1, 1]$, the query we have been using all along. The other eight products work the same way; spot-check any one against the cast table and it lands on the table's numbers.

Each matrix is 2×3: three inputs, two outputs. Training adjusts these projections; the score-softmax-mix recipe stays fixed.

Everyone asks at once

Stack queries as rows of Q and keys as rows of K. A transpose, written $\mathbf{K}^\top$, puts keys in columns. It turns a 3×2 matrix into a 2×3 matrix without changing the entries.

Two 3x3 grids of attention shares, cells brighter for larger shares, rows labeled 'asking' (the, cat, sat) and columns labeled 'being scored' (the, cat, sat). Left panel 'everyone sees everyone': row the 0.47, 0.47, 0.06; row cat 0.01, 0.50, 0.50; row sat 0.09, 0.67, 0.24. Right panel 'no peeking forward': the upper-right triangle cells are blanked to dark with muted dashes, and the kept rows read the 1.00; cat 0.02, 0.98; sat 0.09, 0.67, 0.24. — Each row divides one token’s attention.

Each row of Q dots each column of K-transpose. That makes nine scores, with askers as rows and matched tokens as columns:

$$\mathbf{Q}\mathbf{K}^\top = \begin{bmatrix} 2 & 0 \\ 0 & 2 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 & 0 \\ -1 & 1 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 2 & 0 \\ -2 & 2 & 2 \\ 0 & 2 & 1 \end{bmatrix}$$

In words: each row of the answer is one token's three scores. Spot-check the corner you already know. Row 3, column 2 is $\mathbf{q}_{\text{sat}} \cdot \mathbf{k}_{\text{cat}} = 2$, the score we computed by hand.

Softmax each row; then compare the masked case
Asker	All tokens visible	Only current and earlier tokens
the	0.47 / 0.47 / 0.06	1 / 0 / 0
cat	0.01 / 0.50 / 0.50	0.02 / 0.98 / 0
sat	0.09 / 0.67 / 0.24	0.09 / 0.67 / 0.24

The causal mask excludes later tokens before softmax. Recompute each row's shares over permitted positions. Rounded rows can total 1.01; the unrounded shares total one. A negative score, such as cat's −2 against the, still receives a positive unmasked share.

The thermostat: dividing by √d_k

The key dimension d_k is 2 here. Under roughly independent, comparable-scale components, typical dot-product magnitude grows like $\sqrt{d_k}$. Dividing by that factor prevents large dimensions from making softmax excessively peaked.

Switch on the scale factor

Divide sat's scores by $\sqrt2\approx1.41$. They become approximately 0,1.41,0.71; exponentials 1,4.11,2.03 total about 7.14.

$$\mathrm{softmax}\!\left(\tfrac{0}{1.41},\, \tfrac{2}{1.41},\, \tfrac{1}{1.41}\right) = \mathrm{softmax}(0,\, 1.41,\, 0.71) = (0.14,\, 0.58,\, 0.28)$$

In words: shrink the scores toward each other, then soften as before. Compare side by side: 0.67 softens to 0.58, with the same ranking and a gentler blend. For the rest of the chapter we keep the unscaled $0.09 / 0.67 / 0.24$ shares, and the widget has a $\div\sqrt{d_k}$ switch so you can toggle between the two.

Scaled dot-product attention combines the operations:

$$\mathrm{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}\right)\mathbf{V}$$

In words, right to left: $\mathbf{Q}\mathbf{K}^\top$ is every question dotted with every key; dividing by $\sqrt{d_k}$ is the thermostat; softmax turns each row of scores into shares; and multiplying by $\mathbf{V}$ lets each row of shares mix the stacked value vectors, giving three weighted blends at once. That is the row-as-recipe reading of matrix multiplication from Chapter 3.

The displayed formula leaves the mask implicit; a causal model applies it to scores before softmax.

See it move

Watch (2:40): what to notice. Sat's mint question meets the three amber keys, the scores 0, 2, 1 become shares 0.09, 0.67, 0.24, and the blue values blend tip-to-tail into the new sat-vector. These are the numbers you just computed. The formula then assembles from those parts.

Why the LLM cares

In the decoder transformer we are building, attention is the only place where positions exchange information; every other part of the block works on one position at a time, and we assemble those parts in the next chapter. Real models also run several copies of this scoring-and-mixing step side by side, each with its own $\mathbf{W}_Q$, $\mathbf{W}_K$ and $\mathbf{W}_V$. Each copy is a head, and this chapter has followed one of them. Nobody assigns a head its job; the projections learn what to match.

For n=1,000 tokens, a full attention score grid has one million entries, and that is per head. During generation, caching lets a new token compute a new row against stored keys, rather than rebuilding every earlier row.

What you now know

Queries and keys produce relevance scores.
Softmax turns each row into mixing shares.
Those shares weight values, giving our toy output [2.25,0.85].
Learned matrices generate the three role vectors.
A causal mask blocks later positions.
Dividing by the square root of key dimension controls score scale.

Where we're headed

Attention is one part. Next we'll assemble the transformer: parallel heads, per-token processing, position information, and layers joined by a running stream.