Attention

One frozen arrow isn't enough

Back in Chapter 8 every token arrived as exactly one arrow — and the same arrow every time that token appeared. Let's run that idea straight into a wall. Read these two sentences: "we fished from the river bank" and "we put the cash in the bank." One token, one frozen arrow — and two completely different meanings. The single arrow for "bank" is stuck pointing somewhere between rivers and money, which makes it wrong for both.

Here is the same wall in a different shape: "The cat knocked over the vase because it was fragile." What does "it" mean? Nothing, on its own — "it" is an empty pointer that has to find the vase somewhere earlier in the sentence. No frozen arrow could ever contain "whatever fragile thing this particular sentence happens to mention."

The fix, in one sentence before any math: let every word look at the other words in its sentence and rebuild its own vector out of what it finds. That looking-and-rebuilding mechanism is called attention, and it is the single invention this whole site has been climbing toward — the 2017 paper that launched modern LLMs is literally titled "Attention Is All You Need."

One warm word before the climb: this chapter is the summit of the site, and it uses zero new math. Dot products (Chapter 2), matrix × vector (Chapter 3), softmax (Chapter 9) — you own every tool already. We will run the whole mechanism by hand on a three-word sentence, every digit visible: the cat sat.

Three roles: query, key, value

Intuition first, machinery later. When words look at each other, every word plays three roles at once. It asks: "what am I looking for?" — that is its query. It advertises: "here's the kind of word I am" — that is its key. And it carries a payload to hand over if it gets picked: "here's what I'll actually tell you" — that is its value.

Ground it in our sentence. "sat" is a verb missing its subject, so its query says roughly "looking for a noun, someone who acts." "cat" is a noun, and its key says roughly "noun here, an animal, a doer." Good match — and when the match wins, what "cat" hands over is its value: the actual cat-ness that "sat" will absorb.

Why three separate roles? Because asking and advertising are different jobs. "sat" hunting for its subject is not the same as "sat" offering itself as a verb to someone else's question. The model learns a different lens for each role — three matrices, which we will meet in a few sections. For now, take the vectors as given.

Scores, then shares

Now run the match for "sat." Its query $\mathbf{q}_{\text{sat}} = [1, 1]$ meets every key in the sentence — including its own — and the meeting is a dot product, Chapter 2's one-number agreement score. Here is the loudest one worked in full:

In words: line "sat"'s question up against "cat"'s key, multiply matching slots, and add — a score of 2. The other two go the same way: $\mathbf{q}_{\text{sat}} \cdot \mathbf{k}_{\text{the}} = 1 \times 1 + 1 \times (-1) = 0$ and $\mathbf{q}_{\text{sat}} \cdot \mathbf{k}_{\text{sat}} = 1 \times 0 + 1 \times 1 = 1$.

Read the three scores as a story: "cat" answers sat's question loudest (2), "sat" itself murmurs something useful (1), and "the" contributes nothing to this particular question (0). Scores can be any size and any sign — a strongly mismatched pair would go negative, and one is coming two sections from now.

Raw scores are not yet usable. We want shares of sat's attention — positive numbers that total exactly 1, so they can weight a blend. You met the machine that does precisely this in Chapter 9: softmax. Exponentiate each score, then divide by the sum — $e^{0} = 1.00$, $e^{2} \approx 7.39$, $e^{1} \approx 2.72$, totaling $\approx 11.11$:

In words: turn the three scores into three shares of one whole. We call each share $a_i$ — "a" for attention weight; we avoid $w$, which this site reserves for learned weights, since these shares are computed on the fly, not learned. Said out loud: "sat" pays 67% of its attention to "cat," keeps 24% for itself, and gives 9% to "the." The shares sum to 1.00 — softmax guarantees it. We round every share to two decimals throughout this chapter, so the exact 0.665 becomes 0.67.

The payoff: mixing the values

Attention's output is a weighted blend of the values, using the shares as proportions. This is Chapter 1 arithmetic wearing its work clothes: scale each value vector by its share, then add the scaled arrows tip-to-tail. Call the output $\mathbf{o}_{\text{sat}}$ — "o" for output:

In words: take 9% of the's payload, 67% of cat's, and 24% of sat's own, and add the three scaled arrows.

Here is the click moment. Compare the output to the ingredients: $[2.25, 0.85]$ points almost exactly the way cat's value $[3, 1]$ points. The new sat-vector is two-thirds cat. The verb now carries its subject inside its own vector — "sat" has become "sat-by-a-cat." That is what "context-aware" means, mechanically: nothing more, nothing less.

Now zoom back to the chapter's opening wall. Run this same machinery on "river bank," and the money-cluster words earn near-zero shares while "river" gets a big one — the frozen arrow for "bank" gets blended toward rivers. In the other sentence, toward money. Same token, same starting arrow, different company, different final vector. The wall is down.

Where q, k, and v come from

Time to open the machinery we deferred. Each token arrives as its embedding (Chapter 8). In our pretend mini-model the embeddings are three-number vectors: $\mathbf{x}_{\text{the}} = [1, 0, 0]$, $\mathbf{x}_{\text{cat}} = [0, 1, 0]$, $\mathbf{x}_{\text{sat}} = [1, 1, 1]$ — invented for teaching, small enough to hand-check, and not claimed to encode real meaning geometry. Three learned matrices — $\mathbf{W}_Q$, $\mathbf{W}_K$, $\mathbf{W}_V$, drawn violet in the figure above because these are exactly the kind of weight grids Chapter 3 promised — turn each embedding into its three role-vectors: $\mathbf{q} = \mathbf{W}_Q\,\mathbf{x}$, $\mathbf{k} = \mathbf{W}_K\,\mathbf{x}$, $\mathbf{v} = \mathbf{W}_V\,\mathbf{x}$.

Let's work one product fully, digit by digit — rows dot the input, exactly Chapter 3's move:

In words: row 1 dots $[2, 0, -1]$ with $[1, 1, 1]$ to give $2 + 0 - 1 = 1$; row 2 dots $[0, 2, -1]$ with the same input to give $0 + 2 - 1 = 1$. So $\mathbf{q}_{\text{sat}} = [1, 1]$ — precisely the query we have been using all along. The other eight products are the same move; spot-check any one against the cast table and it lands on the table's numbers.

Now the significance. These three violet grids are where attention learns. Training never touches the recipe — score, soften, mix is fixed forever — it only adjusts the entries of $\mathbf{W}_Q$, $\mathbf{W}_K$, $\mathbf{W}_V$, reshaping what every token asks, advertises, and hands over. When people say a model "learned to resolve pronouns," they mean: gradient descent nudged grids like these until "it"-queries started matching antecedent-keys.

Everyone asks at once

So far we ran attention for one asker, "sat." In reality every token asks simultaneously — the, cat, and sat each dot their query against all three keys. That is nine dot products, arranged in a $3 \times 3$ grid: rows are askers, columns are the tokens being scored. This is the moment matrix notation earns its keep.

Stack the three queries as the rows of a matrix $\mathbf{Q}$ (3 rows, 2 columns) and the three keys as rows of $\mathbf{K}$. To dot every row of $\mathbf{Q}$ against every row of $\mathbf{K}$ with one grid multiplication, $\mathbf{K}$ must be flipped so its keys stand as columns. That flip is the transpose, written $\mathbf{K}^\top$. Transposing swaps rows and columns — a $3 \times 2$ grid becomes $2 \times 3$ — and that is all the little $^\top$ ever means.

One more small first: so far we have only multiplied a grid by a single vector. Grid × grid is nothing new — it just runs the matrix–vector recipe once per row of $\mathbf{Q}$. Entry $(i, j)$ of $\mathbf{Q}\mathbf{K}^\top$ is (row $i$ of $\mathbf{Q}$) · (key $j$): every question dotted with every key, filed in a table.

In words: each row of the answer is one token's three scores. Spot-check the corner you already know — row 3, column 2 is $\mathbf{q}_{\text{sat}} \cdot \mathbf{k}_{\text{cat}} = 2$, the score we computed by hand.

Read the new rows as stories. Row 1 (the, scores $[2, 2, 0]$): softmax gives shares $0.47 / 0.47 / 0.06$ — "the" splits its attention between itself and cat, a little article looking for its noun. Row 2 (cat, scores $[-2, 2, 2] \to 0.01 / 0.50 / 0.50$): there is our promised negative score — cat's query actively mismatches the's key, and softmax squashes it to a 1% share. Softmax is applied to each row separately, so each asker gets its own shares totaling 1 — and because we round to two decimals, a row can read 1.01, which we say once here and then let stand.

One rule of the game we must add. In an LLM playing Chapter 9's prediction game, a token may not peek at words that come after it — those don't exist yet when the prediction is made. So the upper-right triangle of the grid is blanked out before softmax, and each row's shares are recomputed from what remains. This is the causal mask: "the" can only see itself (share 1.00), "cat" sees the and itself ($0.02 / 0.98$), and "sat" — already the last word — keeps its familiar $0.09 / 0.67 / 0.24$.

The thermostat: dividing by √d_k

One engineering touch separates our toy from the real formula. Our queries and keys have $d_k = 2$ numbers each — $d_k$ is the length of the key vectors; ours is 2, a real model's might be 128 — so our dot products stay small. Make the lists thousands long and the sums of products grow: typical scores balloon, and softmax fed huge scores collapses into winner-take-all, one share near 1.00 and the rest near 0.00. You saw this exact behavior in Chapter 9 as low temperature — and a spotlight stuck as a laser pointer can't blend gently, which is the whole point.

The fix: divide every score by $\sqrt{d_k}$ before softmax. Why the square root, in one honest sentence: the typical size of a dot product grows in proportion to the square root of the list length, so dividing by $\sqrt{d_k}$ cancels that growth and keeps scores in softmax's comfortable range at any dimension. It is a built-in thermostat — the same mathematical move as Chapter 9's temperature knob, permanently set to keep the distribution healthy.

Recompute our sat row once, fully, with the thermostat on. Since $\sqrt{2} \approx 1.41$, the scores $(0, 2, 1)$ become $(0, 1.41, 0.71)$; then $e^{0} = 1.00$, $e^{1.41} \approx 4.11$, $e^{0.71} \approx 2.03$, totaling $\approx 7.14$:

In words: shrink the scores toward each other, then soften as before. Compare side by side: 0.67 softened to 0.58 — same ranking, gentler blend. For the rest of the chapter's prose we keep the unscaled $0.09 / 0.67 / 0.24$ spine, and the widget has a $\div\sqrt{d_k}$ switch so you can toggle between the two.

Now assemble the full formula — and notice you have personally computed every symbol in it:

In words, right to left: $\mathbf{Q}\mathbf{K}^\top$ is every question dotted with every key; dividing by $\sqrt{d_k}$ is the thermostat; softmax turns each row of scores into shares; and multiplying by $\mathbf{V}$ lets each row of shares mix the stacked value vectors — three weighted blends at once, which is exactly the recipe-reading of matrix multiplication from Chapter 3.

Close the loop with a sentence of awe kept honest: this one line is the heart of every modern LLM, and you can now read it the way you read arithmetic.

See it move

Why the LLM cares

Here is the forward thread at maximum voltage: attention is the only place in an entire LLM where tokens exchange information. Every other computation — the next chapter shows them all — happens to each position privately. Every pronoun resolved, every "the capital of France is…" completed by looking back at "France," all of it rides through grids of dot products exactly like our $3 \times 3$.

Scale the picture honestly. For a context of $n = 1{,}000$ tokens, the score grid has $1{,}000 \times 1{,}000 =$ one million entries — computed per head (Chapter 11 explains heads), per layer, for every single token generated. When you hear that long conversations get expensive, you are hearing this grid grow with the square of the context length. A clever cache (Chapter 18) keeps it affordable — that seed for now, no more.

Then the honest wonder: nobody hand-programs what queries should ask. Trained on enough text, some heads end up matching pronouns to antecedents, some track syntax, some watch the previous word — jobs discovered, not assigned, by nudging violet grids downhill (Part III's whole story).

And let's cash the site's oldest promissory note. Chapter 2 called the dot product "the operation an LLM performs more than any other." Now you know exactly where they all live: inside attention, asking, for every pair of words, the only question the machine knows how to ask — how much do these two agree?