Grids That Transform

Where the weights live

Every article about large language models eventually says the same mysterious thing: the model has billions of learned weights. Billions of what, exactly, and living where? Here is the answer, up front and unglamorous: they are plain numbers, arranged in rectangular grids — and each grid is a little machine. This chapter opens one such machine, small enough to hold in your hand.

The object has a name. A matrix is a rectangular grid of numbers we treat as a single thing — and, at the same time, a machine that turns one vector into another. That is the same double life we met in Chapter 1, where a vector was an arrow and a list at once. A matrix is that idea one level up: a table and an action.

Set your expectations low, on purpose. Everything here happens with a single 2×2 grid and the vectors you already own. No new arithmetic is arriving — a matrix runs entirely on the dot product you built last chapter. Let's meet it.

A table with a job

Picture the star of the chapter before we say a word of notation. It is a 2×2 grid of four numbers, and we will keep it fixed all the way through — the same grid in the prose, the figures, the video, and the playground below.

Now the notation, which we keep for the rest of the site. A matrix is written as a bold uppercase letter — $\mathbf{W}$, read aloud as "the matrix W." Our grid is:

In words: W is the four numbers 1, 2, 3, 0, laid out in two rows and two columns — top row 1 then 2, bottom row 3 then 0.

We pick out a single number with two subscripts: $W_{ij}$ means the entry in row $i$, column $j$ — read "the entry in row i, column j." Row first, column second, always. The two entries people most often swap are the off-diagonal ones, so confront them side by side: $W_{12} = 2$ sits in row 1, column 2 (top-right), while $W_{21} = 3$ sits in row 2, column 1 (bottom-left). Same digits in the subscript, opposite corners of the grid — read them slowly the first few times and the habit sticks.

One more piece of vocabulary, in a single sentence: this is a 2×2 matrix — rows times columns, rows named first. Real models use grids shaped like 4096×4096, but that changes nothing about how the machine works; it only changes how much patience you would need to run it by hand.

The machine runs: matrix × vector

Time to feed the machine. Our input is the vector $\mathbf{v} = [2, 1]$ — the mint arrow, because input vectors are always mint on this site. The rule, in plain words before any symbols: each row of W takes the dot product with the whole input. Row 1's dot product becomes output slot 1; row 2's dot product becomes output slot 2. One dot product per output number — that is the entire operation.

Now do it by hand, every digit — this small calculation is the spine of the whole chapter.

Written as one display equation, the whole machine in a line:

In words: the top row of W dots the input to make the top output number, 4; the bottom row dots the same input to make the bottom output number, 6. The result is the new vector [4, 6].

That last move — "multiply matching parts, add" — is exactly what you did all through Chapter 2. A matrix is nothing more than dot products in bulk: two rows here, two dot products, two output slots. A 4096×4096 weight grid is 4096 dot products — honestly all the core multiplication in a neural-network layer is.

We write this operation $\mathbf{W}\mathbf{v}$ — matrix on the left, vector on the right, no dot symbol between them — and read it "W times v" or "W applied to v." Notice what came out: the result is a new vector, not a single number. Each row contributed one number, and those answers stack into a column.

Rows are recipes

Here is the same arithmetic read a second way — as weighted mixing. Look at output slot 1 again: $1 \times 2 + 2 \times 1$. Read it as a recipe: "take 1 part of the input's first number and 2 parts of its second, and stir." Row 2's recipe is "$3$ parts of the first number, $0$ parts of the second" — row 2 completely ignores the input's second slot, because its weight there is 0. Change that 0 to something else and suddenly row 2 starts paying attention.

Say it in general, still in words, no new equation needed: every output number is a weighted blend of all the input numbers, and the matrix entries are literally the blend proportions. "Weights" is not a loose metaphor here — the entries really do weight the mixture. That is why the whole field calls them weights.

One sentence of foreshadowing: when a model "learns," what changes is exactly these proportions — the recipe numbers are the knowledge, and nothing else in the machine is adjustable. How they get adjusted is Part III's whole story.

The whole plane moves

Now zoom all the way out, to the third and most beautiful view. Instead of feeding W one vector, feed it every point of the plane at once. The square reference grid warps into a new grid of leaning parallelograms: straight lines stay straight, the origin stays exactly where it was, but the whole plane stretches and tilts. A matrix is a transformation of space.

That last sentence hides the chapter's neatest secret. Where do the two most basic arrows land? Take $[1, 0]$ and run it through W: its dot products are $1 \times 1 + 2 \times 0 = 1$ and $3 \times 1 + 0 \times 0 = 3$, so it lands on $[1, 3]$ — exactly the first column of W. And $[0, 1]$ lands on $[2, 0]$, the second column. A matrix's columns are exactly a list of where the plane's basic arrows end up — so you can read its whole geometric behavior straight off its columns, without computing anything.

To feel how much the four numbers matter, meet two celebrity matrices.

Written out, the rotation looks like this:

In words: the top row dots the input to give $0 \times 2 + (-1) \times 1 = -1$, and the bottom row gives $1 \times 2 + 0 \times 1 = 2$, so the arrow [2, 1] is carried to [−1, 2] — the same arrow, turned a quarter-circle.

So we now have three ways to see the very same four numbers. Dot-products-by-rows tells you how to compute the output. Recipes tells you what the numbers mean. Space-transformer tells you what the machine does to everything at once. Same grid, three faces.

See it move

The three views are easier to feel in motion than to read. The short video runs the machine both ways: watch each row of the violet grid lean in to dot the input, then watch the same matrix pick up the entire plane and carry it.

Now take the controls. The playground lets you edit W's four numbers, drag the mint input around, and watch the arithmetic, the amber output, and the whole warped plane update together.

Why the LLM cares

This is the chapter where the site's title starts to cash out. An LLM is, to a first honest approximation, a very long pipeline of matrix-times-vector operations. A token enters as a vector (we build that in Chapters 7–8) and is passed through weight matrix after weight matrix, layer after layer. Everything you just did to $[2, 1]$ by hand is what happens to a word's vector billions of times as the model reads.

That makes "billions of parameters" suddenly literal. A parameter is one matrix entry. When a spec sheet says a model has 7 billion parameters, it means: count the entries of all its weight grids, one by one, and you reach 7 billion. We bundle all of them under a single symbol, $\theta$ — theta, read "the parameters" — a letter you will see from Chapter 12 onward. We will not do any arithmetic with $\theta$ here; just meet the name.

The site's color language now pays off too: weights are violet in every figure and video from here on. When you see violet, you are looking at the learned, adjustable part of the machine — the numbers training gets to change.

And each of our three views previews a later chapter, one sentence each. The rows view is how attention will score words against each other. The mixing view is how a transformer layer blends information across features. The space view is how embeddings get rotated and reshaped as they rise through the layers. Every later chapter in this site is, at heart, this chapter with more rows.