Grids That Transform · How LLMs Actually Work

Where the weights live

A model has billions of weights. What are they, and where do they live? Many are numbers in a matrix. As with a vector, we get two views: stored numbers and a geometric action. Let's open a machine with four entries.

A table with a job

Subscripts give row first, then column: W₁₂ = 2; W₂₁ = 3.
W	Column 1	Column 2
Row 1	1	2
Row 2	3	0

Bold uppercase $\mathbf{W}$ means “the matrix W.” Its shape is 2×2: two rows, two columns.

$$\mathbf{W} = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}$$

In words: W is the four numbers 1, 2, 3, 0, laid out in two rows and two columns. Top row 1 then 2, bottom row 3 then 0.

A row runs across; a column runs down. The entry $W_{ij}$ is in row i, column j. Thus $W_{12}=2$ but $W_{21}=3$. Row first, always.

The weights are these adjustable numbers.

The machine runs: matrix × vector

Feed in $\mathbf{v}=[2,1]$. Each row dots the whole input; its answer becomes one output component.

Matrix rows [1,2] and [3,0] each dot the whole input [2,1], giving outputs 4 and 6. — Each row produces one output component.

Worked example

Row 1 of W is [1, 2]. Dot it with the input [2, 1]: multiply matching parts and add, $1 \times 2 + 2 \times 1 = 2 + 2 = 4$. That is output slot 1.

Row 2 of W is [3, 0]. Dot it with the same input [2, 1]: $3 \times 2 + 0 \times 1 = 6 + 0 = 6$. That is output slot 2.

Stack the two answers and you have the output vector, $[4, 6]$.

$$\mathbf{W}\mathbf{v} = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix} \begin{bmatrix} 2 \\ 1 \end{bmatrix} = \begin{bmatrix} 1 \times 2 + 2 \times 1 \\ 3 \times 2 + 0 \times 1 \end{bmatrix} = \begin{bmatrix} 4 \\ 6 \end{bmatrix}$$

In words: the top row of W dots the input to make the top output number, 4; the bottom row dots the same input to make the bottom output number, 6. The result is the new vector [4, 6].

We write $\mathbf{W}\mathbf{v}$ without a centered dot. The output is a vector: one scalar answer per row. A 4096×4096 grid performs 4096 such dot products.

Read the dimensions before calculating
Object	Size	Job
Input	2 entries	The values every row reads.
Weights	2 rows × 2 columns	One multiplier per input per output.
Output	2 entries	The two row totals, [4,6].

Rows are recipes

Row 1 takes one part of input 1 and two parts of input 2. Row 2 takes three parts of input 1 and zero of input 2. Changing that zero makes the second output respond to the second input. Weights can be negative and need not total one.

The whole plane moves

A linear transformation applies the same matrix to every point. Lines remain straight, though a singular matrix can collapse them. The origin stays fixed.

Basis arrows become matrix columns.

The basis vectors reveal the whole transformation. W sends [1,0] to [1,3], its first column, and [0,1] to [2,0], its second. Any input is a scaled sum of those basic arrows.

Worked example: the do-nothing machine

The matrix $[[1, 0], [0, 1]]$ dots its rows with $\mathbf{v} = [2, 1]$: row 1 gives $1 \times 2 + 0 \times 1 = 2$, row 2 gives $0 \times 2 + 1 \times 1 = 1$. The output is $[2, 1]$, the input unchanged. Its columns say "$[1, 0]$ stays put, $[0, 1]$ stays put," so it leaves the whole plane as it found it. This is the identity matrix.

Two equal-length arrows show [2,1] rotated 90 degrees counterclockwise to [−1,2] by [[0,−1],[1,0]]. — A quarter-turn preserves length.

Worked example: the quarter-turn machine

The matrix $[[0, -1], [1, 0]]$ on the same $\mathbf{v} = [2, 1]$ gives $0 \times 2 + (-1) \times 1 = -1$ for the first slot and $1 \times 2 + 0 \times 1 = 2$ for the second, giving the output $[-1, 2]$. That is $[2, 1]$ rotated a quarter-turn counterclockwise, with the same length and a new direction. Note the parentheses around $(-1)$, the first negative entry we have met. A loose minus sign here trips people, so keep it wrapped.

$$\begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix} \begin{bmatrix} 2 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \times 2 + (-1) \times 1 \\ 1 \times 2 + 0 \times 1 \end{bmatrix} = \begin{bmatrix} -1 \\ 2 \end{bmatrix}$$

In words: the top row dots the input to give $0 \times 2 + (-1) \times 1 = -1$, and the bottom row gives $1 \times 2 + 0 \times 1 = 2$, so the arrow [2, 1] is carried to [−1, 2], the same arrow turned a quarter-circle.

Four transformations to recognize
Matrix	Effect
[[1,0],[0,1]]	Identity: every vector stays put.
[[2,0],[0,2]]	Scale: double every vector.
[[0,−1],[1,0]]	Rotate: a 90° counterclockwise turn.
[[1,1],[0,1]]	Shear: add the up coordinate to the across coordinate.

Our W combines stretching and tilting. Rows tell us how to compute; columns tell us where the basic arrows land.

See it move

Watch the row arithmetic and the grid transformation describe the same machine. Then edit one entry and predict its effect.

Watch (1:29): what to notice. Each row of W leans in to dot the input, and when the whole plane transforms, the basis arrows land on W's columns, digit for digit.

What training changes

A parameter is one trainable number. Matrix entries are parameters; later we'll also meet added offsets. We collect all parameters under $\theta$, “theta.” In our figures, the learned weight grids are violet.

Why the LLM cares

A token's vector passes through many learned matrices. Each row mixes input features into an output feature; training adjusts those recipes. A large model contains billions of such parameters working together.

Matrices alone can stretch and rotate but cannot supply every shape a model needs. We also need bends between them, which is where functions enter the story.

What you now know

A matrix is both a grid and a transformation.
Read matrix entries row first, column second.
Each row dots the input to produce one output.
Columns show where the basic arrows land.
Weights are adjustable mixing coefficients.

Where we're headed

Stacking linear maps gives another linear map. Next we'll introduce functions and curves that let a network bend.