Part I · Ch. 2 — Space That Stretches

Part I · Chapter 2 of 29

Space That Stretches

A matrix moves a whole space at once


Two arrows move the whole grid

Change one number in a matrix and a whole grid moves. You can already read a matrix’s rows and columns. Each column says where one arrow lands, and those two arrows decide where every point goes.

Start with (1,0), one step along the horizontal axis, and (0,1), one step along the vertical axis. These are unit arrows, each of length one. You reach (2,1) by taking two copies of the first arrow and one copy of the second. A linear map keeps that recipe when it moves the arrows. A transform is the rule that sends each input position to its output position.

Six x-y grids with their 2 by 2 matrices show identity, horizontal stretch, quarter-turn rotation, shear, reflection, and collapse onto a line.
The columns tell us where every point goes, including whether area collapses.

Read the six matrices as movements. The identity [[1,0],[0,1]] leaves both arrows in place. A scale [[2,0],[0,1]] stretches the horizontal direction. The rotation [[0,−1],[1,0]] makes a quarter turn. The shear [[1,1],[0,1]] slants vertical lines. The flip [[−1,0],[0,1]] reverses the horizontal direction. The last one, [[1,1],[0,0]], collapses the plane onto a line.

In every panel the first column gives the new position of (1,0) and the second gives the new position of (0,1). No point gets a movement of its own. Every point follows from those same two column arrows, so straight grid lines stay straight and parallel lines stay parallel, even where several points land on top of each other.

Move one point with the shear

Let the input x be (2,1). The first row gives 1×2+1×1=3. The second row gives 0×2+1×1=1. The result is (3,1). The unit arrows become (1,0) and (1,1), so two copies of the first plus one copy of the second also give (3,1).

A matrix-vector product runs that row calculation once per output. Let the matrix be W, the input vector be x, and the output vector be y. The subscript i picks an output row; j visits each input coordinate. The summation sign says to add the terms as j changes. It is a compact spelling of the arithmetic above.

$$ \mathbf{y}=\mathbf{W}\mathbf{x},\qquad y_i=\sum_j W_{ij}x_j $$

In words: each output is the sum of one matrix row multiplied entry by entry with the input.

One row gives one output

One row of W is a short list of multipliers. A dot product multiplies matching vector entries and adds the products. Before using it as an arithmetic shortcut, look at what it measures in the plane. Take the pair v=(3,2) and u=(1,3).

Vectors v = (3,2) and u = (1,3) on x-y axes have dot product 9. The signed shadow length is 9 divided by square root of 10; multiplying by the length of u recovers 9.
The dot product combines direction and length.

Drop v perpendicularly onto the line through u. The signed length of that shadow says how much of v points along u’s direction. The dot product is that signed length times u’s length. So it is not a distance: doubling u doubles the dot product even when the direction is unchanged.

A similarity is a stated way of comparing how alike two inputs are. A dot product can serve, but its size reflects both direction and magnitude. For direction alone, divide each nonzero vector by its length first. That division is called normalization. A zero vector has no direction to normalize.

Matching entries, then adding

For v=(3,2) and u=(1,3), v·u=3×1+2×3=9. The same rule extends to three entries: (1,2,3)·(4,5,6)=1×4+2×5+3×6=4+10+18=32. One product per matched pair, then one total.

The compact form uses i to visit matching entries in both vectors. The bold letters name whole vectors and the subscripts pick their individual numbers. This is the same matching-and-adding that produced one row of our transformed point, written now without fixing the number of entries.

$$ \mathbf{v}\cdot\mathbf{u}=\sum_i v_i u_i $$

In words: multiply matching entries of the two vectors, then add the products.

How long is an arrow?

A square root is the nonnegative number whose square equals its input. The norm of a planar vector is its length: square each coordinate, add, take the square root. For v that gives √(3²+2²)=√(9+4)=√13. For u it gives √(1²+3²)=√10. The notation $\|\mathbf{v}\|$ names that length, and the same sum-of-squares rule extends to longer lists.

A positive dot product says the two point somewhat the same way, zero says they are perpendicular for nonzero planar vectors, and a negative value says one has a component opposing the other. A model still has to learn which transforms and comparisons help its task.

Area changes; translation needs an extra entry

The unit square measures how much a transform expands or compresses space. Its determinant is the signed area factor. For a two-by-two matrix with rows (a,b) and (c,d), calculate ad−bc. The magnitude is the area factor, and a negative sign records a flip of orientation. A zero determinant means the square’s area collapses.

Two grids compare a shear with a shear shifted by (1, −1). A 3 by 3 constant-coordinate matrix maps (2,1,1) to (4,0,1), using the final 1 to add the bias.
A bias shifts the origin; an ordinary matrix-vector product cannot.

Our stretch has determinant 2×1−0×0=2, so twice the area. The flip has determinant −1×1−0×0=−1: same area, reversed orientation. The collapse has determinant 1×0−1×0=0. Each statement is about a whole small patch of the plane, not about how far one chosen point traveled.

Those two-by-two matrices cannot do one thing: shift the origin. Multiply any row by (0,0) and every product is zero. To move the origin, add a bias, the same offset for every input. A linear map followed by a translation is an affine map.

Add the shift explicitly

Use bias b=(1,−1). The shear sends (2,1) to (3,1), then the bias sends it to (4,0). Append a constant 1 to the original input and use [[1,1,1],[0,1,−1],[0,0,1]] on (2,1,1). The outputs are 2+1+1=4, 0+1−1=0, and 1. The final column multiplies that constant and contributes the bias.

We will usually keep the bias as its own vector, because that leaves the stored parameters easier to inspect. The appended coordinate is why the two descriptions agree. Let y name the final output, W the warp, x the original point, and b the shared translation.

$$ \mathbf{y}=\mathbf{W}\mathbf{x}+\mathbf{b} $$

In words: move the point with the matrix, then shift every result by the same bias vector.

Order changes the result

Composition means applying one transformation after another. It can look as though two movements should commute, so that their order makes no difference. One point settles it. Let A stretch the horizontal direction by two and let B shear the plane. Both preserve straight lines, yet they do different things to a slanted direction.

Matrix columns move the whole grid; bias moves its origin.

See it move

Start with the shear and bias off. Drag a column arrow, then look at the two matrix entries that changed. Type those entries in directly and check that you get the same picture. Switch between the grid, the small center-square picture, and the 300-point teaching lattice. The original coordinates stay fixed; each redraw applies the new transform to those original points.

Reverse the order

Let A=[[2,0],[0,1]] and B=[[1,1],[0,1]]. Starting at (1,1), B then A gives (2,1) then (4,1). A then B gives (2,1) then (3,1). Their combined matrices are AB=[[2,2],[0,1]] and BA=[[2,1],[0,1]]. In AB, the right-hand matrix B acts first.

Turn the bias controls on and repeat the comparison. The second transformation now moves the warped point and the earlier translation together. That is why a combined affine transformation carries both a combined matrix and a combined bias. The live readout shows both beside the selected point, so you can check the picture against the arithmetic.

Where this shows up when you train

Every learned matrix changes how a model arranges its inputs. The grid is a way to see that change without pretending the model stores literal arrows. It stores entries; the arrows show what those entries do. Two columns cover our two-coordinate example, and larger matrices run the same row-by-row recipe over more coordinates.

When you debug a learned transform, feed it a unit input and read the matching column. Then feed it one mixed input such as (2,1). Small probes like these tie a weight table to behavior you can see. They also catch a transposed or reordered table far faster than staring at a large block of entries.

What you now know

  • Matrix columns are the transformed unit arrows.
  • A determinant measures signed area scaling.
  • Bias adds a translation, and composition order can change the result.

Where we’re headed

A learned layer warps, then shifts, then bends. Next comes its wiring diagram beside the matrix, and the arithmetic to go with it. Continue to the next chapter.