The Math Toolkit

Five tools, then we build

Fitting a function takes five tools: hold data, compare inputs, transform them, find a downhill direction, and describe uncertainty. Each one gets a picture and arithmetic you can check.

This toolkit stands alone. Each tool links to the LLM book for the slower build.

Tool 1. Vectors: arrows and lists

A vector is a list of numbers we can draw as an arrow. Walk 3 blocks east and 2 north from the origin.

Vectors use bold lowercase letters. Our pair is $\mathbf{v}=[3,2]$ and $\mathbf{u}=[1,3]$; components $v_1,v_2$ are the entries. In the figures, the input arrow is mint and its partner amber.

Add matching components, or place the arrows tip-to-tail:

In words: line up the two lists, add the top numbers to get 4, add the bottom numbers to get 5, and the answer is the vector [4, 5]. That one trip lands where the two trips end up.

1. First tripWalk 3 east, 2 north.
2. Second tripFrom that tip, walk 1 east, 3 north.
3. Combined tripFinish 4 east, 5 north of the origin.

Scaling is multiplying a vector by a plain single number, a scalar. It stretches every component at once.

In words: double each number, so [3, 2] becomes [6, 4], an arrow twice as long pointing the same way. A positive scalar never changes the direction, which is why direction alone can carry information.

The length is the direct distance to the tip: the long side of a right triangle. Square each component, add, then take the square root.

In words: 3 squared is 9, 2 squared is 4, together they make 13, and the square root of 13 is about 3.61, so the arrow reaches about 3.61 blocks as the crow flies. The double bars around $\mathbf{v}$ are how we write "the length of v."

The same recipe gives $\lVert\mathbf{u}\rVert=\sqrt{1+9}=\sqrt{10}\approx3.16$. We need both lengths for cosine similarity next.

A vector with $d$ components uses the same slot-by-slot arithmetic in $d$ dimensions. Drag the widget’s arrows; the longer explanation is Numbers That Point.

A 300-dimensional vector is a list of 300 numbers. Every operation repeats the same arithmetic in each slot. The pictures stop at three dimensions; the arithmetic never changes.

Tool 2. The dot product: one number for agreement

The dot product answers one question with a single number: how much do these two things agree? If our inputs are arrows, the question becomes "do these two arrows point the same way?"

The recipe is to multiply matching components, then add. It is written with a center dot, $\mathbf{v} \cdot \mathbf{u}$, read "v dot u." On our running pair, every digit shown:

In words: multiply the top numbers (3 times 1 is 3), multiply the bottom numbers (2 times 3 is 6), add those together, and get 9. A positive answer means the arrows agree: they point the same general way.

Matching signs contribute positively; opposite signs contribute negatively. Large components contribute more, so the total depends on both direction and length.

The lengths we just measured refine the comparison. The raw dot product grows with length as well as with alignment, so to compare pure directions we divide out both arrows' lengths and get cosine similarity, a score locked between −1 and +1. The dot product 9 divided by both lengths, $3.61 \times 3.16 \approx 11.4$, gives about 0.79. The arrows point the same general way.

1. Dot product$3×1+2×3=9$.
2. Remove length$3.61×3.16≈11.4$.
3. Compare directions$9/11.4≈0.79$.

The full treatment (projection, the geometry, why the answer is a cosine at all) is the sister site's Measuring Similarity. The general form of the recipe is the two products added, $\mathbf{v} \cdot \mathbf{u} = v_1 u_1 + v_2 u_2$.

Perpendicular arrows have a zero dot product: no agreement and no opposition. Opposite arrows point along the same line in reverse and give a negative dot product. “Unrelated to X” and “the opposite of X” are different, and the sign tells them apart.

Tool 3. Matrices: machines that transform vectors

A matrix is a grid of numbers that is also a machine: feed it a vector, it hands back a new vector. This is where a model's learned numbers live. Our matrix is $\mathbf{W} = \left[\begin{smallmatrix} 1 & 2 \\ 3 & 0 \end{smallmatrix}\right]$. In the figures below it is drawn in violet, because weights and parameters are always violet on this site.

In words: a two-by-two grid, meaning two rows and two columns, rows named first. The four numbers are the machine's settings.

Matrices use bold uppercase letters; entries are addressed by row $i$, then column $j$. A 4096-by-4096 matrix follows the same recipe.

The procedure runs on Tool 2. Feed $\mathbf{W}$ the mint vector $\mathbf{v} = [3, 2]$. Each row of $\mathbf{W}$ takes a dot product with the input: row 1's dot product becomes output slot 1, row 2's becomes output slot 2.

In words: row 1 is [1, 2], and [1, 2] dot [3, 2] is 3 plus 4, which is 7. Row 2 is [3, 0], and [3, 0] dot [3, 2] is 9 plus 0, which is 9. Stack the answers and the output vector is [7, 9]. A matrix does dot products in bulk, one per row.

Each row gives the mixing proportions for one output; these entries are the model’s weights, adjusted during training.

Feed a matrix the whole plane instead of one vector and it warps the grid, stretching, leaning and rotating space, while straight lines stay straight and the origin stays put. Different entries give a different warp. The slow build (the three faces of a matrix, rotation, the columns trick) is the sister site's Grids That Transform.

Tool 4. Derivatives: which way is downhill

To walk downhill in fog, feel the ground’s local tilt. Training uses the same idea to reduce error.

A slope is rise divided by run: one step right and two up gives $2/1=2$. Positive slopes rise, negative slopes fall, and zero means locally flat.

For $f(x)=x^2$, the derivative $\frac{df}{dx}$, read “the derivative of f with respect to x,” gives the slope at one point:

In words: the slope at any point is twice the position. At $x = 1$ the slope is 2, at $x = 3$ it is 6, and at $x = 0$ it is 0. Slope zero is flat, and flat is where a valley bottoms out.

Turning that tilt into a walk is the rule behind training. Step opposite the slope: your new position is your old position minus a small step-size times the slope. That step-size is the learning rate, written $\eta$, the Greek letter eta.

In words: to get your next position, take the slope where you stand, shrink it by the learning rate, and subtract it. The minus sign is what sends you downhill instead of up.

Worked example: one descent

Take $f(x) = x^2$, start at $x = 4$, and use a learning rate of $\eta = 0.25$. That is a quarter of the slope per step, big enough to make visible progress and small enough that a slope as steep as 8 does not throw us clean over the valley.

1. Point$x=4$, $f(x)=16$; use $\eta=0.25$.
2. Slope$2×4=8$.
3. Step$−0.25×8=−2$.
4. New point$4−2=2$; next: slope $4$, step $−1$, point $1$, then $0.5$.

In words: the steps shrink on their own, because the slope shrinks as the ground flattens near the bottom.

A gradient, written $\nabla L$ and read “the gradient of the loss,” collects one slope per parameter. Loss measures wrongness. Subtract the scaled gradient component by component. For the slow build, read Slopes and Descent.

Tool 5. Probability: belief spread across outcomes

A good model rarely gets to be certain. Asked "cat or dog?" about a blurry photo, it should answer with a spread: "80 percent cat, 20 percent dog." Probability is the language for that spread.

A bag holds 10 marbles: 5 mint, 3 amber, 2 violet. You draw one without looking. The probability of drawing mint is the share of mint marbles:

In words: five of the ten marbles are mint, so the chance of mint is five-tenths, or 0.5. The same counting gives $p(\text{amber}) = 0.3$ and $p(\text{violet}) = 0.2$.

Two rules govern every distribution. Every probability sits between 0 and 1, and across all the outcomes they add up to 1. You have one unit of belief to divide among the outcomes, and no way to make more of it.

In words: add the three shares, a half plus three-tenths plus two-tenths, and they land on 1, one whole unit of belief accounted for.

The full list of outcomes with their shares is a probability distribution, and it draws as a bar chart whose heights sum to 1. The probability of something given what you already know, written with a vertical bar and read "given," can differ completely from the plain probability; that is conditional probability. And a distribution can be turned into one concrete choice by sampling, which means rolling a weighted die whose faces match the bars.

1. DistributionMint 0.5, amber 0.3, violet 0.2.
2. SampleDraw one marble without looking.
3. One outcomeAmber can win this draw even though mint is more likely.

The slow build (belief, conditioning, the lumpiness of small samples) is the sister site's Chance and Choice. Chapter 4 gives a model a way to turn a raw score into a probability like these, and multi-class prediction spreads belief across many bars at once.

Where you'll meet this