Fitting a Line

Which line is best?

Here are three measurements that almost line up: the points $(1, 1)$, $(2, 4)$, and $(3, 4)$. Your eye wants to draw a straight line through them, sloping up. But there are infinitely many lines you could draw, and no single one hits all three. Which is best — and could a machine find it without you ever telling it the answer? That question, answered honestly, is your first real piece of machine learning.

What we are about to do has a name: linear regression, the "hello world" of the whole field — the smallest example that still contains every moving part of how machines learn. Get it for real and you have the skeleton of everything from here to the largest models alive.

The plan in one sentence: first a model with a dial, then a way to measure its wrongness as a single number you can see, then the discovery that all those numbers form a smooth bowl, and finally the downhill walk from the toolkit chapter that finds the bottom of that bowl on its own. Let us make the question vivid before any math.

A model with a dial

Start with the simplest function that can fit rising data: a line through the origin, where the prediction is just the slope times the input. Writing the model's guess as $\hat{y}$ — read "y-hat, the model's guess" — the model is

In words: to guess, multiply the input by the dial $w$. That single number $w$ is the slope, and it is the one thing we get to adjust — a weight.

Turning the dial changes the whole line's steepness at once. Watch what three settings do to our three inputs $x = 1, 2, 3$:

So here is the whole game in one sentence: learning this model means finding the one value of the dial $w$ that makes the predictions closest to the real values. To do that, we first have to say what "closest" means with a number.

One honest aside, planted for later: a line through the origin is a deliberate simplification — it always passes through $(0, 0)$. Real lines also have a height, an intercept. We add that second dial near the end; for now, one dial keeps every number checkable by hand.

Wrongness you can see

Before we make the error a number, let us make it visible. For a given line, each data point sits some vertical distance above or below the line — that gap is the residual, the model's miss on that point. We draw residuals as red segments, because on this site red always means error.

Now turn those misses into one number, confronting the design choices out loud. We cannot just add the residuals up: overshoots (positive) and undershoots (negative) would cancel, and a wildly wrong line could score zero. The fix is to square each residual first. Squaring makes every miss positive and punishes big misses far more than small ones — a miss of 2 costs 4, while a miss of 1 costs only 1. Then average over the points to get one score. That score is the mean squared error, our loss $L$.

One piece of notation first, because it turns up in nearly every equation from here on. That tall symbol $\sum$ is a capital Greek sigma, and it means nothing more alarming than "add these up." Written $\sum_{i=1}^{N}$, it says: let $i$ count through the examples — 1, then 2, and on up to $N$ — work out whatever follows it once per example, and add the results. It is a for-loop wearing a robe. The little subscripts do the same counting job: $x_i$ and $y_i$ mean "the input and the real value of example number $i$." For our three points, $\sum_{i=1}^{3}$ is just example 1's answer plus example 2's plus example 3's.

Written for our $N$ examples — here $N = 3$ — the loss is the average over the points of (prediction minus truth) squared:

In words: for each point, take how far off the guess was, square it, and average those squared misses. Notice the loss depends on the dial — a different $w$ gives a different wrongness — which is why we write it $L(w)$.

Look at the pattern, because it gives away the whole ending. The dial at $w = 1$ and the dial at $w = 2$ are equally wrong — both score $1.67$ — while $w = 1.5$ sits between them and is less wrong, at $0.5$. The wrongness dips in the middle. The next section reveals the shape that dip belongs to.

The loss is a bowl

Let us do what that last pattern teased: compute the loss for many settings of the dial and plot $L$ against $w$. The points do not scatter — they fall on a smooth U-shaped curve, a parabola (the very shape from the toolkit's derivative section), with a single lowest point.

Read the landmark heights straight off the bowl, all hand-checkable by the same recipe: $L(0) = 11$ (a flat line stuck at zero, badly wrong), $L(1) = 1.67$, $L(1.5) = 0.5$ (the bottom), $L(2) = 1.67$, and $L(3) = 11$. The floor sits at $L = 0.5$ — not zero, because no straight line through the origin can hit all three points at once. The best available line still misses a little, and that is fine.

This curve has a name that scales to the entire field: it is a loss landscape. Right now it is a one-dimensional bowl because we have one dial. Add more dials and it becomes a landscape in more dimensions — a valley, then a mountain range — but the goal never changes: find the lowest point. Training is landscape-descending.

That is what makes the ending inevitable. The best line is the bottom of the bowl, where the ground is flat — and the toolkit chapter handed us the exact tool for finding flat ground by feel. We do not need to see the whole bowl or solve anything clever; we can walk down it.

Rolling downhill to the answer

Bring back the toolkit's downhill rule and aim it at the loss. To step downhill on the bowl, we need the slope of $L$ at our current dial — the derivative $\frac{dL}{dw}$ — and then we step against it. Re-naming the Greek letter as the house rules ask, $\eta$ is eta, the learning rate, which sets how big each step is:

In words: to get the next dial setting, take the slope where you stand, shrink it by the learning rate, and subtract — the minus sign is what sends you downhill instead of up.

We can build that slope from the residuals — no calculus machinery, just the pattern. The slope of $L$ at a given $w$ is

In words: for each point multiply its input $x_i$ by its signed error (prediction minus truth), add those up, and scale by $2/N$. Points we are far off on, especially at large $x$, pull the slope the most.

One careful note, so nobody trips: this uses the signed error, prediction minus truth — the opposite orientation to the residual we defined earlier as truth minus prediction. Squaring hid that sign, but the slope needs it, so we keep it explicit here.

Where did that formula come from? Two honest pieces, both already in your hands. The $2$ out front is the toolkit's $x^2 \to 2x$ pattern: the loss squares each miss, and the slope of a square is twice the thing being squared, so every miss contributes twice its own size. And each error is weighted by its own input $x_i$ because the dial reaches every prediction through that point's input — nudge $w$ a little and the prediction at $x = 3$ moves three times as far as the prediction at $x = 1$, so the far-out point gets three times the say in which way we step.

And you can check the whole thing against the bowl we already drew. At the bottom, $w = 1.5$, the slope had better read zero. The signed errors there are $-0.5$, $+1$, $-0.5$; weight each by its input and add them: $1(-0.5) + 2(1) + 3(-0.5) = -0.5 + 2 - 1.5 = 0$. Flat, exactly where the bowl bottoms out. The formula and the picture agree.

The rest of the walk we summarize, since the full arithmetic is shown once above. From $w = 1.4$ the slope is about $-0.93$, nudging the dial to $1.49$; from $1.49$ the slope is about $-0.06$, nudging it to $1.50$. The steps shrink on their own as the ground flattens, and $w$ glides into $1.5$ — the bottom of the bowl, the best line — while the loss falls from $11$ to $0.5$. Nobody told the machine that $1.5$ was the answer; it found it by rolling downhill.

Sit with that for a second: this is machine learning. A model with a dial, a number for its wrongness, and a downhill walk that tunes the dial until the wrongness bottoms out. The exact same three parts — model, loss, gradient descent — train a network with a trillion dials. What you just did by hand is what a data center does at scale.

From one dial to two — and to a trillion

Time to pay the debt from earlier: real lines have a height as well as a slope. We add a second dial, the intercept $b$, so the model becomes

In words: multiply the input by the slope dial, then add the height dial $b$. Now the machine can slide the whole line up and down as well as tilt it.

The tools stretch to fit without changing shape. With two dials, the slope of the loss is now two numbers — how $L$ changes as we nudge $w$, and how it changes as we nudge $b$ — stacked into a little vector, the gradient $\left[\frac{dL}{dw}, \frac{dL}{db}\right]$. The update rule is unchanged in spirit: step both dials against their own slopes at once. This is exactly the gradient from the toolkit, now earning its keep.

Reassure yourself with our own numbers: for this data the best intercept turns out to be $b = 0$ — the ideal line already passes through the origin, because the data's centroid, the average point (the inputs average to $2$, the real values average to $3$), sits at $(2, 3)$, which lies exactly on $y = 1.5x$. That is the tell that no vertical shift of the line would help. Adding the second dial does not change our answer here; it just proves the method generalizes. That $b = 0$ is a feature of these three points, not a general rule — for most data the best $b$ is nonzero, and the machine finds it by the same walk. (The widget below lets you switch on the intercept and watch this.)

Now scale the picture up: a neural network is this same story with millions or billions of dials instead of two — a model that makes a guess, a loss that scores it, and a gradient (now a very long vector) that says which way every dial should move. The bowl becomes a mountain range nobody can see, but the walk is identical. You have met the engine of the entire field.

See it move

The video runs the whole story in one motion: the red residuals shrink as the line tilts, every line's wrongness collapses into a single dot that traces out the bowl, and then the ball rolls down that bowl from $w = 0$ while the line snaps to the best fit — the same numbers as this chapter, with the loss falling $11 \to 0.5$ and the slope landing on $1.5$.

Now you hold the dials. Drag the line over the three points and feel the loss respond; then press "Let it learn" and watch gradient descent do to the dials exactly what your hand was doing — only guided by the slope instead of by your eye.

Where you'll meet this

Name the three parts you just built, because you will meet them in every remaining chapter of this site. A model (a function with dials), a loss (one number scoring its wrongness), and gradient descent (walking the dials downhill). Every method ahead is a choice of model and loss, trained by this same walk.

The next step is close by. Next chapter, Drawing Boundaries, keeps the model's score $\hat{y} = w \cdot x + b$ almost unchanged, but instead of reporting a number it squashes that score into a probability and asks "which class?" — turning this chapter's regression into classification. The line you just fit becomes the boundary between two categories.

The leap into Part II is the same idea again: an artificial neuron is literally this chapter's weighted sum followed by one of the next chapter's squashes; a deep network is many such neurons stacked, trained by exactly the gradient descent you ran by hand — just with backpropagation to compute the giant gradient efficiently. There is no new learning principle coming; there is only more of this one.

So "the machine learns" stopped being a metaphor in this chapter. You watched a dial find its own best value by feeling for downhill. Everything ahead is that same sentence, scaled up and dressed up — and now you know what is underneath the dress.