Which line is best?
Three measurements almost line up: $(1,1)$, $(2,4)$, and $(3,4)$. Your eye wants to draw a rising line through them, but no line hits all three. Which one is best, and could a machine find it without being told the answer?
Your eye may prefer the steeper candidate, but that preference needs a score before a machine can reproduce it. The three measured outputs are the answers we train against. Choosing a model and an error rule is our job; finding a good setting within that model is the machine’s job.
Linear regression answers that question by adjusting a line to reduce its errors. It is the smallest example that contains the model, loss, and update used to train much larger networks.
We give the model a dial, measure its wrongness, and use the downhill walk from the toolkit to improve it. Start with what the line can and cannot do.
- Model: one shared slope makes predictions.
- Loss: score their combined misses.
- Update: use the loss slope to improve the weight.
A model with a dial
Start with a line through the origin. Its prediction $\hat{y}$, read “y-hat, the model’s guess,” is the slope times the input:
$$\hat{y} = w \cdot x$$In words: multiply the input by dial $w$, the slope and our adjustable weight.
Turning this dial changes every prediction together. Compare three settings on inputs $x=1,2,3$:
Worked example: three settings of the dial
| dial $w$ | guess at $x=1$ | guess at $x=2$ | guess at $x=3$ |
|---|---|---|---|
| $w = 1$ | 1 | 2 | 3 |
| $w = 1.5$ | 1.5 | 3 | 4.5 |
| $w = 2$ | 2 | 4 | 6 |
One model, three settings, and none of them matches the real values $1, 4, 4$.
The dial is shared across examples: we cannot choose one slope for the first point and another for the second. That constraint is what makes this a model of a relationship instead of a list of answers. Raising $w$ raises all positive-input predictions, so improving one miss may worsen another. Training must balance them.
Learning means finding the dial value whose predictions land closest overall to the real values. “Closest” needs a precise meaning before a machine can choose.
| During prediction | During training |
|---|---|
| Input changes; weight stays fixed | Compare predictions with labels, then change the shared weight |
A line through the origin always passes through $(0,0)$. That is a deliberate simplification: fixing its height leaves one dial and hand-checkable arithmetic. We add the intercept later, so the model can also move up and down.
Wrongness you can see
Each data point sits above or below the line. Its residual is that miss. We write it as prediction minus truth so the sign tells us which way the miss went; the red segments in the picture show the distances.
Overshoots and undershoots would cancel if we added signed residuals. Squaring prevents cancellation and penalizes large misses more: a miss of 2 costs 4, while a miss of 1 costs 1. Averaging gives the mean squared error, our loss $L$.
- Subtract: prediction minus truth gives a signed miss.
- Square: prevent cancellation and emphasize large misses.
- Average: divide the sum by the number of examples.
The tall symbol $\sum$ is capital Greek sigma: “add these up.” In $\sum_{i=1}^{N}$, let $i$ count through examples 1 to $N$, compute what follows once per example, and add. It is a for-loop in different notation. Subscripts do the same counting job: $x_i$ and $y_i$ are example $i$’s input and true value.
| Notation | Here |
|---|---|
| $N$ | 3 examples |
| $i$ | Example number: 1, 2, then 3 |
| $x_i,y_i$ | Input and truth for that example |
| $\hat{y}_i$ | The model’s prediction for that example |
For our $N=3$ examples, average the squared misses:
$$L(w) = \frac{1}{N} \sum_{i=1}^{N} \left( \hat{y}_i - y_i \right)^2 = \frac{1}{N} \sum_{i=1}^{N} \left( w \cdot x_i - y_i \right)^2$$In words: for each point, subtract truth from the prediction, square the miss, and average. A different $w$ changes the predictions and therefore changes $L(w)$.
Worked example: the loss at $w = 1.5$
At $w=1.5$, predictions $1.5,3,4.5$ face true values $1,4,4$. Prediction minus truth gives $+0.5,-1,+0.5$; their squares are $0.25,1,0.25$:
$$L(1.5) = \frac{(0.5)^2 + (-1)^2 + (0.5)^2}{3} = \frac{0.25 + 1 + 0.25}{3} = \frac{1.5}{3} = 0.5$$In words: square each of the three misses, add them, and divide by three. The dial at $w = 1.5$ scores $0.5$, well under the $1.67$ that $w = 1$ and $w = 2$ each score.
| Setting | Squared misses | Loss |
|---|---|---|
| $w=1$ | $0,4,1$ | $5/3\approx1.67$ |
| $w=1.5$ | $0.25,1,0.25$ | $0.5$ |
| $w=2$ | $1,0,4$ | $5/3\approx1.67$ |
The settings $w=1$ and $w=2$ are equally wrong, and $w=1.5$ between them is better. That dip hints at the shape we will walk downhill.
| Loss $0.5$ means | It does not mean |
|---|---|
| Average squared miss of $0.5$ | Half the predictions are wrong |
The error rule matters. Squaring makes the middle miss of magnitude 1 contribute as much as four misses of magnitude 0.5. That is useful when large errors deserve extra attention, but it also makes unusual measurements influential. “Best line” always means best under the chosen loss and within the allowed model; it does not mean every point is correct.
The loss is a bowl
Compute the loss for many settings of the dial and plot $L$ against $w$. The points fall on a smooth U-shaped curve, a parabola, the same shape as the toolkit's derivative section, with a single lowest point.
Do not confuse the two horizontal axes. In the first picture we move along inputs and read a prediction from a line. Here we move among candidate weights and read the total error of each candidate. One point on the bowl represents an entire line evaluated against all three examples; the machine walks among models, not among data points.
| Dial | Loss | Meaning |
|---|---|---|
| $0$ | $11$ | Flat line, large misses |
| $1$ | $1.67$ | Too shallow |
| $1.5$ | $0.5$ | Best slope |
| $2$ | $1.67$ | Too steep |
| $3$ | $11$ | Large misses again |
The floor is $0.5$, not zero: no line through the origin hits every point. Minimizing loss means finding the best available model, which can still make errors.
This curve is a loss landscape. Right now it is a one-dimensional bowl because we have one dial. Add more dials and it gains dimensions, but the goal stays the same: find the lowest point. That search is what training does.
| Data plot | Loss plot |
|---|---|
| Input $x$ versus output $y$ | Weight $w$ versus loss $L$ |
| A dot is one measurement | A dot scores a whole candidate line |
At the bottom, the ground is flat. The toolkit’s derivative measures that slope locally; we can find the bottom by repeated small steps.
Rolling downhill to the answer
To descend, measure the loss slope $\frac{dL}{dw}$ and step against it. Let $\eta$, eta, be the learning rate:
$$w_{\text{new}} = w - \eta \, \frac{dL}{dw}$$In words: subtract the current slope times the learning rate; the minus sign sends a sufficiently small step downhill.
- Measure: get the current loss slope.
- Move: subtract $\eta$ times that slope.
- Repeat: recompute at the new weight.
A negative derivative means loss decreases when the weight increases slightly; subtracting that negative number moves right. A positive derivative sends the next step left. Near the bottom the derivative becomes small, so the same learning rate produces smaller changes. The learning rate controls the multiplier, not a fixed distance traveled on every update.
Compute the slope from the signed residuals:
$$\frac{dL}{dw} = \frac{2}{N} \sum_{i=1}^{N} x_i \left( w \cdot x_i - y_i \right)$$In words: multiply each input by its prediction-minus-truth error, sum, then scale by $2/N$; large inputs magnify their errors’ influence.
| Signed slope | Small downhill move |
|---|---|
| Negative | Increase weight |
| Positive | Decrease weight |
The factor 2 comes from the square’s derivative. The input $x_i$ appears because a nudge to $w$ moves the prediction at $x=3$ three times as far as at $x=1$.
Worked example: check the bottom
Check the bottom: at $w=1.5$, signed errors are $+0.5,-1,+0.5$, giving $1(0.5)+2(-1)+3(0.5)=0.5-2+1.5=0$. The slope is zero.
| Weighted signed errors | Sum |
|---|---|
| $0.5-2+1.5$ | $0$: flat at $w=1.5$ |
Worked example: the first downhill step, from $w = 0$
Start at $w=0$: predictions are $0$, errors $-1,-4,-4$. Weight each by its input:
$$\sum_i x_i(w x_i - y_i) = 1(-1) + 2(-4) + 3(-4) = -1 - 8 - 12 = -21$$In words: input times signed error, summed, gives $-21$.
| Weighted total | Scale | Slope |
|---|---|---|
| $-21$ | $2/3$ | $-14$ |
Multiply by $2/3$, twice the average:
$$\frac{dL}{dw} = \frac{2}{3}(-21) = -14 \quad \text{at } w = 0$$The negative slope points downhill to the right. Choose $\eta=0.1$: slope magnitude $14$ then gives a visible step of $1.4$ without overshooting the bottom at $1.5$.
$$w_{\text{new}} = 0 - 0.1 \times (-14) = +1.4$$In words: subtracting one tenth of $-14$ moves the dial from $0$ to $1.4$.
Repeat with the same learning rate; displayed values below are rounded, but each update uses the unrounded previous result:
Worked example: four gradient steps
- Step 1: $w=0$, slope $-14$ → $1.4$.
- Step 2: $w=1.4$, slope $-0.9333$ → $1.4933$ (about $1.49$).
- Step 3: $w=1.4933$, slope $-0.0622$ (about $-0.06$) → $1.4996$ (about $1.50$).
- Step 4: $w=1.4996$, slope $-0.0041$ → $1.5000$.
The steps shrink as the slope flattens. Loss falls from $11$ toward $0.5$, and nobody ever told the machine that the answer was $w=1.5$. A model, a loss, and gradient descent are all it took.
From one dial to two, and to a trillion
Real lines have a height as well as a slope. Add the intercept $b$:
$$\hat{y} = w \cdot x + b$$In words: multiply input by slope $w$, then add height $b$; the line can tilt and slide.
| Dials | What the bowl becomes | What changes |
|---|---|---|
| One weight | Curve | Read one slope |
| Weight and intercept | Surface | Read two partial slopes |
| Millions to a trillion | High-dimensional landscape | Compute a gradient with one entry per dial |
The gradient now has two entries: how loss changes with $w$ while $b$ is fixed, and with $b$ while $w$ is fixed. Update both dials together, using slopes measured at the same starting settings.
For our points, the best intercept is $b=0$. Their centroid pairs the mean input $2$ with mean output $3$. The line $y=1.5x$ passes through this average point $(2,3)$; its residuals also sum to zero. Together with the zero slope derivative already checked, that confirms a stationary point for both dials on this convex bowl. Other datasets usually need a nonzero intercept.
| Our data’s averages | Candidate line |
|---|---|
| Mean input $2$; mean output $3$ | $1.5\times2+0=3$ |
The same update scales to networks with millions or billions of parameters. What changes is the model and landscape: a neural network’s loss need not have one bowl-shaped minimum.
| Calibration | Our fit |
|---|---|
| Known references | Training examples |
| Slope and height | Learned dials |
Two dials do not mean fitting two separate lines. They jointly describe one prediction rule, and each change affects the same residuals. Holding the other dial fixed when measuring a slope separates those influences; after measuring both, we make the update together.
See it move
The video connects residuals, line, and loss bowl: $w$ moves from $0$ toward $1.5$ as loss falls $11\to0.5$.
Drag the line over the three points and watch the loss respond, then press “Let it learn.” Notice which way the line tilts when the bowl’s slope is negative and when it is positive.
Keep the three input points unchanged when comparing settings, so the changes come from the dials.
Watch the loss trace as well as the final line. Two runs can finish near the same answer by very different routes: one settles smoothly, another crosses the valley again and again. Reset before comparing learning rates so the starting weight is controlled. With the intercept enabled, the path also depends on how the two dials interact.
Where you'll meet this
| Where | What it looks like | Chapter link |
|---|---|---|
| Classification | A linear score becomes a probability and a boundary | Drawing boundaries |
| Neurons | A weighted sum followed by an activation | The neuron |
| Deep learning | Many dials, a loss, and gradients computed through layers | Teaching the network |
What you now know
- Linear regression adjusts a line’s slope and, when allowed, its intercept.
- Mean squared error averages squared misses; it is not a percentage of wrong answers.
- Our best slope is $1.5$, with loss $0.5$.
- Gradient descent follows local slopes, using a learning rate to control the step.
- More dials enlarge the gradient and can increase the risk of fitting noise.
Where we’re headed. That answers “how much?” The next question is “which one?” Drawing boundaries turns a linear score into a probability and uses it to separate categories.