Drawing Boundaries

From "how much" to "which one"

Last chapter our machine answered how much? — a number on a dial, an absorbance, a price, a temperature. But most decisions are not quantities at all; they are categories. Is this email spam or not? Is this tumor benign or malignant? Is this photo a cat or a dog? The output is a choice, not a measurement, and that one change reshapes the whole problem — this is the classification we first named back in Chapter 1.

Here is the plan, and the single clever move the chapter turns on. We keep last chapter's score, $\hat{y} = \mathbf{w} \cdot \mathbf{x} + b$ — a straight-line combination of the inputs — but that score can be any number at all, from minus a million to plus a million, while a decision wants a clean answer between 0 and 1: the probability of the positive class. So we need a function that squashes the whole number line down into the strip from 0 to 1. That function is the sigmoid, and getting it exactly right is most of this chapter.

And I want to promise the payoff honestly. By the end you will not just use the sigmoid — you will know why it has that precise S-shape, where it quietly sabotages training, and which of its several cousins to reach for instead. Choosing the squashing function is one of the most consequential decisions in all of deep learning, and almost nobody explains why. We will.

A score that roams too far

Start with the score from Chapter 3, now doing a new job. We compute a weighted combination of the inputs plus a bias $b$ — the same intercept dial from last chapter, the number that slides the whole line up and down; once it lives inside a scoring unit the field calls it the bias, and we will use that name from here on. Call the result the logit $z$:

In words: multiply each input by its weight, add them all up (that is the dot product from Chapter 2), then add the bias $b$. With one input, that center dot is the ordinary multiplication we used last chapter; with several, it is Chapter 2's dot product — the same recipe, one slot or many. For a single input feature this is a line; for several inputs it is a dot product plus a number. Here $\mathbf{w}$ is the vector of weights, $\mathbf{x}$ the input, and $b$ a single number.

Now name the mismatch precisely. The logit $z$ can land anywhere on the number line, but a probability must live between 0 and 1. We need a machine that takes any number and hands back a number in that strip: a very negative $z$ maps to near 0, a very positive $z$ to near 1, and $z = 0$ to the fence-sitting 0.5.

The squash is built from one new mathematical ingredient, so let us meet it in a single compact beat: the exponential, $e^z$. Here $e$ is a fixed number, about $2.718$, and the one property we need is that $e^z$ is always positive — no matter what $z$ is. Feed it a huge negative number and you get a tiny positive one; you can never make it zero and never make it negative. It grows by multiplying: each step up in $z$ multiplies the value again. A few hand values to anchor it: $e^0 = 1$, $e^1 \approx 2.72$, $e^2 \approx 7.39$, $e^{-1} \approx 0.37$, and $e^{-2} \approx 0.14$. A negative exponent just means "one divided by": $e^{-2}$ is $1/e^{2}$, one over $7.39$, which is about $0.14$ — so the last two anchors are not new facts to memorise, they are the first two turned upside down. (For the full, slow build of $e^x$ and its partner $\ln x$, the sister site's Functions and Curves has it; here we need only "always positive, grows by multiplying".)

And "always positive" is exactly the raw material for building something that can never leave the 0-to-1 strip — which is the sigmoid, next.

The sigmoid, in full

Picture first, then symbol. Look at the top panel of the figure below: an S that hugs 0 on the far left, hugs 1 on the far right, never quite touching either, and rises fastest right through the middle at $(0,\, 0.5)$. That curve is the sigmoid, written $\sigma(z)$ — read "sigma of z":

In words: take $e$ to the minus $z$, add 1, and divide 1 by the result. Because $e^{-z}$ is always positive, the bottom is always bigger than 1, so the whole fraction always lands between 0 and 1.

Let us work the three checkpoints by hand, using the $e$-values we just met — these are the anchors for the whole chapter. At the center, $\sigma(0) = 1/(1+1) = 0.5$. For a positive score,

In words: $e^{-2}$ is about $0.14$, so the bottom is about $1.14$, and 1 divided by $1.14$ is about $0.88$ — a confident vote for class 1.

The mirror case is $\sigma(-2) = 1/(1 + e^{2}) = 1/(1 + 7.39) \approx 0.12$. Notice the symmetry: $\sigma(2) + \sigma(-2) \approx 1$, and more generally $\sigma(-z) = 1 - \sigma(z)$. Flipping the score's sign flips which class the probability favors. Reading the curve as behavior: extreme scores get flattened toward certainty, and all the action — the steep, decisive part — happens near $z = 0$. That center is the decision boundary, where $\sigma = 0.5$: on one side the model votes class 1, on the other class 0.

This whole method — fitting $\mathbf{w}$ and $b$ so that $\sigma(\mathbf{w} \cdot \mathbf{x} + b)$ matches the 0/1 labels — has a name: logistic regression, regression's classifying sibling and the direct ancestor of a single neuron.

Why THIS exact shape

This is the section the whole chapter exists for, so let us not rush it. Ask the question almost no explanation asks: of all the curves that could squash the number line into 0-to-1, why this one? The honest answer is a wish list — a short set of things we need — and the sigmoid satisfies every item on it.

Requirement 1 — the range must be exactly 0 to 1, so the output can be read as a probability. The sigmoid delivers: because $e^{-z}$ is always positive, $1 + e^{-z}$ is always bigger than 1, so 1 divided by it always lands strictly between 0 and 1, approaching the ends but never reaching them. That "never reaching" matters — a probability that hit exactly 0 or 1 would be claiming impossibility or certainty, which no honest classifier should ever do from a finite score.

Requirement 2 — it must be smooth, with a slope everywhere, because training is gradient descent (Chapters 2 and 3) and gradient descent needs a slope to walk down. Picture the alternative: a hard step function that reads 0 below the boundary and jumps instantly to 1 above it. It would classify fine, but it has zero slope everywhere except one infinite spike at the jump — there is no downhill to feel, so the model could never learn by our method. The sigmoid is that step function's smooth, trainable cousin.

Requirement 3 — it should be monotonic and symmetric. Monotonic means a bigger score always gives a bigger probability, never reversing, so a higher logit always reads as "more confident it's class 1". Symmetric means it treats the two classes even-handedly: $\sigma(-z) = 1 - \sigma(z)$, the relation we already checked. The sigmoid is both, cleanly.

Now the beautiful part — and the trap. The sigmoid's own slope has a remarkably elegant form:

In words: the slope of the sigmoid at any point is its own output times one-minus-its-output. That is largest at the center — at $z = 0$, where $\sigma = 0.5$, the slope is $0.5 \times 0.5 = 0.25$ — and it shrinks toward zero at both tails, where $\sigma$ is near 0 or near 1. The bottom panel of the figure above shows this hump-shaped slope aligned right under the S-curve.

Deriving that form honestly needs calculus we are not carrying, so let us at least check it where we can, with the rise-over-run from the toolkit. Walk across the center from $z = -0.2$ to $z = +0.2$: the curve climbs from about $0.45$ to about $0.55$. That is a rise of about $0.10$ over a run of $0.4$ — a slope of about $0.25$, which is exactly the $0.5 \times 0.5$ the formula promised.

Requirement 2 is satisfied — but only barely, at the tails. And that is the trap that motivates every alternative in the rest of this chapter. Far from the center, the sigmoid is nearly flat, so its slope is nearly zero — and gradient descent multiplies by that slope. A neuron whose score is very positive or very negative gets almost no downhill signal; it learns painfully slowly, or stops. Stack many sigmoids into a deep network and those near-zero slopes multiply together into an even tinier number, so the earliest layers barely learn at all. This is the famous vanishing gradient problem, and it is exactly why deep networks moved away from the sigmoid in their hidden layers.

Drawing the boundary in two dimensions

Time to cash the chapter's title. With two input features, the score is still $z = \mathbf{w} \cdot \mathbf{x} + b$, where $\mathbf{w}$ is now a weight vector (Chapter 2's dot product doing the work). The decision boundary — where $z = 0$ and $\sigma = 0.5$ — is a straight line across the plane. On one side of that line the sigmoid reads above 0.5 (vote class 1); on the other, below (vote class 0). The classifier literally draws a line between the classes.

Here is a worked example you can check by hand. Take $\mathbf{w} = [1, 1]$ and $b = -3$, so the score and its boundary are

In words: add the two features, subtract 3, and that is the score; the boundary is the set of points where the two features add up to exactly 3.

A point at $[1, 1]$ gives $z = 1 + 1 - 3 = -1$, so its probability is $\sigma(-1) = 1/(1 + e^{1}) = 1/(1 + 2.72) \approx 0.27$ — class 0. A point at $[2, 3]$ gives $z = 2 + 3 - 3 = 2$, so its probability is $\sigma(2) \approx 0.88$ — class 1. Two points, two sides, two confident-ish votes.

One sentence of geometry ties it together: the weight vector $\mathbf{w}$ points perpendicular to the boundary, in the direction of increasing class-1 confidence. Steeper weights make the S-transition across the boundary sharper (a more decisive classifier); gentler weights make it fuzzier. Training sets $\mathbf{w}$ and $b$ to place and tilt the line.

And how does that training work? Exactly the loop from Chapter 3 — a loss and gradient descent — but with a loss suited to probabilities, called cross-entropy loss, not the squared error we used for regression. Cross-entropy heavily penalizes confident wrong answers, which is precisely what squared error fails to do: squared error barely punishes a confident mistake on a 0/1 label, so it is a poor fit here. (The sister site's Measuring Wrongness builds cross-entropy in full.) The model, loss, gradient-descent skeleton is unchanged from last chapter; only the model's squash and the loss changed.

The family of squashing functions

The vanishing-gradient trap forced the field to build a toolbox, so let us open it. The sigmoid has several relatives, each a squash chosen for a specific job. Collectively these are called activation functions, and the figure lays them out side by side so the shapes can be compared at a glance.

tanh — the zero-centered cousin. It has the same S-shape but runs from $-1$ to $+1$ instead of 0 to 1, and passes through the origin. In one line, $\tanh(z) = 2\, \sigma(2z) - 1$. Why it exists: its outputs are centered on zero, which tends to make the downhill walk behave better than the sigmoid's all-positive outputs. Where you'll meet it: historically the default for hidden layers, and still common in some recurrent networks. But it still saturates at the tails, so it does not fully escape the vanishing-gradient trap.

ReLU — the one that changed everything. The ReLU is brutally simple:

In words: pass positive scores straight through unchanged, and clamp every negative score to zero. That is the entire function.

Why it took over hidden layers: for any positive input its slope is exactly 1, so it does not saturate on that side — gradients flow through undiminished, curing the vanishing-gradient problem that hobbled deep sigmoid networks. It is also dirt cheap to compute. Its flaw, named honestly: for negative inputs the slope is 0, so a neuron stuck negative gets no gradient and can "die".

The patches — Leaky ReLU and GELU. Leaky ReLU gives negatives a small nonzero slope instead of a flat zero, so a dead neuron can revive. GELU is a smooth, slightly-curved version of ReLU — it eases through the corner at zero rather than kinking sharply — and is the default activation inside modern transformer networks, the architecture behind large language models. You do not need their formulas here, only their jobs: these are the current state of the art for hidden layers.

Now the organizing principle, so the zoo becomes a decision instead of a list. The squash for a hidden layer is chosen to keep gradients flowing — so today, ReLU or GELU. The squash for the output layer is chosen to match the question — a probability for a yes/no question, which is the sigmoid, and its many-class generalization is the next section. Right tool, right place, again.

Softmax: the many-doors sibling

So far, yes or no. But when there are more than two classes — which of ten digits, which of a thousand objects, which of fifty thousand possible next words — we need not one probability but a whole distribution over the options: bars that each sit between 0 and 1 and together sum to 1 (Chapter 2's probability tool). The function that does this is the softmax.

The recipe in words before symbols: take each option's raw score, run it through $e^x$ to make it positive (the exponential again — bigger scores become much bigger positive numbers), then divide each by the total so they all sum to 1.

In words: the probability of option $i$ is $e$-to-its-score divided by the sum of $e$-to-every-score. The bottom is the same total for all options, so it is really just "each exponentiated score as a share of the whole".

And here is the clean fact that ties the family together: the sigmoid turns out to be softmax for the two-class case. With two scores, softmax gives the first one $e^{z_1} / (e^{z_1} + e^{z_2})$; divide the top and the bottom by $e^{z_1}$ and it becomes $1 / (1 + e^{z_2 - z_1})$ — which is the sigmoid formula, fed the gap between the two scores. Squashing a single score into 0-to-1 is what softmax does when there are two options and you track just one of them. So the sigmoid and softmax are not two ideas but one, at different widths. Where a hidden layer uses ReLU to keep learning, the final layer uses softmax to state the answer as a distribution. This is the exact machinery a language model uses to turn its internal scores into "the probability of each possible next word" — the same softmax, just with tens of thousands of doors instead of three.

See it move

The video runs the whole chapter in one motion: watch a score roam the number line and get squashed into 0-to-1 by the S-curve, see the threshold at 0.5 become a boundary line between two classes, then watch the squasher morph through tanh, ReLU, and GELU, and finish with three scores becoming three probability bars — $0.665$, $0.245$, $0.090$, the same numbers as this chapter.

Now you hold the dials. Every squasher in the chapter lives in one grapher. Drag the input $z$ and watch the output; switch functions and feel each personality; turn on the slope overlay to see the vanishing gradient at the tails; then flip to softmax mode to push three scores around and watch the probability bars rebalance.

Where you'll meet this

Here is the through-line that makes the whole squashing zoo cohere: every layer of every neural network ends in one of these functions, and the choice encodes a decision — keep gradients flowing (ReLU or GELU in the hidden layers) or state an answer in the right shape (sigmoid for yes/no, softmax for which-of-many at the output). Nothing else in a network's design carries so much meaning in so small a choice.

The immediate next step is precise. An artificial neuron (Chapter 5) is literally Chapter 3's weighted sum, $z = \mathbf{w} \cdot \mathbf{x} + b$, followed by one of this chapter's squashes. That is the entire atom of deep learning: a dot product and a squash. Everything in Part II is millions of these wired together, and the squash is the small bend that lets a network do more than draw straight lines.

And the stakes are real, so value what you just learned. After "what data do I have", the choice of activation is among the most consequential design decisions in a network — the single change from sigmoid to ReLU in hidden layers is a large part of why deep learning suddenly worked in the last decade. You now understand that decision from the inside, which most people who use these tools never do.

We went from "how much" to "which one" by squashing a score into a probability, and we did not just accept the sigmoid — we interrogated it, found where it shines and where it breaks, and met the family it belongs to. That habit, asking why this piece and not another, is what separates using the tools from understanding them, and it is the habit the rest of this site is built to reward.