The unit every network is built from
Every deep network is built from one small arithmetic unit repeated many times: the neuron. The name borrows from biology, but the unit itself is plain arithmetic.
An artificial neuron takes a weighted sum, adds a bias, then squashes the result. We already own the pieces: the dot product from the math toolkit and the sigmoid from drawing boundaries. One neuron draws one straight boundary across a two-dimensional input, and a stack combines those cuts into complicated regions. We calculate one neuron, meet a pattern it cannot separate, then build a network that can.
- Weight: multiply inputs by learned weights and sum.
- Shift: add the learned bias.
- Squash: turn the score into an activation.
The anatomy of a neuron
The diagram labels the whole unit. Inputs form vector $\mathbf{x}$, “the vector x”; learned weights form $\mathbf{w}$. Each weight says how much its input matters; a negative weight argues against firing. The bias $b$ shifts the decision.
When we present a new example the inputs change, while the weights and bias stay put. Learning changes those internal dials over many examples; evaluating a trained neuron uses their current settings. That distinction is why the same unit can give different answers without learning anything new between the two predictions.
Call the pre-squash total $z$, the raw score:
$$z = \mathbf{w} \cdot \mathbf{x} + b = w_1 x_1 + w_2 x_2 + b$$In words: multiply each input by its weight, add the products, then add the bias to get the score $z$.
An activation function turns that score into output $a$, the activation. We use the sigmoid, $\sigma$ or sigma, from drawing boundaries:
$$a = \sigma(z) = \frac{1}{1 + e^{-z}}$$In words: squash the score into a firing strength between 0 and 1; near 0 is quiet, near 1 strong.
Worked example: one neuron, two verdicts
- Set the dials: keep weights and bias fixed for both inputs.
- Evaluate: compute each raw score, then its sigmoid.
Take weights $\mathbf{w}=[1,1]$ and bias $b=-1$: equal positive weights count both inputs equally, with the bias setting a threshold at a sum of 1. Feed in $\mathbf{x}=[2,1]$:
$$z = (1)(2) + (1)(1) + (-1) = 2 \qquad a = \sigma(2) \approx 0.88$$The activation is $0.88$, or 88% of the output scale; interpreting it as confidence requires a model trained for that purpose. Recall the midpoint $\sigma(0)=0.5$. Now feed the same neuron $\mathbf{x}=[-1,0]$:
$$z = (1)(-1) + (1)(0) + (-1) = -2 \qquad a = \sigma(-2) \approx 0.12$$In words: unchanged weights give the second point a score of $-2$ instead of $+2$, so its activation is $0.12$ instead of $0.88$; the inputs determine which side of the threshold it reaches.
| Input | Score | Activation |
|---|---|---|
| $[2,1]$ | $2$ | $0.88$ |
| $[-1,0]$ | $-2$ | $0.12$ |
The model is weighted arithmetic; a biological neuron is a living cell.
A neuron is a line
The two verdicts came out opposite because the inputs lie on opposite sides of a boundary. With a sigmoid, the halfway activation sits at score $z=0$. In the picture, that boundary is a straight line.
A sigmoid plots activation against one score; our boundary map plots the two inputs against each other. Its boundary asks which input pairs produce the midpoint activation. A curved score-to-output graph therefore does not imply a curved boundary in the input plane.
$$w_1 x_1 + w_2 x_2 + b = 0 \;\Rightarrow\; x_1 + x_2 = 1$$In words: the decision boundary contains all points whose score is zero; with our weights and bias, their two coordinates add up to 1.
Our point $[2,1]$ sums to 3, on the high side, and fires at $0.88$; $[-1,0]$ sums to $-1$, on the low side, and fires at $0.12$. That is logistic regression under a different name: one side scores above $0.5$, the other below.
| What we change | What to look for |
|---|---|
| Bias alone | The cut slides without changing its orientation. |
| Relative weights | The cut rotates because inputs count differently. |
| Positive scale on all weights and bias | The midpoint stays put while the switch becomes sharper. |
The bias slides the line; changing relative weights tilts it. Scaling all weights and the bias together sharpens the sigmoid across the same boundary. None of these adjustments curves the boundary.
| Detector | Region accepted |
|---|---|
| One threshold | Everything above a cutoff |
| Two combined thresholds | Only the band between cutoffs |
A receiver’s squelch opens the speaker above a signal threshold. Like a neuron’s decision, it asks whether one weighted quantity crossed a cutoff. But an alert that should fire only in a middle band needs two tests: above a lower cutoff and below an upper cutoff. Combining thresholds is the escape from a single straight cut.
The problem one line can't solve
XOR, “exclusive or,” is true when exactly one of two inputs is on. Its four cases expose a limit of single-layer networks:
Worked example: the XOR pattern
| $x_1$ | $x_2$ | XOR | class |
|---|---|---|---|
| $0$ | $0$ | $0$ | OFF |
| $0$ | $1$ | $1$ | ON |
| $1$ | $0$ | $1$ | ON |
| $1$ | $1$ | $0$ | OFF |
Plot those four points and the trouble shows itself: the two ON points, $(0,1)$ and $(1,0)$, sit at opposite corners, and the two OFF points, $(0,0)$ and $(1,1)$, sit at the other opposite corners. The classes are diagonal to each other, interleaved.
The training labels are consistent: there is no noise or ambiguity to blame. The limitation belongs to the model’s possible shapes. This is a useful diagnostic whenever training stalls: separate “we have not found good weights” from “no allowed weights can express the required answer.” XOR gives us a case where we can prove the second.
Try drawing a line that puts both ON corners on one side and both OFF corners on the other. Any attempt strands an ON with an OFF. More training cannot repair a model that cannot represent the answer.
- Check labels: exactly one input on means ON.
- Try a line: the diagonal classes cannot be separated.
- Change the model: give an output neuron two hidden verdicts to combine.
The 1969 critique of single-layer networks made this limitation famous. Keep the conclusion precise: it limits a single layer, not every possible network.
We need two ordinary straight-line neurons looking at the input and a third combining their verdicts. Two cuts enclose a diagonal region, the same way the receiver’s middle-band detector accepts what lies between two thresholds.
Stack them and the boundary bends
Build two inputs, a hidden layer of two neurons, and one output neuron. The output reads hidden activations instead of raw inputs.
For hand calculation, idealize every squash as a hard threshold: positive scores become 1, negative scores 0. The weights below specify threshold locations; to approximate these switches with sigmoids, multiply each unit’s weights and bias by a large positive factor. Without that scaling, $\sigma(-0.5)\approx0.38$, not 0.
Hidden neuron $h_1$ detects OR, at least one input on; $h_2$ detects AND, both on. Their thresholds sit between the possible input sums 0, 1, and 2:
| Unit | Weights; bias | Boundary |
|---|---|---|
| $h_1$: OR | $[1,1]$; $-0.5$ | $x_1+x_2=0.5$ |
| $h_2$: AND | $[1,1]$; $-1.5$ | $x_1+x_2=1.5$ |
The output fires for “at least one, but not both”: weights $[+1,-1]$ on $[h_1,h_2]$, bias $-0.5$. Its threshold again lies halfway between possible integer inputs. Walk all four cases through:
Worked example: the hidden layer solves XOR
| input | $h_1$ (at least one) | $h_2$ (both) | score $+h_1 - h_2 - 0.5$ | squashed | |
|---|---|---|---|---|---|
| $(0,0)$ | $0$ | $0$ | $0 - 0 - 0.5 = -0.5$ | $0$ → OFF | ✓ |
| $(0,1)$ | $1$ | $0$ | $1 - 0 - 0.5 = +0.5$ | $1$ → ON | ✓ |
| $(1,0)$ | $1$ | $0$ | $1 - 0 - 0.5 = +0.5$ | $1$ → ON | ✓ |
| $(1,1)$ | $1$ | $1$ | $1 - 1 - 0.5 = -0.5$ | $0$ → OFF | ✓ |
All four come out correct: a negative score squashes toward 0 and reads OFF, a positive one squashes toward 1 and reads ON.
Changing what the output sees is what fixes the problem. The raw ON inputs occupy opposite corners, but both become the same hidden pair: $[h_1,h_2]=[1,0]$. The OFF inputs become $[0,0]$ and $[1,1]$. The hidden layer has turned a troublesome arrangement into a representation where one output threshold works. “Hidden” means intermediate, not mysterious; its values are ordinary numbers we just calculated.
| Raw inputs | Hidden pair |
|---|---|
| ON: $(0,1)$ or $(1,0)$ | $[1,0]$ |
| OFF: $(0,0)$ | $[0,0]$ |
| OFF: $(1,1)$ | $[1,1]$ |
Each hidden neuron draws a straight line; the output keeps the corridor between them. More hidden neurons supply more cuts to combine into more complicated regions.
| Layer object | Contains |
|---|---|
| $\mathbf{W}$ | One weight row per neuron |
| $\mathbf{b}$ | One bias per neuron |
A whole layer uses the matrix multiplication from the math toolkit. Let $\mathbf{a}$ collect activations and $\mathbf{b}$ biases:
$$\mathbf{a} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b})$$In words: stack each neuron’s weights into a row of $\mathbf{W}$; multiplying by input vector $\mathbf{x}$ computes all weighted sums, adding $\mathbf{b}$ shifts them, and applying the sigmoid to each entry produces $\mathbf{a}$.
| One neuron | Combined network |
|---|---|
| One straight cut | A region defined by multiple cuts |
Any shape at all
A pair of opposing sigmoid neurons builds a bump: subtract a rising sigmoid shifted right from an earlier rising sigmoid. Between their rises, the first is high while the second is low; elsewhere their outputs nearly cancel. The picture shows how that difference confines a response to a strip of input.
Two controls decide what a bump is good for: the threshold positions choose where it sits and how wide it is, and the output weight sets how much it contributes to the sum. Subtracting the second sigmoid is itself a weighted sum with one negative weight. Nothing new has been added to the neuron; the same pieces are arranged to give a localized response.
Line up bump-detectors over different strips and scale each with an output weight. Their sum traces a curve as a staircase of bumps; more, narrower bumps improve the fit. This is universal approximation: one hidden layer, wide enough, can approximate a continuous input-to-output relationship over a bounded region.
| The theorem gives | It does not give |
|---|---|
| Existence of approximating weights | A method for finding them |
| Close fit with enough capacity | A small network or success on unseen data |
Read the theorem narrowly. “Can approximate” says suitable weights exist; it does not tell us how to find them, how many examples we need, or how many neurons a shape requires. A single wide layer can be enormous. Depth can represent useful relationships more economically by composing features across layers.
| Arrangement | How it builds a shape |
|---|---|
| Wide layer | Many separate bumps cover different input regions. |
| Deep stack | Later units reuse combinations that earlier layers already computed. |
All of that shows what chosen weights can represent. We hand-set the XOR weights because we knew the answer, and learning has to find useful weights from examples instead. Networks usually start randomly and improve through gradient descent. The next experiment shows that difference.
See it learn
The video follows the same argument: one line fails on XOR; two hidden neurons with weights $[1,1]$ and biases $-0.5$ and $-1.5$ supply cuts that the output combines.
Try a network learning from scratch. Choose a dataset and hidden-layer size, then press Train. Watch both its boundary and its loss: one shows the learned shape, the other scores the errors. The widget shows gradient descent working. How the error reaches buried weights is the next chapter’s question.
Keep the dataset fixed when changing the hidden-layer size, so you can attribute the difference to capacity. Then keep capacity fixed when resetting, so you can see the effect of the starting weights. A failed run alone does not prove a pattern is impossible: the one-neuron XOR case is special because we already have a geometric proof.
Where you'll meet this
| Where | What it looks like | Chapter link |
|---|---|---|
| Images | Weighted sums and activations wired to nearby pixels | Seeing machines |
| Sequences | Neural layers connected through time or attention | Remembering machines |
| Graphs | The same arithmetic combining connected neighbors | Networks of relationships |
| Language models | Neural layers stacked with attention to produce text | Inside the transformer |
What you now know
- A neuron weights its inputs, adds a bias, and applies an activation function.
- A single sigmoid neuron has a straight decision boundary and cannot separate XOR.
- Two hidden thresholds plus an output neuron separate all four XOR cases.
- Weighted sums of bumps explain universal approximation, an existence result without a training guarantee.
- Depth can reuse features to represent complex shapes efficiently.
- Learning finds weights from examples; our hand-set XOR network only demonstrated representation.
Where we’re headed. How does a wrong answer tell a buried weight which way to move? Teaching the network follows that error backward, then asks how to learn the real pattern instead of memorizing training points.