The Neuron

One cell of a brain that isn't

The whole towering edifice of deep learning — every image generator, every chatbot, every protein-folder — is built from one absurdly small part, repeated billions of times. That part has a name borrowed from biology: the neuron. And that borrowing causes more confusion than any other word in the field. So we are going to build the real thing first, out of arithmetic you already own, and let the biology stay a loose metaphor — never a claim.

An artificial neuron does exactly three things, in order: it takes a weighted sum of its inputs, it adds one extra number called the bias, and it passes the result through a squashing curve. We have already met every one of those pieces — the dot product back in the math toolkit, the sigmoid in drawing boundaries. This chapter just wires them into a single unit, and then discovers what happens when you wire thousands of those units together.

Here is the target, stated up front so you know where we are headed. One neuron can only ever draw a single straight line across its world. That sounds like a crippling limit — and it is, for one neuron. The astonishing part, the part that makes deep learning work at all, is that a modest stack of these straight-line units can carve out any shape at all. By the end of this chapter you will have watched a network do exactly that, live, in your browser.

The anatomy of a neuron

Let us build the neuron one wire at a time. It receives a handful of input numbers; collect them into an input vector $\mathbf{x}$ — "the vector x", the same kind of object from the math toolkit. It owns one weight per input; collect those into a weight vector $\mathbf{w}$, "the weights". And it owns one lone number $b$, the bias. What are the weights, plainly? They are dials the neuron will learn (the very thing fitting a line taught us to do), one per input, each saying how much that input matters and in which direction — a negative weight means "this input argues against firing".

The weighted sum is a dot product — the exact similarity score from the math toolkit, now doing a real job. Call the pre-squash total $z$, the raw score (the same letter drawing boundaries fed to the sigmoid):

In words: multiply each input by its weight, add the results together, then add the bias. That single number $z$ is the neuron's opinion before it decides how loudly to voice it.

Now the squash. The score $z$ can be any number on the whole line — wildly positive, wildly negative. The neuron passes it through an activation function, and for now we use the sigmoid from drawing boundaries. Call the output $a$, the neuron's activation:

In words: squash the score into a firing strength between 0 and 1 — 0 means silent, 1 means shouting. Nothing the neuron ever outputs escapes that strip.

A neuron is a line

Why did the same neuron give opposite verdicts? Because of a hidden geometry — and it is the bridge straight back to drawing boundaries. The neuron fires hardest where $z$ is large and stays silent where $z$ is very negative; the fence between "firing" and "silent" sits exactly where $z = 0$. Set our worked neuron's score to zero and see what that fence is:

In words: the neuron's decision boundary is the set of points where its score is exactly zero — here, all the points whose two coordinates add up to 1. That is the equation of a straight line across the plane.

Now walk the two worked points onto that map. The point $[2, 1]$ has coordinates summing to 3 — well on the high side of the line — so the neuron shouts $0.88$. The point $[-1, 0]$ sums to $-1$ — on the low side — so it whispers $0.12$. Everything on one side of the line fires above $0.5$; everything on the other side fires below. A single neuron cleaves its whole world into two half-planes with one straight cut. This is precisely the logistic regression from drawing boundaries, now wearing the word "neuron".

And now name the limit honestly, because the rest of the chapter is about escaping it. The bias slides the line; the weights tilt it and steepen it. But no setting of two weights and a bias can make that boundary anything other than straight. A neuron is a ruler, never a French curve. Any pattern that a single straight line cannot separate is, for one neuron, unlearnable — full stop.

The problem one line can't solve

Meet the pattern that broke the first neural networks and nearly killed the field: XOR — "exclusive or", true when exactly one of two inputs is on. Laid out as a small truth table, it looks harmless:

Try it in your head, or on the figure. Draw one straight line that puts both ON corners on one side and both OFF corners on the other. It cannot be done — because the classes are diagonal to each other, any straight cut always leaves one ON and one OFF stranded together on the same side. The red line is one failed attempt; every other angle fails the same way. A single neuron, being one straight line, is therefore helpless against XOR.

The historical stakes, in two sentences: in 1969 this exact observation — that a single-layer neuron cannot do XOR — was published, funding evaporated, and neural-network research went cold for over a decade. The escape was known in principle but not yet practical, and it is almost embarrassingly simple to state: add a layer.

Here is the mechanism the next section pays off. We do not need a curvier single neuron. We need two ordinary straight-line neurons looking at the input, each drawing its own line, and a third neuron that combines their two verdicts. Two straight cuts, combined, can fence off a diagonal region. Straightness stacked becomes bentness.

Stack them, and space bends

Let us build the smallest network that beats XOR: two inputs, a hidden layer of two neurons, and one output neuron. Each hidden neuron is an ordinary weighted-sum-and-squash from two sections ago. The twist is what the output neuron reads: not the raw data, but the two hidden neurons' activations.

To read the network by hand, we will idealize the squash as a near-hard switch — and it is worth being explicit about the cheat, because it matters. Imagine the hidden neurons have large weights; then the sigmoid snaps close to 0 or 1 with almost nothing in between, so we can read each activation as a clean YES or NO. (With real, moderate weights $\sigma$ would return soft values like $\sigma(-0.5) \approx 0.38$; the idealization is a reading aid, not a claim that the sigmoid outputs exact 0s and 1s.)

Give the two hidden neurons concrete, hand-checkable jobs. Hidden neuron $h_1$ fires when at least one input is on — an OR detector with weights $[1, 1]$ and bias $-0.5$, so its boundary is the line $x_1 + x_2 = 0.5$. Hidden neuron $h_2$ fires only when both inputs are on — an AND detector with weights $[1, 1]$ and bias $-1.5$, so its line is $x_1 + x_2 = 1.5$. Two straight lines, drawn by two straight-line neurons.

Now combine them in the output neuron: fire when $h_1$ says "at least one" and $h_2$ says "not both". That is exactly XOR. Give the output neuron weights $[+1$ on $h_1$, $-1$ on $h_2]$ and bias $-0.5$, and walk the four inputs through. Each row shows the raw score the output neuron computes, then what the squash makes of it (reading the idealized 0/1 activations throughout):

Read the geometry of what just happened, slowly, because it is the whole chapter in one idea. Each hidden neuron carved the plane with one straight line. The output neuron kept only the sliver between the two lines. The final decision boundary is not straight — it is a bent corridor — even though every single neuron in the network is a perfectly straight ruler. Bending emerged from combining straight cuts. Add more hidden neurons and you get more cuts; more cuts fence off more intricate regions.

One last tidy-up, and it is a notation handshake back to the math toolkit. A whole layer of neurons is one matrix multiply plus a squash:

In words: a layer stacks each neuron's weights into the rows of a matrix $\mathbf{W}$, so one matrix multiply computes every neuron's score at once; add the bias vector, squash each entry on its own, and out comes the layer's activations. The matrix $\mathbf{W}$ you multiplied by hand back in the math toolkit was, all along, a full layer of neurons.

Any shape at all

Now generalize the XOR trick into the result that makes deep learning plausible. If two straight cuts can fence a corridor, then a pair of opposing sigmoid neurons can build a single bump — a hump that is high over a chosen strip of the input and low everywhere else. Picture one rising sigmoid minus a second rising sigmoid shifted a little to the right: where the first has already climbed but the second has not yet caught up, the difference stands tall; everywhere else it cancels to nearly zero. A bump, built from two straight-line neurons.

Now add bumps. Line up many such bump-detectors, each covering a different strip, each scaled up or down by the output neuron's weights, and their sum can trace the outline of any curve you like — a staircase of bumps approximating a smooth shape as closely as you please, just by using more, narrower bumps. This is the universal approximation property: one hidden layer, wide enough, can approximate essentially any input-to-output relationship.

But immediately fence the theorem, because it is the most over-sold sentence in the field. "Can approximate" is an existence promise, not a recipe — it tells you a good set of weights exists, not how to find it, and "enough neurons" can mean an absurd number for a single wide layer. The practical reason real networks are deep rather than merely wide is that stacking layers builds complex shapes far more cheaply: each layer composes the previous layer's features into richer ones, so ten modest layers routinely outperform one monstrous one. Depth is efficiency; the rest of Part III is about matching that depth to the shape of the data.

One honest caveat before you go watch it work. We have shown a network can represent XOR and curves — but we set those weights by hand, knowing the answer. Real networks are never hand-set. They start with random weights (a scribble of a boundary) and grind toward a good one by gradient descent from fitting a line, one nudge at a time. The next thing you should do is watch that happen.

See it learn

The video runs the whole argument in one motion: watch one neuron draw its single straight line, watch that line fail on XOR at every angle, then watch two hidden neurons' lines combine into a bent boundary that finally separates it — the same $[1,1]/{-0.5}$ and $[1,1]/{-1.5}$ neurons as the prose, animated.

Now you drive it. The trainer below drops you in front of a live network learning a 2D pattern from scratch. Pick a dataset (start with XOR — the four corners), choose how many hidden neurons the network gets, and press Train. A random-scribble boundary bends itself into one that fits, with the loss falling as it goes — gradient descent from fitting a line, running on a real network in front of you. One honest note: you are watching gradient descent work, not learning how the blame reaches each buried weight — that mechanism, backpropagation, is the next chapter's job.

Where you'll meet this

Zoom out to the whole field. Every deep model the rest of this site covers — the convolutional networks that see (seeing machines), the recurrent and attention models that read (remembering machines), the graph networks that reason over networks of relationships (networks of relationships) — is this same atom, the weighted-sum-and-squash neuron, wired into a different shape to suit a different kind of data. Learn the neuron and you have learned the common ancestor of all of them.

The through-line reaches the sister site too: the transformer inside every large language model is a colossal stack of exactly these layers, $\mathbf{a} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b})$ repeated with attention mixed between the stacks — the reader who understands XOR here understands the primitive that, multiplied a few billion times, writes essays over there.

Name the one thing still missing, honestly, because it sets up the next chapter. We can now build a network that can fit a pattern, and we have watched gradient descent nudge it toward one — but we waved our hands at how the blame for a wrong answer travels back through a hidden layer to tell each buried weight which way to move. That mechanism is backpropagation, and it — plus the craft of getting a network to learn the right pattern instead of just memorizing the training points — is the whole of the next chapter.

Close warm. You have personally computed a neuron ($0.88$ and $0.12$), proven with a truth table why one line can't do XOR, and hand-wired two lines into a bend that can. That bend — straightness stacked into any shape — is the entire reason a pile of arithmetic can recognize a face, fold a protein, or hold a conversation.