Page 4 of 9
The bend: one fold changes what is possible
After each slot adds up its inputs, replace every negative total with zero. That one rule folds space, and folding is the thing no table can do.
The rule
level = total if the total is positive level = 0 if the total is negative
a slot reports how far past zero its total is, and reports nothing otherwise.
These pages call this a gate, or the bend. Textbooks call it ReLU, short for rectified linear unit, and they call the general idea an activation function or a nonlinearity. The word rectified is borrowed from electronics, where a rectifier does the same thing to a signal: it passes one direction and blocks the other.
A slot with a gate answers a question about the input. Its weights pick a direction. Its offset picks a starting line. Its level says how far past that line the input is, and on the near side of the line it says zero.
What happened in the picture
Stage 1 is a table, the kind of move from the last page. It was chosen so that the two orange clusters, far apart on the page, both come out with small or negative totals.
Stage 2 is the gate. Everything that went negative is pressed flat onto a wall. The two orange clusters end up together near the corner. The two cyan clusters stay out on the walls, each a long way from the corner. Now one straight cut is enough.
The division of labor is the same in a network of any size. Tables turn and stretch, so that the right things land on the right sides of the gates. The gates do the folding, and a final straight cut reads off the answer.
The bend was there from the start
It is easy to imagine that someone stacked tables, saw them collapse into one, and added a bend to rescue them. The history ran the other way. The first design already had the bend.
So the field began with units that add and then bend, and it never had a stack of plain tables to rescue. The argument that stacked tables collapse came later, as the reason the bend cannot be taken out. What changed over the next eighty years was the shape of the bend, and page 7 follows that.
One input: hinges
With a single input the picture is simpler. A gated slot is flat up to a corner and then a straight ramp. That shape is a hinge. Add several hinges and you get a connect-the-dots curve, because each hinge changes the slope of the total at its own corner.
Eight hinges follow one cycle of a sine wave to within 0.07. Thirty-two get within 0.005, and sixty-four within 0.0012. With enough hinges the sum follows any curve as closely as you like. When people say a network can represent anything, this is the content of the claim.
Almost any bend gives full range, then. Range does not single one out. What does is the subject of the next page: whether a correction can pass back through it.
Rows of folds
One row of two gated slots can fold a strip in half. Feed the result to another such row and it folds again.
Eight rows of two make 256 straight pieces from 16 slots. A single row would need 255 slots to draw the same shape. Each row multiplies what the rows before it built, and that is the case for stacking rows.
Montúfar and others, On the number of linear regions of deep neural networks ↗ · Telgarsky, Benefits of depth in neural networks ↗
CARRY THIS FORWARD
Table, then gate, repeated, can separate anything. The open question is how a machine finds the tables that do it.