Page 3 of 9
A table of weights moves the whole space
Put several weighted sums side by side and every point is carried to a new place. Turning, stretching and shearing are what that looks like when you draw it.
A row of slots
A slot is one place that holds a number. Textbooks call it a neuron or a unit. The machine on the last page had one slot, and the number it held was the score.
Now put two slots side by side. Both look at the same two inputs, and each has its own pair of weights. A set of slots working side by side like this is a row, which textbooks call a layer.
new level 1 = a × old level 1 + b × old level 2 new level 2 = c × old level 1 + d × old level 2
each new level is its own weighted sum of the old levels.
That is four weights, written as two rows of two. The block of numbers is a table of weights, which textbooks call a matrix. Two numbers go in and two numbers come out. A point on the page is two numbers, so a point goes in and a point comes out. The table sends every point of the page to a new place.
Turning and stretching were not chosen
People describe networks as stretching, rotating and squashing space, and it can sound as though someone picked those operations for a purpose. Nobody did, because they are what a table of weighted sums does to a picture. Try the presets above. Every table you can make is some mix of a turn, a stretch, a shear and a flip.
Every table keeps straight lines straight, keeps evenly spaced lines evenly spaced, and leaves the center where it is. A table can move space around. It cannot tear it and it cannot fold it.
Why a table
Nothing told anyone to use a matrix either. One slot's weights are a list. A row of slots looking at the same inputs is a stack of lists, and a stack of lists is a table. Position 2 in row 1 of the table is the weight on the wire from input 2 to slot 1.
On Rosenblatt's machine the table was a physical object. The Smithsonian's record of the Mark I Perceptron describes “the array of potentiometers that comprises the A unit.” A matrix is that array written down. When a program multiplies by a matrix, it is computing every slot's weighted sum at once.
If lists and tables are as far as you care to picture, you have everything you need. One input is a list. The weights between two rows are a table. The larger arrays in real programs are stacks of these, so that many inputs can go through the same table together.
Two tables in a row are one table
Send a point through one table and then through a second. The result is always something a single table could have done.
first table second table the two, as one table 2 0 1 1 3 1 1 1 0 1 1 1 (1, 1) → (2, 2) → (4, 2) (1, 1) → (4, 2)
in two moves or in one, the point lands in the same place.
Switch on Add a second table above and press Play both to watch it happen. This is the reason a stack of rows that only add is no better than one row. A hundred tables in a row are one table, and one table followed by a straight cut is still one straight cut.
What a table cannot do
Put the ring in a ring on the grid above and try every table you like. The center stays inside the ring. The best straight cut gets 124 of the 160 points right before the table, and no table makes that number go up.
To sort things that are tangled like this, something has to fold the page, and no table can.
Why sums at all: every smooth relationship is a table up close
There is a reason beyond history for building with weighted sums. Over a small enough range, every smooth relationship between inputs and outputs is a weighted sum plus a constant. Scientists use this whenever they linearize: small-angle approximations, Ohm's law, first-order corrections.
One weighted sum forced to hold everywhere is too rigid. The networks on the following pages use a different table in each region of space, and they learn where the borders between regions go. In a check on a small network with gates, the whole network equals one table plus offsets for any single input, to sixteen decimal places, and 200,000 random inputs used 24 different tables.
CARRY THIS FORWARD
A table of weights turns and stretches space. It never folds it.