Page 5 of 9
How a weight deep inside gets corrected
The bottom row has a miss to work from. A weight in the middle does not. That was the unsolved problem for 25 years, and the fix was to let every slot pass an amount.
The wall
The rule from page 2 needs a miss for each slot. A slot in the bottom row has one, because the right answer is given. A slot in a middle row has none. Nobody says what its level should have been.
Every learning machine of the 1960s could adjust its last row of weights and nothing above it. In Rosenblatt's Mark I the wiring into the 512 middle slots was random and stayed fixed. Widrow's group worked for years on a machine with two rows of weights and never found a rule that trained both.
Why a ball cannot be taught
Picture a pegboard with a ball dropped in at the top. The ball bounces down one route and lands in a bin. Make one peg slightly larger. The ball still goes the same way, unless it happened to be balanced on that peg, and then the outcome flips. There is no “slightly better” to follow.
The early machines were boards of this kind, because every slot fired all or nothing. The picture at the top of this page is one of them. Drag its gold weight and watch what the error does.
On a flat shelf, rocking a weight shows nothing, and calculus has nothing to work with either. That is the wall.
The way through is to change what travels down the board. Let each slot pass an amount, and an input stops being a ball on one route. It becomes a pattern of amounts spread across each row, with some of it on every wire. Then a small change to any weight shifts every bin's score a little, and there is a direction to follow.
The fix, in its authors' words
That paragraph is the design decision. The unit has to bend, or middle rows add nothing. The bend has to have a slope, or no correction can pass through it. They chose a smooth S-shaped curve, a rounded version of the 1943 step.
Rates multiply along a chain
Take the shortest chain that has a gate in it: one input, a weight, a gate, and a second weight.
input 2 → × first weight 0.5 → gate → × second weight 0.5 → prediction 0.5 target 1 slope of the second weight = miss × level it carried = (−0.5) × 1 = −0.5 slope of the first weight = miss × second weight × gate's slope × input = (−0.5) × 0.5 × 1 × 2 = −0.5
to find how the first weight affects the error, multiply the rates along the wires between it and the output.
Rates multiply the way unit conversions do. Turning the first weight moves the slot's total 2 for 1. The open gate passes that along 1 for 1. The second weight moves the prediction 0.5 for 1. So the first weight moves the prediction 1 for 1.
Each of the four factors is a condition on the first weight. If there was no miss, it does not change. If the wire from its slot to the output is weak, it changes a little. If the gate was shut, it does not change. If its input was zero, it does not change. Step both weights by a tenth of their slopes and the error falls from 0.125 to 0.078.
The third factor is where the history turned. For an all-or-none slot it is zero everywhere, so the whole product is zero and nothing reaches the first weight. For the 1986 curve it is a number between 0 and 0.25. For the gate it is 1 or 0.
One token through a whole board
The board below has eight weights. An input to a board is called a token from here on, the word language models use. Follow one token down, then follow the correction back up.
The quantity that travels upward is called blame on these pages. A slot's blame is how fast the error would rise if that slot's total went up. Negative blame means the slot should have been higher.
Four weights moved, and all four sit on wires that carried signal for this token. The other four sit on wires out of slot B, which was silent, or into slot B, whose gate was shut.
Each change has a reason you can say aloud. Slot A was on and bin 1 should have been higher, so the weight from A to bin 1 went up. Slot A was on and bin 2 should have been lower, so the weight from A to bin 2 went down.
The rule for any board
going down level of a slot = gate( sum of weight × level above, plus offset ) coming up blame of a slot = (1 if its gate was open, else 0) × sum of weight × blame below adjusting weight = weight − step size × blame below × level above
signal goes down through the weights, blame comes back up through the same weights, and each weight changes by the product of the two numbers at its ends.
The blame comes back up the same wires because of what a weight does on the way down. A wire with a weight of 3 delivers three times the upper slot's level to the lower slot. So the lower slot's blame counts three times toward the upper slot's.
This procedure is called backpropagation. The whole of it is on this page.
Why backward, and what it saves
There is one error and there are many weights. Starting from the error, the part of the calculation that every weight shares gets done once.
On a network with 6,154 weights and offsets, rocking each one takes 6,155 runs of the network. The backward pass takes one run down and one run up, and gives the same slopes to nine decimal places. In a timing test the rocking took about 4,900 times longer. A model with a billion weights would need a billion and one runs for every example.
The pegboard, corrected
If you picture a network as a pegboard, most of the picture holds. The table marks the parts that need changing.
| The pegboard | Keep or change | The network |
|---|---|---|
| Rows of pegs | Keep | Rows of slots, with a table of weights between each pair of rows. |
| Bins at the bottom | Keep | One score per bin. The scores become probabilities. |
| Pegs of adjustable size | Keep | Weights. Each one scales what passes along one wire, and a negative weight subtracts. |
| A ball dropped in at the top | Change | A pattern of amounts across the whole top row. |
| The ball takes one route | Change | Every wire carries some of the pattern, and each slot adds up what arrives. |
| Different balls reach different bins | Keep | Gates open and shut depending on the input, so different inputs use different sets of wires. |
| A springy row that bounces the ball | Change | The gate acts in place. A positive total passes and a negative total becomes zero. Nothing goes back up while an input is being sorted. |
The first change is the one the field made in 1986. During training something does travel back up the board. It is the blame, and it uses the same wires the signal came down.
CARRY THIS FORWARD
Signal goes down through the weights. Blame comes back up through the same weights. Each weight moves by the product of the two numbers at its ends.