Skip to the page
HOW MACHINES LEARN
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9

Page 2 of 9

One row of weights draws one straight line

Give the machine several inputs and a weight for each. It learns by the same rule, with a guarantee, and the only thing it can make is a straight cut.

Press Learn. The three dials are adjusted by the rule on this page, and the line swings into place by itself. Then switch the points to Four corners or Ring in a ring and press Learn again.

A thing becomes a list of numbers

Anything a machine works on has to be numbers first. A dot on a page is two numbers, across and up. A photograph is one number per pixel. A word, as a later page shows, becomes a list of a few hundred numbers.

Whatever numbers a machine is holding at some place are called its levels on these pages. Textbooks call them activations. An activation is nothing more than a number that a part of the machine is holding at the moment.

A weighted sum is a vote

With two inputs the machine has a weight for each, plus one more adjustable number, the offset, which shifts the total.

score = weight 1 × input 1  +  weight 2 × input 2  +  offset

multiply each input by its own weight, add the results, and add the offset.

Each weight says how much its input counts, and in which direction. A positive weight votes for, a negative weight votes against, and a weight near zero means the input is ignored. If the score comes out above zero the machine says orange. Below zero, it says cyan.

The points where the score is zero form a straight line. The weights set which way the line tilts, and the offset slides it sideways. So this machine, whatever its weights, splits the page with one straight cut.

The same rule, once per weight

Nothing new is needed to train it. Each weight gets its own slope, by the calculation from the last page.

miss              = score − target       target is +1 for orange, −1 for cyan
slope of weight 1 = miss × input 1
slope of weight 2 = miss × input 2
slope of offset   = miss

every weight is corrected in proportion to the miss and to how much signal it was carrying.

Why it is guaranteed to work

For a weighted sum, the error as a function of the weights is a bowl with one lowest point. There is no false bottom to get caught in. Stepping downhill from any start reaches the best line this machine can make.

Widrow wrote of finding “the bottom of the bowl.” The authors of the 1986 paper that page 5 turns on say the same of this one-row case: “the error surface is shaped like a bowl with only one minimum, so gradient descent is guaranteed to find the best set of weights.”

Frank Rosenblatt's machine, built at the same time, used a slightly different rule and had its own guarantee. If some line sorts every example correctly, his rule finds one after a finite number of corrections.

Why weights, and why adding

Why store weights instead of answers

The obvious way to make a machine give right answers is a table that lists the right answer for every input. Rosenblatt weighed that choice in 1958. In one view of memory, he wrote, what is stored is a record of each thing seen, and recognizing something means comparing it against the records. He took the other view, in which nothing is recorded and

His reason: if what is stored is connections, a new input that resembles old ones will “make use of these new pathways which have been created, automatically activating the appropriate response without requiring any separate process for their recognition or identification.”

The logic underneath is sharing. A table has one entry per possible input, and correcting one entry does nothing for any other input. A weight sits on a wire that many inputs use. Correct it for one input and the answer changes for every input that sends signal down that wire.

A small test shows the size of the effect. Take five yes-or-no inputs, which allow 32 different inputs, and a true rule that is a weighted vote. Shown 20 of the 32, a weighted sum got 94% of the 12 it had never seen right, averaged over 2,000 trials. A table has no entry for those 12 and can only guess. The test favors weights, because the true rule was itself a weighted vote. It shows how sharing works. It does not promise the same score on other rules.

Why adding is a sound way to combine clues

The first reason is the one the designers had. A nerve cell responds to the total of what arrives at it, and Rosenblatt gave each connection its own strength.

The second is that a weighted sum was the one kind of formula scientists already knew how to fit to data, by Legendre's method.

The third is about evidence. If each input is an independent clue, the rule for combining clues, Bayes' rule, says to add one number per clue. In a check with five yes-or-no clues, Bayes' rule and “weighted sum, then squash to a probability” agree on all 32 combinations to sixteen decimal places.

Alan Turing and Jack Good did this arithmetic by hand at Bletchley Park in 1940. As evidence came in for a setting of the Enigma machine they added up units they called decibans, and Good later called the running total the weight of evidence.

MacKay, Information Theory, Inference, and Learning Algorithms, chapter 18 ↗ · Wikipedia, Ban (unit) ↗

Where it stops

Switch the points at the top of the page to Four corners. No straight line puts both orange clusters on one side and both cyan clusters on the other. Press Learn and the line drifts without settling anywhere useful. The rule is doing its job. The machine has no setting that works.

Marvin Minsky and Seymour Papert proved limits of this kind in 1969. The way out was already known in principle: more rows. Warren McCulloch and Walter Pitts had shown in 1943 that networks with rows in the middle compute rules that one row cannot. The trouble was training them, and that takes the next three pages.

CARRY THIS FORWARD

One row of weights learns, with a guarantee. All it can learn is one straight cut.