Skip to the page
HOW MACHINES LEARN
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9

Page 9 of 9

How we got here

Two centuries of decisions, who made each one, and the reason they gave. Most were reasoned out or proven. A few were found by trying.

Choose a marker to read what was decided and why. Each entry links to its source. The shaded stretch is the 25 years when the obstacle was known and unsolved.

What the record shows

The usual short version is that people stumbled on a design that happened to work. The papers tell a different story, and they tell it in their authors' words.

A single row of adjustable weights was learning from examples by 1960, with a proof that it reaches its best setting. It had been known since 1943 that rows in the middle would give a network far more range. From 1961 the obstacle had a name: no way to hand a correction to a slot in a middle row, because slots fired all or nothing and an all-or-nothing step has no slope.

The 1986 design removed that obstacle on purpose. Its authors wrote down the requirement, that the bend must have a slope, and chose a unit that met it. The method for sending a correction backward through many stages already existed. Engineers had built it for rocket trajectories a quarter of a century earlier.

What nobody could reason out in advance was how the finished design would behave: whether stepping downhill would get stuck, and whether what it learned would hold on new inputs. Those were found by trying. The first turned out well, more so as networks grew. The second is still not fully explained.

So “it was designed” and “it was discovered” are both true, about different halves. The construction was designed. How well the construction trains was discovered.

Sources

Each entry says what it supports on these pages and links to where it can be read. One report, cleared for unlimited public release, is also offered here as a download.

Fitting and stepping downhill

The unit, the weights, and the first learning machines

The wall

Sending the correction backward

Range: what the bend makes possible

The shape of the bend

Why training works, and what is open

Language models

Adding evidence

The checks

The worked numbers on these pages come from two short programs. They need Python 3 and the numpy library, and nothing else.