Skip to the page
HOW MACHINES LEARN
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9

Page 7 of 9

What was proven, what was found by trying, and what is still open

The construction was reasoned out. How well it trains came as a surprise, and part of the surprise is still unexplained.

Proven

None of these says that stepping downhill will find a good setting once there are middle rows. With middle rows the error is no longer a bowl. It can have places where every direction looks flat or uphill and the answers are still wrong.

Found by trying: it rarely gets stuck

You can run that experiment here. The task is the four corners from page 2, reduced to four points. Two gated slots are enough to hold the answer.

Press Run 100 starts at every width. Each tile is one training run from fresh random weights. Mint means it reached the answer and rose means it got stuck. Tap a tile to see how that run ended.

These are the rates from 2,000 starts at each width, trained by plain descent.

Slots in the middle row123481632
Starts that reached the answer0%24.9%51.5%71.5%97.2%100%100%

Two slots can hold the answer, and three starts in four still fail to reach it. With sixteen slots, eight times more than the answer needs, no start failed. The failed runs were stuck. Running them five times longer, or with a smaller step, left every rate unchanged.

The extra slots are not there to hold the answer. They are there so that descent can reach it.

The field did not take it on faith

A review written in 2015 by three of the people who kept working on these networks describes the mood of the late 1990s: “it was commonly thought that simple gradient descent would get trapped in poor local minima.” Hinton's lecture notes say that by then “most serious researchers in machine learning had given up on it.” The same review reports what practice showed: “poor local minima are rarely a problem with large networks.”

Later work supplied reasons. In a space with very many weights, a place where the slope is zero nearly always has some direction that still leads down. And for networks much wider than the data requires, plain descent provably reaches zero error on its training examples, under stated assumptions.

The worst case stays bad. Avrim Blum and Ronald Rivest proved in 1992 that for a network of three all-or-none units, deciding whether any weights fit a set of examples is among the hardest kinds of search problem. Practice has been far kinder than the worst case, and nobody can yet say exactly why.

LeCun, Bengio and Hinton, Deep learning ↗ · Hinton, lecture notes ↗ · Dauphin and others ↗ · Du and others ↗ · Blum and Rivest ↗

The shape of the bend

The theorems on page 4 say almost any bend gives full range. So the shape was chosen for the other job: letting the correction through. On its way back up the board, the correction is multiplied by the bend's slope at every row it passes.

Choose a bend and move the Rows slider. The amber bars show how much of a correction is left after passing back through that many rows.
YearShapeWhy it replaced the one beforeHow it was chosen
1943Step: fire or don'tThe model of a nerve cell. It gives logic.Reasoned out
1986Smooth S-curve: a rounded stepA slope has to exist for the correction to pass.Reasoned out
1990sThe same curve, centered on zeroOutputs that average near zero suit the next row better.Reasoned out
2010–12The gate: zero when negative, unchanged when positiveSlope of 1 whenever it is on, so the correction does not shrink. Cheap to compute.Reasoned out then measured
2016–17The gate with its corner rounded (GELU, Swish, SiLU)Small measured gains.Found by trying

The fault in the smooth curve is that it flattens. Its slope is 0.25 at best, 0.045 when a slot's total is 3, and 0.0025 when the total is 6. Ten rows at the best possible slope leave about one millionth of a correction. The gate's slope is 1 whenever it is open, so ten open gates leave all of it.

Still open

The open question is why the weights that descent finds work on inputs the network never saw.

In 2017 Chiyuan Zhang and four coauthors showed that standard networks “easily fit a random labeling of the training data.” A network that can memorize noise has room to memorize anything, so its size alone cannot explain why, on real labels, it does well on new inputs. Two years later another group showed that the textbook trade-off between fitting and overfitting does not describe these models.

There are partial accounts. One is the idea from the last page, that corrections shared by many examples reinforce each other. Another comes from physics: the relationships that nature produces are a small, structured subset of all possible ones, and these networks express that subset cheaply.

How the weights get adjusted is fully understood. Why the adjusted weights are as good as they are is partly understood.

Zhang and others ↗ · Belkin and others ↗ · Chatterjee ↗ · Lin, Tegmark and Rolnick ↗

CARRY THIS FORWARD

The design was reasoned out, one part at a time. That it trains so well was found by trying, and more slots than the answer needs is a large part of why.