Page 7 of 9
What was proven, what was found by trying, and what is still open
The construction was reasoned out. How well it trains came as a surprise, and part of the surprise is still unexplained.
Proven
- One row reaches its best setting. Its error is a bowl with one bottom, so stepping downhill arrives (page 2).
- The slopes are exact. For any number of rows, the backward pass gives the true slope for every weight (page 5).
- The range is there. With enough slots in a middle row, some setting of the weights matches any continuous function (page 4).
None of these says that stepping downhill will find a good setting once there are middle rows. With middle rows the error is no longer a bowl. It can have places where every direction looks flat or uphill and the answers are still wrong.
Found by trying: it rarely gets stuck
You can run that experiment here. The task is the four corners from page 2, reduced to four points. Two gated slots are enough to hold the answer.
These are the rates from 2,000 starts at each width, trained by plain descent.
| Slots in the middle row | 1 | 2 | 3 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|---|---|
| Starts that reached the answer | 0% | 24.9% | 51.5% | 71.5% | 97.2% | 100% | 100% |
Two slots can hold the answer, and three starts in four still fail to reach it. With sixteen slots, eight times more than the answer needs, no start failed. The failed runs were stuck. Running them five times longer, or with a smaller step, left every rate unchanged.
The extra slots are not there to hold the answer. They are there so that descent can reach it.
The field did not take it on faith
A review written in 2015 by three of the people who kept working on these networks describes the mood of the late 1990s: “it was commonly thought that simple gradient descent would get trapped in poor local minima.” Hinton's lecture notes say that by then “most serious researchers in machine learning had given up on it.” The same review reports what practice showed: “poor local minima are rarely a problem with large networks.”
Later work supplied reasons. In a space with very many weights, a place where the slope is zero nearly always has some direction that still leads down. And for networks much wider than the data requires, plain descent provably reaches zero error on its training examples, under stated assumptions.
The worst case stays bad. Avrim Blum and Ronald Rivest proved in 1992 that for a network of three all-or-none units, deciding whether any weights fit a set of examples is among the hardest kinds of search problem. Practice has been far kinder than the worst case, and nobody can yet say exactly why.
LeCun, Bengio and Hinton, Deep learning ↗ · Hinton, lecture notes ↗ · Dauphin and others ↗ · Du and others ↗ · Blum and Rivest ↗
The shape of the bend
The theorems on page 4 say almost any bend gives full range. So the shape was chosen for the other job: letting the correction through. On its way back up the board, the correction is multiplied by the bend's slope at every row it passes.
| Year | Shape | Why it replaced the one before | How it was chosen |
|---|---|---|---|
| 1943 | Step: fire or don't | The model of a nerve cell. It gives logic. | Reasoned out |
| 1986 | Smooth S-curve: a rounded step | A slope has to exist for the correction to pass. | Reasoned out |
| 1990s | The same curve, centered on zero | Outputs that average near zero suit the next row better. | Reasoned out |
| 2010–12 | The gate: zero when negative, unchanged when positive | Slope of 1 whenever it is on, so the correction does not shrink. Cheap to compute. | Reasoned out then measured |
| 2016–17 | The gate with its corner rounded (GELU, Swish, SiLU) | Small measured gains. | Found by trying |
The fault in the smooth curve is that it flattens. Its slope is 0.25 at best, 0.045 when a slot's total is 3, and 0.0025 when the total is 6. Ten rows at the best possible slope leave about one millionth of a correction. The gate's slope is 1 whenever it is open, so ten open gates leave all of it.
Still open
The open question is why the weights that descent finds work on inputs the network never saw.
In 2017 Chiyuan Zhang and four coauthors showed that standard networks “easily fit a random labeling of the training data.” A network that can memorize noise has room to memorize anything, so its size alone cannot explain why, on real labels, it does well on new inputs. Two years later another group showed that the textbook trade-off between fitting and overfitting does not describe these models.
There are partial accounts. One is the idea from the last page, that corrections shared by many examples reinforce each other. Another comes from physics: the relationships that nature produces are a small, structured subset of all possible ones, and these networks express that subset cheaply.
How the weights get adjusted is fully understood. Why the adjusted weights are as good as they are is partly understood.
Zhang and others ↗ · Belkin and others ↗ · Chatterjee ↗ · Lin, Tegmark and Rolnick ↗
CARRY THIS FORWARD
The design was reasoned out, one part at a time. That it trains so well was found by trying, and more slots than the answer needs is a large part of why.