Two centuries of decisions, who made each one, and the reason they gave. Most were reasoned out or proven. A few were found by trying.
Choose a marker to read what was decided and why. Each entry links to its source. The shaded stretch is the 25 years when the obstacle was known and unsolved.
What the record shows
The usual short version is that people stumbled on a design that happened to work. The papers tell a different story, and they tell it in their authors' words.
A single row of adjustable weights was learning from examples by 1960, with a proof that it reaches its best setting. It had been known since 1943 that rows in the middle would give a network far more range. From 1961 the obstacle had a name: no way to hand a correction to a slot in a middle row, because slots fired all or nothing and an all-or-nothing step has no slope.
The 1986 design removed that obstacle on purpose. Its authors wrote down the requirement, that the bend must have a slope, and chose a unit that met it. The method for sending a correction backward through many stages already existed. Engineers had built it for rocket trajectories a quarter of a century earlier.
What nobody could reason out in advance was how the finished design would behave: whether stepping downhill would get stuck, and whether what it learned would hold on new inputs. Those were found by trying. The first turned out well, more so as networks grew. The second is still not fully explained.
So “it was designed” and “it was discovered” are both true, about different halves. The construction was designed. How well the construction trains was discovered.
Sources
Each entry says what it supports on these pages and links to where it can be read. One report, cleared for unlimited public release, is also offered here as a download.
Fitting and stepping downhill
Adrien-Marie Legendre (1805). On least squares, English translation of the appendix to his book on comet orbits.Choosing adjustable numbers by making the sum of squared errors as small as possible.Read at the source ↗
Augustin-Louis Cauchy (1847). General method for the solution of systems of simultaneous equations, English translation.Stepping downhill when the minimum cannot be solved for, with orbit calculations as the motive.Read at the source ↗
The unit, the weights, and the first learning machines
Warren McCulloch and Walter Pitts (1943). A logical calculus of the ideas immanent in nervous activity. Read here through lecture notes on the paper.The add-then-fire unit, modeled on the neuron, and networks of such units computing logical rules.Read the lecture notes ↗
Donald Hebb (1949). The Organization of Behavior. The postulate is quoted in Gerstner and others, Neuronal Dynamics, section 19.1.Learning as a change in the strength of a connection.Read at the source ↗
Frank Rosenblatt (1958). The perceptron: a probabilistic model for information storage and organization in the brain.Storing connection strengths in place of records, and the reason he gave.Read at the source ↗
Smithsonian National Museum of American History (object record). Electronic neural network, Mark I Perceptron.The machine itself: who built it, who paid for it, and its array of potentiometers.Read at the source ↗
Archived Futures (1960 manual). The Mark I Perceptron operators' manual, summary page.400 photocells, 512 middle units, 8 output units, and weights in motor-driven potentiometers.Read at the source ↗
Bernard Widrow (2005). Thinking about thinking: the discovery of the LMS algorithm.Rocking each weight in 1957 and 1958, the blackboard in 1959, and the rule running within half an hour.Read at the source ↗
Albert Novikoff (1962). On convergence proofs for perceptrons.If some line sorts every example, the perceptron rule finds one after a finite number of corrections.Read at the source ↗
The wall
Marvin Minsky (1961). Steps toward artificial intelligence.The credit-assignment problem, named.Read at the source ↗
Marvin Minsky and Seymour Papert (1969). Perceptrons. The passage on many-layered machines, traced by Quote Investigator.What one row cannot do, and their guarded judgment on more rows.Read at the source ↗
Bernard Widrow and Michael Lehr (1992). Backpropagation and its applications.Why the idea was missed through the 1960s: slots that fired all or nothing.Read at the source ↗
Wikipedia (accessed November 2026). Bernard Widrow.The limits of the Madaline work on a second row of weights.Read at the source ↗
Sending the correction backward
Stuart Dreyfus (1990). Artificial neural networks, back propagation, and the Kelley-Bryson gradient procedure.The chain rule run backward for rocket trajectories, 1960 to 1962.Read at the source ↗
Jürgen Schmidhuber (2014, updated). Who invented backpropagation?Linnainmaa's 1970 general method, and the order of the other contributions.Read at the source ↗
Paul Werbos (1974). Beyond Regression: new tools for prediction and analysis in the behavioral sciences.The backward calculation applied to networks, and the requirement that every operation be differentiable.Read at the source ↗
Paul Werbos (2015). Backwards differentiation in AD and neural nets: past links and new opportunities.Why early units were all or nothing.Read at the source ↗
David Rumelhart, Geoffrey Hinton and Ronald Williams (1986). Learning internal representations by error propagation, chapter 8 of Parallel Distributed Processing, volume 1.The requirement for a slope that picked the unit, the bowl for one row, the two contributions, and the cost of the backward pass.Read at the source ↗
David Rumelhart, Geoffrey Hinton and Ronald Williams (1986). Learning representations by back-propagating errors, Nature 323.Middle slots come to represent features, and extra connections give paths around the barriers.Read at the source ↗
Yuxi Liu (2023). The backstory of backpropagation.Rumelhart's own account, quoted from his 1993 interview for the book Talking Nets.Read at the source ↗
Geoffrey Hinton (2023). Interview on the algorithm powering modern AI, Radical Ventures.How Rumelhart's rule was recast as descent on an error.Read at the source ↗
Range: what the bend makes possible
George Cybenko (1989). Approximation by superpositions of a sigmoidal function.One middle row can approximate any continuous function.Read at the source ↗
Kurt Hornik, Maxwell Stinchcombe and Halbert White (1989). Multilayer feedforward networks are universal approximators.The same result, proved independently the same year.Read at the source ↗
Moshe Leshno, Vladimir Lin, Allan Pinkus and Shimon Schocken (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function.Any bend that is not a polynomial gives full range.Read at the source ↗
Andrew Barron (1993). Universal approximation bounds for superpositions of a sigmoidal function.Middle rows that learn beat middle rows that are fixed, more so as inputs grow.Read at the source ↗
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho and Yoshua Bengio (2014). On the number of linear regions of deep neural networks.Each row folds the space the rows before it built.Read at the source ↗
Matus Telgarsky (2016). Benefits of depth in neural networks.Some shapes need exponentially many slots in a shallow network.Read at the source ↗
The shape of the bend
Yann LeCun, Léon Bottou, Genevieve Orr and Klaus-Robert Müller (1998). Efficient BackProp.Why a curve centered on zero trains better than one that is not.Read at the source ↗
Vinod Nair and Geoffrey Hinton (2010). Rectified linear units improve restricted Boltzmann machines.The gate, under the name it has kept.Read at the source ↗
Xavier Glorot, Antoine Bordes and Yoshua Bengio (2011). Deep sparse rectifier neural networks.The correction passes whole along active paths; the worry about the hard zero, tested.Read at the source ↗
Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton (2012). ImageNet classification with deep convolutional neural networks.Six times faster with the gate than with the smooth curve.Read at the source ↗
Dan Hendrycks and Kevin Gimpel (2016). Gaussian error linear units (GELUs).A smoothed gate, with the history of the unit in its introduction.Read at the source ↗
Prajit Ramachandran, Barret Zoph and Quoc Le (2017). Searching for activation functions.Swish, found by automated search.Read at the source ↗
Stefan Elfwing, Eiji Uchibe and Kenji Doya (2017). Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.The same formula, proposed as SiLU.Read at the source ↗
Why training works, and what is open
Yann LeCun, Yoshua Bengio and Geoffrey Hinton (2015). Deep learning, Nature 521.The belief in the late 1990s that descent would get trapped, and what practice showed.Read at the source ↗
Geoffrey Hinton (lecture notes). The ups and downs of backpropagation.By the late 1990s most researchers had given up on it, and why.Read at the source ↗
Yann Dauphin and five coauthors (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.Flat spots in many dimensions are mostly saddles.Read at the source ↗
Simon Du and four coauthors (2019). Gradient descent finds global minima of deep neural networks.With enough slots, descent provably reaches zero training error.Read at the source ↗
Zeyuan Allen-Zhu, Yuanzhi Li and Zhao Song (2019). A convergence theory for deep learning via over-parameterization.The same conclusion by another route.Read at the source ↗
Avrim Blum and Ronald Rivest (1992). Training a 3-node neural network is NP-complete.No guarantee is possible in the worst case.Read at the source ↗
Chiyuan Zhang and four coauthors (2017). Understanding deep learning requires rethinking generalization.Networks fit random labels, so size alone does not explain why they work on new inputs.Read at the source ↗
Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal (2019). Reconciling modern machine-learning practice and the classical bias-variance trade-off.The textbook trade-off between fitting and overfitting does not describe these models.Read at the source ↗
Satrajit Chatterjee (2020). Coherent gradients: an approach to understanding generalization in gradient descent-based optimization.The proposal that corrections from similar examples reinforce each other.Read at the source ↗
Henry Lin, Max Tegmark and David Rolnick (2017). Why does deep and cheap learning work so well?An argument from physics about which functions need learning at all.Read at the source ↗
Language models
Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin (2003). A neural probabilistic language model.Learning each word's list of numbers along with the weights.Read at the source ↗
Ashish Vaswani and seven coauthors (2017). Attention is all you need.The block: two tables with a gate between, 512 wide at the ends and 2048 in the middle, and the table of alternatives.Read at the source ↗
Mark Sandler and four coauthors (2018). MobileNetV2: inverted residuals and linear bottlenecks.Why to widen before a gate and leave the narrowing step without one.Read at the source ↗
Mor Geva, Roei Schuster, Jonathan Berant and Omer Levy (2021). Transformer feed-forward layers are key-value memories.What the middle slots do in trained language models.Read at the source ↗
Nelson Elhage and coauthors (2021). A mathematical framework for transformer circuits.Every part of the model reads from and writes to one running list per token.Read at the source ↗
Adding evidence
David MacKay (2003). Information Theory, Inference, and Learning Algorithms, chapter 18.Turing and Good adding up weights of evidence at Bletchley Park.Read at the source ↗
Wikipedia (accessed November 2026). Ban (unit).The ban and the deciban, and Good's reading of them as weight of evidence.Read at the source ↗
The checks
The worked numbers on these pages come from two short programs. They need Python 3 and the numpy library, and nothing else.