Artificial intelligence & machine learning
How do machines learn?
A brief history of AI and machine learning, built one piece at a time. Every page has something to move, the arithmetic written out, and the reason someone thought of it.
The answer, before the details
A machine can learn from examples when three things are true.
- ONEIt has numbers it can adjustThey are called weights, and a model has thousands to billions of them. Everything the machine knows is in their values.
- TWOSome setting gives right answersThe machine has to be flexible enough that good values exist. One small operation, the bend, is what gives it that range.
- THREEEach number can be told which way to moveFor every weight, a short calculation says whether raising it would make the answer better or worse. Repeat that, and the weights settle.
The multilayer perceptron is the smallest design that has all three: rows of adjustable weights with a bend between the rows. It sits inside nearly every modern AI model, the language models included.
People often describe it as a lucky find. The record says otherwise. Each part was put there for a reason that someone wrote down, between 1805 and 1986, and these pages quote the reasons. What nobody could say in advance was how well the finished design would train. That was found by trying, and part of it is still unexplained. Page 7 keeps the two apart.
THE PART THAT MAKES THE DIFFERENCE
The whole design turns on one small operation, the bend. Without it every row is a table of weights, and the machine can only draw one straight line. With it, the same weights can fold space, and folding is what lets tangled things come apart. Page 4 shows the fold, and page 5 shows why the bend had to be given a slope before anything could learn.
The path
The path is nine short pages. Each one adds a single idea and opens on a picture you can work with your hands.
- One weightThe smallest learning machine: one number to adjust, a miss to shrink, and two ways to find which way is downhill.
- One row of weightsSeveral inputs and a weight for each. It learns with a guarantee, and all it can make is one straight cut.
- A table moves the spaceSeveral sums side by side carry every point somewhere new. Turning and stretching are what that looks like.
- The bendReplace negative totals with zero and space folds. Folding is the thing the tables could not do.
- Blame travels backwardHow a weight deep inside finds out which way to move. Unsolved for 25 years, then fixed on purpose.
- The space takes shapeEvery example tugs its point toward its own bin. Watch the kinds come apart.
- Why it works as well as it doesWhat was proven, what was found by trying, and what nobody can explain yet.
- Inside a language modelThe same block, thousands of slots wide, applied to one token at a time.
- How we got hereTwo centuries of decisions, who made each one and why, with every source.
Two centuries, one decision at a time
Most steps were reasoned out or proven. A few were found by trying. For 25 years in the middle the obstacle was known and unsolved.
Words used on these pages
The field's vocabulary grew up over eighty years and is harder than the ideas. These pages use plain words and keep to them. The textbook word is on the right for when you meet it elsewhere.
| On these pages | What it means | In textbooks |
|---|---|---|
| slot | One place that holds a number. | neuron, unit |
| level | The number a slot is holding right now. | activation |
| weight | The number on a wire between two slots. It multiplies what passes along the wire. | weight, parameter |
| offset | A number a slot adds to its total. | bias |
| row | A set of slots that work side by side. | layer |
| table of weights | All the weights between one row and the next. | weight matrix |
| gate, the bend | Pass a positive total unchanged. Replace a negative total with zero. | activation function, nonlinearity, ReLU |
| score | A bin's total before it is turned into a probability. | logit |
| error | One number for how wrong an answer is. | loss |
| slope | How fast the error changes when one weight changes. | gradient, partial derivative |
| blame | How fast the error changes when one slot's total changes. | error signal, delta |
| step size | How far to move a weight for a given slope. | learning rate |
| token | A word or a piece of a word, once it has been turned into a list of numbers. | token, embedding |
Where the numbers come from
The worked numbers on these pages come from two short programs, and each quotation links to the paper it came from. The programs and their output are here to run or read: verify_mlp_claims.py ↗, worked_board.py ↗, verify_results.json ↗, worked_board_results.json ↗. The sources are listed on the last page.