Skip to the page
HOW MACHINES LEARN
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9

Artificial intelligence & machine learning

How do machines learn?

A brief history of AI and machine learning, built one piece at a time. Every page has something to move, the arithmetic written out, and the reason someone thought of it.

A small network learning to tell two kinds of points apart. The first picture shows the points as given. The second shows where the network has moved them. Each adjustment shifts every weight a little, the points drift, and the two kinds come apart. Switch the bend off to see what it was doing.

The answer, before the details

A machine can learn from examples when three things are true.

  1. ONEIt has numbers it can adjustThey are called weights, and a model has thousands to billions of them. Everything the machine knows is in their values.
  2. TWOSome setting gives right answersThe machine has to be flexible enough that good values exist. One small operation, the bend, is what gives it that range.
  3. THREEEach number can be told which way to moveFor every weight, a short calculation says whether raising it would make the answer better or worse. Repeat that, and the weights settle.

The multilayer perceptron is the smallest design that has all three: rows of adjustable weights with a bend between the rows. It sits inside nearly every modern AI model, the language models included.

People often describe it as a lucky find. The record says otherwise. Each part was put there for a reason that someone wrote down, between 1805 and 1986, and these pages quote the reasons. What nobody could say in advance was how well the finished design would train. That was found by trying, and part of it is still unexplained. Page 7 keeps the two apart.

THE PART THAT MAKES THE DIFFERENCE

The whole design turns on one small operation, the bend. Without it every row is a table of weights, and the machine can only draw one straight line. With it, the same weights can fold space, and folding is what lets tangled things come apart. Page 4 shows the fold, and page 5 shows why the bend had to be given a slope before anything could learn.

The path

The path is nine short pages. Each one adds a single idea and opens on a picture you can work with your hands.

  1. One weightThe smallest learning machine: one number to adjust, a miss to shrink, and two ways to find which way is downhill.
  2. One row of weightsSeveral inputs and a weight for each. It learns with a guarantee, and all it can make is one straight cut.
  3. A table moves the spaceSeveral sums side by side carry every point somewhere new. Turning and stretching are what that looks like.
  4. The bendReplace negative totals with zero and space folds. Folding is the thing the tables could not do.
  5. Blame travels backwardHow a weight deep inside finds out which way to move. Unsolved for 25 years, then fixed on purpose.
  6. The space takes shapeEvery example tugs its point toward its own bin. Watch the kinds come apart.
  7. Why it works as well as it doesWhat was proven, what was found by trying, and what nobody can explain yet.
  8. Inside a language modelThe same block, thousands of slots wide, applied to one token at a time.
  9. How we got hereTwo centuries of decisions, who made each one and why, with every source.

Two centuries, one decision at a time

Most steps were reasoned out or proven. A few were found by trying. For 25 years in the middle the obstacle was known and unsolved.

Choose a marker. The full account, with quotations and sources, is on the history page.

Words used on these pages

The field's vocabulary grew up over eighty years and is harder than the ideas. These pages use plain words and keep to them. The textbook word is on the right for when you meet it elsewhere.

On these pagesWhat it meansIn textbooks
slotOne place that holds a number.neuron, unit
levelThe number a slot is holding right now.activation
weightThe number on a wire between two slots. It multiplies what passes along the wire.weight, parameter
offsetA number a slot adds to its total.bias
rowA set of slots that work side by side.layer
table of weightsAll the weights between one row and the next.weight matrix
gate, the bendPass a positive total unchanged. Replace a negative total with zero.activation function, nonlinearity, ReLU
scoreA bin's total before it is turned into a probability.logit
errorOne number for how wrong an answer is.loss
slopeHow fast the error changes when one weight changes.gradient, partial derivative
blameHow fast the error changes when one slot's total changes.error signal, delta
step sizeHow far to move a weight for a given slope.learning rate
tokenA word or a piece of a word, once it has been turned into a list of numbers.token, embedding

Where the numbers come from

The worked numbers on these pages come from two short programs, and each quotation links to the paper it came from. The programs and their output are here to run or read: verify_mlp_claims.py ↗, worked_board.py ↗, verify_results.json ↗, worked_board_results.json ↗. The sources are listed on the last page.