Let’s train a model
Hi, I’m Sean Egan. In this workshop we follow the numbers from a single input to a trained model, then train one on data of our own.
Part I
What the Machine Holds — and How to Shape Work for It
Read the shapes and find the independent work.
-
1A Number with a ShapeRead the containers a model holds7 min
2Space That StretchesA matrix moves a whole space at once7 min
3A Layer Is a MatrixWarp the input space, shift it, then bend it7 min
4Why a GPU, Not a CPUFollow the arithmetic to find the parallel work7 min
5From Loops to MatricesTurn repeated work into tensor operations7 min
6Shaping a Problem for the GPUChoose the axes before writing the computation7 min
7The Forward Pass, Every NumberFollow an input through two layers7 min
Part II
What the Machine Does, One Step at a Time
Follow the loss, gradients, updates, and dashboard.
-
8How Wrong Is It: Loss, Entropy, Cross-EntropyPut a number on a prediction’s error7 min
9Downhill: the GradientFit a line, one visible update at a time7 min
10Backwards: How the Gradient Reaches Every LayerTrace the chain rule through the model7 min
11The Training LoopA batch, a prediction, a loss, and an update8 min
12Reading the Graphs: What Training Should Look LikeThe four curves everyone watches, and the shapes a healthy run makes9 min
13What Gets Tracked, and What It Looks LikeRead the changing measurements of a training run11 min
Part III
The Architectures, Each Trained for Real
Inspect six model families as they learn.
-
14The Dense Network (MLP)Watch stacked layers learn a spiral boundary7 min
15The Convolutional Network (CNN)Train shared filters on handwritten digits7 min
16The Recurrent Network (RNN, GRU)Carry a hidden state through a sequence7 min
17The TransformerWatch attention change while a model learns names7 min
18The Graph Network (GNN)Learn by passing messages between connected nodes7 min
19The AutoencoderCompress a digit to two numbers and reconstruct it7 min
Part IV
Doing It Yourself
Prepare data, design a run, and evaluate your own model.
-
20Getting Data into TensorsShape, split, and normalize your examples7 min
21Designing the RunChoose the objective, model, batch, and learning rate7 min
22Did It Actually Learn? EvaluationCompare held-out predictions with a baseline7 min
23When It Goes WrongConnect symptoms to causes and checks7 min
24Train Your Own ModelRun trainkit on your data and read your weights10 min
25Glossary and NotationFind a term, its meaning, and the chapter that introduced it7 min
Part V
The Weather Models We Trained Here
Four weather models from the machine this book was written on: three we trained and one we designed. The same loop and the same graphs, on real satellite and radar data.
-
26True Color from Three BandsLearning a missing-green correction against a polar-orbiter reference9 min
27Super-Resolution, 2×A residual network that starts as bicubic and learns only what bicubic gets wrong9 min
28Nowcasting the RadarSixty minutes ahead, five minutes at a time, measured against the classical forecasts9 min
29Doppler EverywhereTurning cloud-top temperatures into reflectivity where no radar reaches9 min