Training, weights and checkpoints

1. Data
2. Forward pass
3. Loss
4. Update the weights
5. Checkpoint
6. Evaluate

One turn of the training loop. Frontier models run it billions of times; the loop is the same at every scale.

Weights: what the file is

A model is a very large array of numbers, its parameters or weights, arranged in the layers the spec sheet page describes. A model with seven billion parameters stored at two bytes each is a fourteen-gigabyte file. That file holds nothing you would recognize: no sentences, no documents, no database of facts. It holds the settled state of the numbers after training, and the model's behavior is what those numbers do to a stream of tokens. The training data shaped the numbers, and it is not inside them in any form you can retrieve. That is why a model can state a fact fluently and be unable to say where it came from.

Quantization stores the same numbers in fewer bits to make the file smaller and the model faster. Open-weight models publish the file; closed-weight models keep it behind a service. Nothing about the file changes between a chat and an API call. The harness around it does.

Training: what happened to the numbers

Training starts with the numbers set at random and moves them, a little at a time, toward values that make the model's predictions less wrong. Each turn of the loop in the strip above does four things. A batch of text goes forward through the model. The model's prediction for each next token is compared with the token that came next, and the difference is summarized in one number, the loss. The maths works out which direction each weight should move to reduce that number. Every weight then takes a small step that way. Repeat over all the data, many times, and the loss falls. The model gets better at predicting the next token, which is the only thing it is asked to do.

For a language model that happens in stages, and the stages are where the assistant comes from:

Pre-training is the expensive stage, measured in thousands of GPUs and months, and it is why only a few organizations make frontier models. Everything after it is cheaper by orders of magnitude and is where a model's character is set. Fine-tuning is the same loop again on your data, with the numbers starting where the vendor left them.

What creating one involves

The loop is a small part of the work. A run that produces a usable model needs, in rough order of effort:

• Data: collecting it, removing duplicates, filtering what is wrong, harmful or not yours to use, and deciding the mix. Most of the calendar goes here.
• A tokenizer and an architecture: the vocabulary of pieces and the shape of the layers.
• The run itself: the hardware, the schedule of step sizes, the monitoring. Runs crash; that is what checkpoints are for.
• Evaluation on data the model never trained on, throughout, because the loss falling on the training data tells you almost nothing about behavior on new data.
• Post-training, the three lower rows of the table, each with its own data and its own evaluation.
• Safety and release: testing what the model will do when asked for the wrong thing, and writing down what it is and is not for.

The same list, scaled down, is the list for training a small classifier on your own data, and the training book walks through that with weather imagery: the loop, the loss, the held-out set and the decisions.

Checkpoints: which copy of the weights to keep

A checkpoint is a copy of the weights saved at a point during training. Runs save them on a schedule so a crash costs hours rather than weeks. The bigger use is choosing which one to ship. As training goes on, the loss on the training data keeps falling, but the score on held-out data rises, peaks, and then often gets worse as the model starts to memorize its training set instead of learning from it. The best model is the checkpoint at that peak, which is rarely the one at the end. Picking it needs a held-out set, a metric decided in advance, and the discipline to choose once. Trying every checkpoint against the final test until one looks good turns the test into more training data. The Evidence and evaluation page is that discipline for the models you use rather than the ones you train.

The four books

This guide stops here. Each of these goes further, and they were written to be read in any order.

What to do with this

Open the training book at the chapter on telling whether a model learned anything, and read it with a model your shop uses in mind. Then read how to know a model is good without training it yourself: Evidence and evaluation.