One falling number
A training dashboard turns weeks of computation into a changing loss curve. The cycle behind it is the one we built: predict, grade, backpropagate, update. This chapter adds the data, the batches and the file at the end.
Pretraining builds the model before assistant-specific training.
The test writes itself
Self-supervised learning takes its labels from the text: at each position, the next token is already present.
How much text is a trillion tokens?
At roughly 0.75 English words per token, one trillion tokens is 750 billion words. Dividing by 250 words per minute gives 3 billion minutes. Dividing by 60×24×365 gives about 5,700 years of nonstop reading. This estimates text volume, not equivalence to human learning.
Labels need no separate answer-writing step, but data collection is not free. Data curation removes repeated or unwanted material and balances sources. Those choices affect what the model learns.
A noisy compass
A batch replaces a full-corpus calculation with a manageable sample.
Two batches, unchanged model
$$L_{\text{batch}} = \frac{0.7 + 2.3 + 0.4 + 1.0}{4} = 1.1$$In words: add the four penalties and divide by four, and this batch reads 1.1.
A second batch has losses 0.5,1.3,0.9,0.9, averaging 0.9. Different examples change the reading even with fixed weights.
Stochastic gradient descent uses this noisy estimate. “Stochastic” means involving randomness. Practical runs often use Adam or another optimizer with batch gradients. Noise changes the trajectory; it is not guaranteed to help.
A step changes the parameters once. An epoch is one full pass through the training data. A million tokens per update, repeated a million times, processes a trillion tokens.
A GPU performs many multiplications together. Training may split examples across devices, split the model itself, or both; communication and memory can limit the speedup.
Reading the curve
A uniform predictor over V=50,000 candidates gives each target probability 1/50,000. Since $\ln10000\approx9.2$ and $\ln5\approx1.6$:
$$L_{\text{start}} = -\ln\!\left(\frac{1}{50{,}000}\right) = \ln 50{,}000 \approx 9.2 + 1.6 = 10.8$$In words: a know-nothing model's loss is the log of the vocabulary size, about 10.8, because spreading belief evenly over 50,000 tokens gives the truth one part in 50,000.
| Shape | Possible meaning |
|---|---|
| Steep early fall | Easy statistical patterns improve predictions quickly. |
| Slow decline with fuzz | Smaller improvements plus varying batch difficulty. |
| Sudden spike | An unusual batch, numerical trouble, or an unstable update; inspect the run before diagnosing. |
| Training falls, validation flattens or rises | Possible overfitting; consider an earlier checkpoint and check data or distribution differences. |
A validation set checks performance on data not used for gradient updates. Overfitting can lower training loss while validation worsens.
Interpret average loss using geometric mean target probability: L=4 corresponds to $e^{-4}\approx2\%$; L=2.3 to about 10%; L=2 to about 14%. Uniform loss ln V is a reference, not a mandatory starting point. A plateau need not be language's irreducible floor. Tokenizer and evaluation data affect the comparison.
Laws of scale
Scaling laws summarize families of experiments. Equal-ratio ticks on logarithmic axes can make a power-law trend look nearly straight.
| Lever | Lesson |
|---|---|
| Model size | More parameters can help when paired with sufficient data and training. |
| Data size | A larger model can be undertrained; parameter and token budgets must be balanced. |
| Compute | Small experiments can forecast larger runs within a tested regime. |
| Specific capabilities | A predicted loss does not promise a particular skill or guarantee extrapolation. |
The 2022 compute-optimal training work showed that several contemporary large models were underfed with data. It did not say that the number of tokens should equal the number of parameters.
The file at the end
A checkpoint records training state. Its model parameters fill the arrays in the architecture.
Size the weights
Seven billion parameters at two bytes each need 14,000,000,000 bytes, or 14 GB in decimal units. That is weights alone. Resuming training also requires optimizer state, progress and random-state information; running the model needs its architecture and tokenizer.
Periodic saves let a run recover after failure or return to an earlier useful state. The base model is the result before assistant shaping. It learned to continue text, so a question can prompt an answer or more questions in a worksheet.
See it move
Why the LLM cares
Pretraining is where broad language patterns enter the parameters. We assess them with held-out evaluation, inspect anomalies, and save useful states. A falling training loss alone cannot establish reliable generalization.
The resulting base model can continue many genres. To make it reliably respond as an assistant, we need examples and objectives that reward that behavior.
What you now know
- Next-token text supplies its own training targets.
- Batches estimate loss and gradients with sampling noise.
- Training and validation curves answer different questions.
- Scaling laws guide budgets within their measured regime.
- Weights alone are smaller than a complete resumable checkpoint.
- A pretrained base model has not yet been specifically shaped as an assistant.
Where we're headed
Next we'll shape that base model into an assistant, using new examples and preference signals.