The Training Run

One falling number

Somewhere, right now, a team of engineers is watching a dashboard the way NASA watches a launch. It shows one number. That number is this site's loss $L$ — Chapter 12's average of $-\ln p$ over fresh text — and it has been falling, slowly and expensively, for weeks. Behind it: buildings full of computers, a river of electricity, and a nine-figure bet that the curve will keep bending down.

Here is the good news you have earned: you already own every part of the machine that moves it. You can guess the next token, grade the guess, flow the blame backward, and take a step downhill. What you have not seen is that heartbeat running at industrial scale — what it eats (the data), how it takes its bites (batches), what its health chart looks like (the loss curve), why anyone could budget for it in advance (scaling laws), and what is in the box when it is over (the checkpoint). This chapter has no new math. It has new zeros.

The whole process — one enormous next-token exam over roughly everything humanity has written — has a name: pretraining. The "pre" is a promise that something comes after. Chapter 16 keeps it.

The test writes itself

Start with the food. Pretraining feeds on a corpus of text on the order of trillions of tokens — web pages, books, code, reference works, forums, most of the written world that could be gathered and cleaned. A trillion is a number that refuses to be felt, so let us force it. By Chapter 7's rule of thumb, a token is about three-quarters of a word, so a trillion tokens is roughly 750 billion words. Read nonstop at a brisk 250 words a minute and you would need about 5,700 years to finish. Modern runs do not stop at one trillion; several to many trillions is ordinary, which pushes the reading time toward the better part of a hundred thousand years. The model does the equivalent in weeks.

Now the idea that makes the whole enterprise possible — and it is worth saying slowly, because almost everyone gets it wrong. Nobody writes the answer key.

Slide a window along any sentence, and every position becomes an exam question whose answer is already sitting there — the next token of the actual text. Take "the cat sat on the ___": the text itself says mat. Grade the guess with Chapter 12's ruler, slide the window one token right, and the next position is a fresh question with its own free answer. The test writes itself out of the material being learned — no graders, no annotators, no labels bought by the item. This is self-supervised learning, the economic miracle at the heart of every large language model.

Sit with why that free-ness matters. Labeled data — a sentence paired with a human-written "correct output" — is bought one item at a time from slow, expensive people. Self-supervised data is bought by the terabyte from text the world has already produced. A trillion tokens is a trillion exam questions, at no cost beyond storage. No approach that needed humans to write answers could ever feed a model with billions of dials; this one can, and that single fact is why LLMs exist and older ideas did not scale.

One honesty note. "Roughly everything" is a convenient lie: real corpora are filtered, deduplicated, and weighted — clean code counted more valuable than comment spam, some sources upsampled and others discarded — and those unglamorous editorial choices shape a model's personality as surely as its architecture does. Data curation is a genuine craft; this site only waves at it.

A noisy compass

There is a problem hiding in that trillion. Chapter 12's honest loss $L(\theta)$ averages the penalty over all $N$ questions — trillions of them — but Chapter 13's walker needs a fresh downhill direction at every single step, and re-grading the whole corpus for one footstep is unthinkable. So each step grades a small random handful of questions instead. That handful is a batch.

Watch a toy batch of four questions come back from the grader with losses 0.7, 2.3, 0.4, and 1.0. The batch loss is their average:

In words: add the four penalties and divide by four — this batch reads 1.1.

Now draw a different random four, dials frozen at the same instant: the losses come back 0.5, 1.3, 0.9, 0.9, averaging to 0.9. Same model, same moment, different bite of the exam — different reading. The number the model reports about itself jitters with which questions it happened to be asked.

And so does the direction it decides to step. The gradient from a batch is not the true downhill; it is an estimate — a compass that points roughly downhill but wobbles from batch to batch. The algorithm that steps along these noisy estimates is the one the entire industry runs on: stochastic gradient descent (SGD). "Stochastic" is the formal word for "with dice," and the dice are Chapter 6's.

Here is the twist worth the whole section. That jitter is not a defect to engineer away — it is a gift: exactly the thermal jiggle from Chapter 13's annealing story, delivered for free. A perfectly steady compass marches into the nearest little dip and freezes there. A shivering one rattles the walker out of shallow scratches, and — paired with a decaying learning-rate $\eta$ (eta, the step size) — settles only into basins wide enough to hold it against the shaking. Cheap and useful is a rare combination; SGD is both at once.

Let us fix the vocabulary of scale while we are here. A step is one full heartbeat: grade a batch, flow the blame backward, nudge every dial once. Real frontier batches are enormous — on the order of millions of tokens each — and a run takes hundreds of thousands to millions of steps. Multiply it out loud: a bite of millions of tokens, taken hundreds of thousands of times, is how you chew through trillions.

One last piece — why a heartbeat is fast enough to repeat that often. A GPU is, at heart, a matrix-multiplication engine, and Chapter 3 already said the quiet part: the model is matrix multiplications. Thousands of GPUs share each batch, splitting both data and model among themselves, so a heartbeat that would take a single machine minutes takes the fleet a second or two. Run that for months at megawatt scale, and the entire cost of training is electricity and machine-time for a few hundred thousand heartbeats.

Reading the curve

The dashboard's health chart is simple to draw: loss on the vertical axis, steps on the horizontal. Before we read a real one, let us derive where it must begin — the payoff of a seed Chapter 12 planted on purpose. A freshly initialized model knows nothing, so it spreads its belief evenly across all $V \approx 50{,}000$ tokens of Chapter 7's vocabulary: $p = 1/50{,}000$ for every candidate. Its loss on any token is therefore $-\ln(1/50{,}000) = \ln 50{,}000$, and we can assemble that from Chapter 4's anchors without a calculator. Every $\times 10$ adds 2.3 to a log — that is Chapter 4's table read in the mirror, where each $\div 10$ subtracted 2.3, giving $\ln 0.1 \approx -2.3$. Ten thousand is four tens multiplied together, so its log is four 2.3s added up: $\ln 10{,}000 = 4 \times 2.3 = 9.2$. And $\ln 5 = \ln 10 - \ln 2 = 2.3 - 0.69 \approx 1.6$. Since $50{,}000 = 10{,}000 \times 5$, the logs add:

In words: a know-nothing model's loss is the log of the vocabulary size — about 10.8 — because spreading belief evenly over 50,000 tokens gives the truth just one part in 50,000.

Every pretraining curve on Earth starts near its own $\ln V$. And every altitude on the way down can be read back as a belief, running Chapter 12's ruler in reverse — each $-\ln$ undone by $e^x$. At $L = 10.8$ the truth gets 1 in 50,000: pure guessing. At $L = 4$, the truth's share is $e^{-4} = 1/54.6 \approx 2\%$ (you computed $e^4 = 54.6$ back in Chapter 9). At $L = 2.3$ it is exactly 10%. And at $L \approx 2$, it is $e^{-2} \approx 0.14 = 14\%$ (from Chapter 4's table) — and a 14% average share for the exact next token, chosen out of 50,000 candidates on arbitrary human text, is astonishingly good. Modern strong runs land somewhere around two-ish and below, varying with the data and the tokenizer; treat that as typical territory, not a specification.

Now read the curve's shape, feature by feature — this is the skill the section title promises, and the figure is its map. (1) The cliff. Within the first sliver of training the loss plunges from 10.8, because the cheapest wins are the biggest: the model learns that "the" is common and "zqx" is not, and that alone buys enormous progress. (2) The long grind. Then the expensive part — years of reading per week, shaving hundredths at a time. Grammar tends to settle first, then facts, then style and long-range coherence; researchers loosely observe that ordering when probing checkpoints, but it is an observation, not a law. (3) The fuzz. The line is never perfectly smooth — a permanent tremble, the noisy compass made visible. (4) The spikes. Occasionally the loss jumps upward: a batch of weird data, or a too-hot moment of the $\eta$ schedule, sends the walker briefly uphill — Chapter 13's rampage in miniature. Usually it recovers on its own; sometimes it forces engineers to rewind to an earlier save and route around the damage (hold that thought). (5) The slow flattening. Finally the curve bends toward the floor Chapter 12 promised — text's own irreducible uncertainty. It never touches zero, and never should.

Laws of scale

One question should be nagging you. Pretraining a frontier model costs a small nation's ransom — so how did anyone know, before spending it, that the curve would keep falling instead of stalling at 6? The answer is genuinely surprising: the endpoint of training turns out to be predictable. Make the model bigger, feed it more tokens, spend more compute, and the final loss falls along smooth, regular curves. Those empirical regularities are called scaling laws.

Here is how the regularity shows itself, using nothing but Chapter 4's log trick. Plot the final loss against the compute spent, but mark both axes in equal-multiples steps — each tick a $\times 10$ — the same compress-the-range-with-a-log move as Chapter 4's decibel aside. Do that, and the endpoints from many runs of many different sizes fall close to a straight line — and straight lines can be extended with a ruler. Train a family of small, cheap models, draw the line through their points, and read off what a run a thousand times larger should score before you build it. That extrapolation is what lets a lab bet the datacenter.

Two honest caveats keep this from sounding like magic. First, the budget has two levers, not one: model size and data size. For a fixed pile of compute there is a best balance between them, and a famous 2022-era result showed that many models of the day had been built oversized and underfed — researchers found that comparable amounts of parameters and tokens beat a giant model starved of data. Second, scaling laws promise the loss, and only the loss. They do not promise which specific abilities will appear at which altitude; that part still surprises people, in both directions, and no tidy story yet explains it.

The file at the end

The run ends. What do the engineers walk away with? A checkpoint — the walker's position, saved: the current value of every single dial, and essentially nothing else. All the matrices you have met — the embedding table of Chapter 8, every $\mathbf{W}_Q$, $\mathbf{W}_K$, and $\mathbf{W}_V$ of Chapter 10, the MLP and normalization weights of Chapter 11 — flattened into one enormous list of numbers and written to disk.

Size it by hand. A 7-billion-parameter model, stored at the standard 2 bytes — sixteen on/off switches — per number (Chapter 18 explains exactly what those switches buy you), is $7{,}000{,}000{,}000 \times 2 = 14{,}000{,}000{,}000$ bytes — 14 GB, a file that fits on a thumb drive and holds everything the model "knows." When someone says they "downloaded a model," this list of $\theta$ values is what moved across the wire. The code that runs it is comparatively tiny; the file is the model.

Checkpoints are saved throughout the run, too, for a blunt reason. At datacenter scale something is always failing, so a run writes itself to disk every few hours and resumes from the last good save after a crash — and when a spike poisons the walk (the rewind from a section ago), engineers restart from a pre-spike checkpoint and route around the bad stretch. The dashboard's falling number is punctuated by these quiet acts of insurance.

The final checkpoint of pretraining earns a name the next chapter leans on hard: the base model. Here is its one-sentence portrait. It is the most knowledgeable autocomplete ever built — and only an autocomplete. Ask it a question and it may answer, or it may continue your question with three more questions, because worksheets full of questions are a genre of text it has read plenty of. It has no preference for being helpful. It has a preference for being likely.

Why the LLM cares

Return to the dashboard from the hook, which you can now read completely. The falling number is the average of Chapter 12's $-\ln p$ over fresh batches. Its fuzz is the noisy compass. Its spikes are Chapter 13's $\eta$ pathologies at scale. Every downtick was purchased by Chapter 14's two-pass heartbeat — one forward, one backward — running across thousands of GPUs at once. And the team knew the destination months in advance because scaling laws had already drawn the line. Not one clause of that sentence is borrowed on credit; you built every part.

Place pretraining honestly in the model's biography. This run is where essentially all of the knowledge comes from — the facts, the grammar, the styles, the code idioms — because this is where essentially all of the data and compute are spent. Everything after (Chapter 16, and the LoRA trick of Chapter 17) is, in tokens and dollars, a rounding error by comparison — which makes what that rounding error achieves all the stranger.

Because there is a gap the base model cannot close on its own. Prompt it with "What is the capital of France?" and plausible continuations include the answer, Paris — but also " What is the capital of Germany?" (it has read a thousand worksheets listing questions in a row), or an essay about French geography teachers. It does not prefer the helpful continuation to the likely one, because to it there is no difference: helpfulness is not a thing it was ever graded on. The distance between "likely" and "helpful" is the last mechanism this site owes you.

Close on this. You have now watched the numbers inside the matrices — the very numbers you multiplied by hand back in Chapter 3 — actually get made. Nobody chose them. A falling red number chose them, one nudge at a time, several hundred thousand times over.