The instruction number that was one digit off
A colleague asks the model which instruction covers a routine program. It answers in about a second. The series is right, the office is right, the shape of the number is right, and one digit is wrong. Nothing in the reply looks like a guess. That is what makes it dangerous, and it is what people mean by a hallucination.
The model did not stop because it had no reason to stop. The grading rule it trained under pays nothing for silence, so a guess beats an abstention. And remembering a fact, inside this model, is the same act as finishing a sentence.
The exam has no blank answer
Training pays nothing for "I don't know" because the grading rule has nowhere to put it. Back in Chapter 12 we built that rule. At every position in the training text the model produces a probability distribution over the whole vocabulary: a share of belief for every token it might put next, with the shares adding to one. The text then reveals which token came next, and the rule reads one number off that distribution:
$$L = -\ln p(\text{the token that actually came next})$$In words: find the share the model gave the token the text used, take its natural log, and flip the sign. That log is the exponent you would raise $e \approx 2.718$ to in order to get that share, so full belief scores zero and every division of the share by ten adds about 2.3 to the penalty. This grade is called cross-entropy, and the unit of the natural-log version is the nat.
Read the rule again for what it does not ask. It never asks whether the model knew. It asks how much belief landed on what came next. Put three models in front of the same position, the token right after "OPNAVINST", and grade them.
| Model | What it does | Share on the right token | Loss $L$ (nats) |
|---|---|---|---|
| A | Commits to the digits it half-remembers | 0.60 | 0.51 |
| B | Hedges across five plausible instructions | 0.20 | 1.61 |
| C | Commits to the wrong digits | 0.05 | 3.0 |
Grading the three habits
Model A puts 0.6 of its belief on the digits that came next. Model B spreads 0.2 across five candidates, one of them right. Model C leaves 0.05 on the truth. Grade all three with the same rule:
$$-\ln 0.6 = 0.51 \qquad -\ln 0.2 = 1.61 \qquad -\ln 0.05 = 3.0$$In words: the confident model that lands pays 0.51, the hedger pays 1.61 every time, and the confident model that misses pays 3.0. Hedging costs about a nat on every question, including the ones where the guess would have landed.
There is no fourth row for silence, because silence is not a token. A model that says nothing still has to put its shares somewhere, and whatever it withholds from the true token is charged to it. The closest thing to abstaining is spreading belief thin, which is row B, and row B loses to row A whenever the guess lands. Across trillions of training positions the habit that scores better on average survives, and on any fact the model has seen enough of, that habit is to commit.
The second half of training keeps the same shape. After pretraining comes the tuning we met in Chapter 16, where human preferences and public leaderboards pick the winner between two replies. Those graders mostly score a plausible answer at least as well as "I don't know". A scoreboard that gives a point for a right answer and zero for both a wrong answer and an abstention rewards guessing, the way a multiple-choice test does. Kalai, Nachum, Vempala and Zhang argue this in "Why Language Models Hallucinate" (2025): the scoring selects for hallucination, so it is the strategy that survives training.
Recall is generation
When you ask a model for a fact, nothing different happens inside it. It runs the same pass we built in Chapter 9: scores for every token, softmax to turn them into shares, then one sampling draw from those shares. Asking "what is the capital of France" and asking it to finish "the cat sat on the" are the same operation. Only the shape of the distribution differs.
After "The capital of France is", the training text agreed with itself several million times, and the shares collapse onto one token. After "My favorite color is", the text never agreed, and the shares spread out. One number summarizes that spread: the entropy, which is 0.16 for the spike and 2.23 for the spread. See what a spread die costs in bits. Those two are in bits rather than nats, the usual unit for entropy; one nat is about 1.443 bits.
Now take a fact the training text mentioned twice. Its distribution looks like the right-hand panel: a spread over several instruction numbers that all look right. The sampler has no way to read that spread as "unsure". It draws from it the way it draws from every other position, and out comes a well-formed instruction number. Fluent wording tells you nothing about how concentrated the shares were.
What the weights hold
The weights hold how language connects, not a set of records waiting to be queried. Training built a network that tracks how words connect to phrases, phrases to sentences, and sentences to ideas. Predicting the next token is the objective the network was trained against; it is not a description of what the network does. See objective versus mechanism.
Facts do live in there, as a side effect. To predict text about France well, the network had to encode that Paris follows "the capital of France is". A fact repeated a million times gets pressed into the weights hard enough to make a spike; a fact seen twice leaves a smudge. Pressing is compression rather than filing, so there is no index, no record boundary, and no way for the model to report that a particular smudge is thin.
So put the material in front of it
The same model that guesses badly from memory is reliable when the answer is in front of it, and we built the reason in Chapter 10. Attention lets a token look back over the context and pull what it needs. Stack two layers and the network forms an induction head, which finds where a string appeared earlier and copies what came after it. Copying from the context is the most strongly learned behavior these models have, because in real text it is right far more often than any guess.
So paste the instruction's own paragraph into the prompt and ask the question again. The context now contains the digits. The shares that were spread over five candidates collapse onto the one sitting there in the text.
The same question, twice
Asked from memory, the share on the right digits was 0.20, for a loss of 1.61 nats. Asked with the paragraph attached, the share on those same digits is 0.97:
$$-\ln 0.97 \approx 0.03$$In words: the penalty falls from 1.61 to about 0.03. The model learned nothing in between. The same weights, given a context that contains the answer, put nearly all their belief in one place.
| From memory | With the paragraph attached | |
|---|---|---|
| Where the answer comes from | Shares pressed into the weights | Tokens sitting in the context |
| Share on the right digits | 0.20 | 0.97 |
| What varies between runs | Which plausible number is drawn | Wording, not the digits |
| What you can check | Nothing, because there is no source | The paragraph you supplied |
Use the model the way training built it. As an archive it is poor. At reading, connecting and transforming material you hand it, it is good.
Why the LLM cares
Everything in Part IV follows from this chapter. Retrieval-augmented generation, the subject of Chapter 20, is this idea industrialized: search a library for the passages that answer the question, staple them into the context, and let the model read them. Telling an assistant "say you don't know rather than guess" does help, but it only reshapes which continuations the model treats as acceptable. It does not put the instruction number into the weights.
What you now know
- Cross-entropy grades the share placed on the token that came next, so silence earns nothing and a guess that lands earns full marks.
- Hedging across five candidates costs 1.61 nats every time, where committing and landing costs 0.51, so guessing is the habit training selects.
- Recall and generation are the same step: a distribution over next tokens and one draw from it.
- A fact seen a million times gives a 0.16-bit spike; a context the text never agreed on gives a 2.23-bit spread, and a spread distribution still reads fluently.
- The weights hold how language connects rather than a queryable archive, so put the material in the context and let the model work from it.
Where we're headed
So far the weights have been fixed and we changed behavior by changing the context. The other lever is to change the weights, cheaply, in one narrow direction, without paying for a second training run. Next we'll adapt a large model with small matrices.