The parrot in the room
Ask a base model for France's capital and it might answer “Paris,” or continue with questions about Germany and Spain. Both resemble text it has seen. Post-training shifts the model toward useful assistant responses.
| Stage | Data | Objective | What changes |
|---|---|---|---|
| Pretraining | Broad text. | Predict observed next tokens. | General language representations and knowledge. |
| Supervised fine-tuning | Curated demonstrations. | Produce desired responses. | Task behavior, format, and potentially knowledge. |
| Preference training | Response comparisons or feedback. | Favor preferred behavior. | The relative likelihood of possible responses. |
This is a common pattern rather than a requirement that every assistant has three distinct runs.
Teaching by demonstration
Fine-tuning reuses learned weights. Supervised fine-tuning, SFT, teaches the desired response through examples.
A chat template marks user and assistant turns. In this common SFT setup, loss masking excludes the user prompt from the scored targets. Assistant outputs are graded token by token.
Worked example
Take one training conversation. The user turn reads "Translate hello to French." and the assistant turn reads "Bonjour." Look at the first response token, "Bon." Before SFT, in this chat format the base model spreads its belief, so the helpful answer gets about a one-in-ten share and its loss on that token is the landmark from Chapter 12:
$$p(\text{Bon}) \approx 0.1 \quad\Rightarrow\quad L = -\ln 0.1 = 2.3$$In words: if the right first token gets only a one-in-ten share of the model's belief, the ruler charges 2.3, the one-in-ten penalty from Chapter 12.
The descent does its work over thousands of such examples. After SFT, that same token gets nearly all of the belief:
$$p(\text{Bon}) \approx 0.9 \quad\Rightarrow\quad L = -\ln 0.9 \approx 0.11$$In words: once "answer helpfully here" is the near-certain continuation, the penalty falls to 0.11, the same landmark as in Chapter 12. The ruler did not change; the model's belief did.
Our example illustrates making an existing French response likelier. Fine-tuning can also add knowledge or damage existing skills; that depends on data, objective, and update size.
The taste problem
Two answers can both resemble helpful demonstrations while differing in accuracy, clarity, or appropriate caution. Comparing them may be easier than writing a perfect answer from scratch. Human judgments can still disagree or favor superficial traits.
Learning from comparisons
Each pair has a chosen response and a rejected response. Preference learning uses those comparisons.
A reward r rates a response. One model of human preference applies the sigmoid σ, “sigma,” to the reward gap.
In words: the bigger the reward gap, the surer we are the human picks A, and the sigmoid squashes that into a probability. The gap $r_A - r_B$ plays the role of the sigmoid's input $z$ from Chapter 4.
Worked example
Grade it by hand with Chapter 4's sigmoid landmarks. Before training the scorer cannot tell the two apart and gives both the same score, $r_A = 1$ and $r_B = 1$, so the gap is 0 and the predicted chance of preferring A is a coin flip. Its loss for disagreeing with the human who chose A is the coin-flip landmark. After training, descent has nudged the scores apart to $r_A = 2$, $r_B = 0$, a gap of 2:
$$\text{before:}\; \sigma(1-1) = \sigma(0) = 0.5,\; L = -\ln 0.5 = 0.69 \qquad \text{after:}\; \sigma(2-0) = \sigma(2) \approx 0.88,\; L = -\ln 0.88 \approx 0.13$$In words: a scorer that can't tell the answers apart carries the coin-flip loss 0.69; once training opens a gap of 2, its confidence in the human's pick rises to 0.88 and the loss falls to about 0.13, just above the $p = 0.9 \to 0.11$ landmark. The ruler and the walk downhill are the same as before, and the target is now agreement with human judgment.
| Method | Recipe |
|---|---|
| RLHF | Fit a reward model to comparisons, generate responses, and optimize for reward with constraints. |
| DPO | Use comparison pairs directly, favoring the chosen response relative to the rejected response and a reference model. |
Both use gradients, but their objectives are not identical to next-token cross-entropy. DPO's reference-relative probability ratios matter; it is more specific than blindly raising every chosen probability and lowering every rejected one.
Reward hacking can reward flattery or other superficial behaviors. A regularization term can limit drift, though it cannot make a flawed reward perfect.
Why it listens at all
The model still predicts tokens from context. Training makes useful answers more likely after assistant markers. That learned behavior does not add a separate truth checker.
A jailbreak probes weaknesses in those restrictions. Fluent mistakes likewise show that learning a helpful style is not the same as verifying each claim. Preference data rewarding accurate uncertainty can help without eliminating errors.
The relative sizes in the figure illustrate a common early training pattern. Post-training can be substantial and can change capabilities; a small dataset does not guarantee that earlier knowledge stays intact.
See it move
Why the LLM cares
Assistant behavior comes from training objectives and data applied to the same parameterized model. Pretraining, demonstrations, and preference feedback contribute different pressures rather than independent “knowledge” and “manners” modules.
We can also adapt a model to a narrower task. Updating every parameter is costly; the next chapter shows how a small structured update can reduce that burden.
What you now know
- A base model predicts continuations without a guaranteed assistant response.
- SFT learns from demonstrations in a conversation format.
- Preference learning uses comparisons between responses.
- RLHF commonly uses a reward model; DPO uses a direct preference objective.
- Post-training changes behavior and can change knowledge too.
- Helpful style and preference scores do not guarantee truth.
Where we're headed
Next we'll ask why a fluent assistant can still be confidently wrong: Why It Makes Things Up traces hallucination to the same sampling step behind every answer. After that, adapting a large model with small matrices keeps the base weights fixed while learning an added change.