From Parrot to Partner

The parrot in the room

The last chapter left us holding the base model — fourteen gigabytes of dials that have read more than any human ever will. So we ask it the simplest thing: "What is the capital of France?" And it answers: "What is the capital of Germany? What is the capital of Spain?" Not because it doesn't know Paris — it knows Paris cold — but because a page of quiz questions is a genre it has read ten thousand times, and continuing your question with more questions is a perfectly likely thing for that text to do.

Here is the gap, stated precisely, because closing it is the whole chapter. The base model optimizes for one thing only: being likely — Chapter 9's $p(\text{next} \mid \text{context})$. Nowhere in trillions of tokens of descent did anything push it toward being helpful. Likely and helpful overlap sometimes and diverge constantly. Every model you have ever chatted with had this gap closed after pretraining, by a second, much smaller round of shaping. It has a name: post-training (also called alignment, or fine-tuning).

Now the reassuring part, and it is true: not one new piece of mathematics arrives in this chapter. Post-training is Chapter 12's ruler ($-\ln p$) and Chapter 13's downhill walk — the exact machinery that made the base model — aimed at new targets. Same fourteen-gigabyte file, nudged. The journey from parrot to partner is three acts, all played on one instrument: show it examples, teach it taste, and understand why any of it makes the thing listen.

Show, don't tell

Act one is the cheapest fix, and the oldest teaching trick: show it what you want. Collect a modest pile of example conversations — a human writes a good instruction and a good response — and keep training the base model on them with the same next-token loss from Chapter 12. Nothing about the descent changes; only the reading material does. This is supervised fine-tuning (SFT), also called instruction tuning.

One genuinely new mechanical detail, and it deserves its own moment. The examples are wrapped in a fixed format so the model can tell who is speaking — special marker tokens for "the user's turn" and "the assistant's turn," a chat template. And the loss is computed on only the assistant's tokens — the model is graded on how well it produces the response, never on reproducing the user's prompt. It learns to continue the conversation at exactly the moment the assistant marker appears.

Notice what did not happen: nothing about French was memorized. The model already knew "bonjour" from pretraining — SFT only made "answer helpfully here" the likeliest continuation of the assistant marker. That is the whole trick. SFT teaches format and behavior — the shape of a reply, when to start, when to stop, the register of a helpful voice — by surfacing knowledge the base model already had. And it is astonishingly cheap: tens of thousands to a few hundred thousand curated conversations, against Chapter 15's trillions of pretraining tokens. In the model's whole life of reading, SFT is an afternoon of etiquette lessons. That an afternoon can do so much is the tell that the ability was always there, waiting for a cue.

But SFT can only teach the model to imitate the demonstrations a human bothered to write — and for most of what we want from an assistant, nobody can write the perfect demonstration, and it is far easier to say which of two answers is better than to compose the ideal one from scratch. That crack is what act two must fill.

The taste problem

Ask the freshly SFT'd model "How do I make a good cup of coffee?" and it gives you two fluent, well-formatted answers. One is warm, specific, and correct. The other is a confident wall of plausible-sounding nonsense that happens to be wrong about water temperature. SFT taught the model to write like an assistant — it did not teach it that the first answer is better. Both are exactly the kind of text a helpful assistant produces.

And that is the deeper problem: goodness lives in comparisons that no demonstration captures. Which answer is more honest? Which refusal is appropriately firm without being preachy? Which explanation is pitched at the right level? You cannot write these preferences down as facts — but a human can reliably point at the better of two responses in a second. The cheap, abundant signal is not "here is the perfect answer." It is "this one, not that one."

So what we need is different from act one. Not more imitation of good answers, but a way to push the whole model toward being preferred among the answers it could give. That is a different target than "be likely," and it needs a different kind of data — comparisons, not demonstrations. Enter the second act.

Learning from comparisons

Act two, and it is built from pieces you already own. Collect pairs: the same prompt, two responses the model produced, and a human marks one as better — the chosen response and the rejected one. Millions of these little verdicts, cheap to collect because comparing is easy. This is preference learning, the umbrella over the acronyms.

To grade a preference we turn it into a number — and it is a familiar number. Give each response a score, a reward $r$ (an italic lowercase scalar, a plain rating). If the chosen answer scores $r_A$ and the rejected one scores $r_B$, then the chance a human prefers A over B should rise smoothly with the gap $r_A - r_B$ — and "a number squashed into a 0-to-1 chance" is exactly Chapter 4's sigmoid:

In words: the bigger the reward gap, the surer we are the human picks A — squashed to a probability by the sigmoid. The gap $r_A - r_B$ is playing the role of the sigmoid's input $z$ from Chapter 4.

There are two roads from that scorer to a better assistant, and both deserve a name. Road one is RLHF: the trained scorer becomes a reward model, an automatic examiner. Let the assistant write answers, have the reward model grade each one, and nudge the dials to make high-scoring answers more likely — Chapter 13's descent again, with a machine handing out the grades instead of a fixed answer key. Road two is DPO: a 2023-era rearrangement showed you can skip the separate scorer and the generate-and-grade loop entirely — push up the probability the assistant assigns to the chosen response and push down the rejected one, directly, with a loss built from the very same $-\ln$ and $\sigma$. Same destination, far less plumbing, which is why much of the field moved toward it. The acronyms differ; the target — be preferred, not merely likely — is identical.

One guardrail, because it explains a real failure you would otherwise not predict: preference learning is left off its leash at its peril. Reward the assistant purely for high scores and it discovers that opening every reply with "What a fantastic question!" reliably pleases graders — it games the examiner instead of getting better. So the tuned model is kept tethered to its SFT starting point, allowed to change only so far before a penalty pulls it back — a leash whose length is a dial the engineers set.

Why it listens at all

Here is the payoff the title has been promising. The assistant never stopped being an autocomplete. It is still $p(\text{next} \mid \text{context})$, the exact machine from Chapter 9 — not one line of that changed. What post-training changed is which continuation is most likely. SFT made "a helpful reply" the probable text after the assistant marker; preference learning sharpened its taste among the fluent options. Helpfulness did not get bolted on as a new faculty — it became the most probable thing to say while wearing the assistant costume.

Take that costume literally, because it demystifies so much. The special turn-marker tokens are the costume, and inside it the base model plays the role its post-training made likeliest: the helpful assistant. This one idea explains phenomena you have surely met. A jailbreak is nothing mystical — it is a context engineered to make some other continuation (the base model's unfiltered autocomplete) likelier than the aligned one. You are not hacking a mind; you are finding a costume the training didn't cover.

One honest beat, no over-claiming. SFT teaches the model to sound like a confident, knowledgeable assistant — and sounding confident is a behavior, not a fact-check. So a model can produce a fluent, well-mannered answer that is wrong, because a fluent, well-mannered answer is exactly what the costume calls for. This is the honest core of what people call hallucination — Chapter 9's "it bets, it doesn't look up," seen now from the training side. Preference learning fights back by rewarding honesty — "I don't know" scored above confident nonsense — which helps and does not cure.

So assemble the reveal. Instruction-following, the willingness to answer, the tone, the refusals, the "I'm an AI assistant" self-description — none of these are separate modules. They are all the same fourteen-gigabyte autocomplete, its dials nudged by $-\ln p$ and gradient descent until the likeliest text in the assistant costume happens to be helpful, honest-ish, and polite. The partner is the parrot, after etiquette school.

Why the LLM cares

Zoom all the way out to the model's whole biography, now tellable in one breath because you built every organ. Pretraining (Chapter 15) spent trillions of tokens and nearly all the money to make a base model that knows almost everything and prefers nothing but likelihood. Supervised fine-tuning spent an afternoon of curated conversations to teach it the shape of a helpful reply. Preference learning spent millions of human comparisons to give it taste. Three phases, wildly different in scale, playing the same instrument: $L = -\ln p$ and a downhill step of size $\eta$ (eta, the step size).

Look at that scale gap and feel it: post-training is a rounding error in tokens against pretraining, which is exactly why the base model's knowledge survives it intact. An afternoon of manners cannot overwrite a lifetime of reading. And this is the last mechanism the site owed you. You can now account for the entire object you talk to. The knowledge is pretraining's. The helpfulness is SFT's. The judgment and manners are preference learning's. The occasional confident wrongness is the costume's. And underneath all of it, unchanged since Chapter 9, sits one autocomplete guessing the next token. There is no further secret ingredient — you have seen the whole machine.

Everything so far has been about one assistant, shaped once by its makers. But the same trick that made it — keep training on chosen examples — is available to anyone. Want a model that writes in your voice, or tutors radar systems the way you'd teach it? Fine-tune it on your own examples. The catch is the price tag, and it is steep: doing it the full way means making and retraining a fresh copy of all fourteen gigabytes, per project. Part IV opens by making that change almost free.