The Prediction Game

One job: guess the next token

You have heard the dismissal — "it's just fancy autocomplete." Here is the uncomfortable truth this chapter delivers: that is accurate. And it is also one of the most misleading true sentences in all of technology. By the end you will be able to defend both halves — the "true" and the "wildly misleading" — with a straight face.

Start with the job, stated using everything you now own. Given the context — the tokens so far (Chapter 7), each riding in as its arrow (Chapter 8) — the model produces a probability distribution over every token in its vocabulary: one bar per token, all $V$ of them, roughly fifty thousand bars, rebuilt from scratch. Then one token is sampled — Chapter 6's weighted die — appended to the context, and the whole thing runs again. The entire job fits in one expression:

In words: the probability of each possible next token, given all the tokens so far. That conditional bar "$\mid$" is Chapter 6's "given" — read it as "the next token, given the context."

This generate-append-repeat cycle has a name: autoregressive generation — a fancy word for "the output is fed back in." And that feedback has a consequence worth pausing on: one adventurous pick early in a sentence changes the context for every pick after it. That is why two runs of the same prompt can drift into entirely different paragraphs — a snowball, not a script. There is no plan; there is only the next token, over and over, each one reshaping the ground for the one after it.

One honest promise about scope before we open the toolbox. Today we take the model's scores as given — a box that somehow rates all fifty thousand candidates. How the box computes good scores is the story of Chapters 10 and 11. Today we nail down what the scores are and how they become choices: raw scores, then probabilities, then one sampled token.

The box's raw scores: logits

Here is what actually falls out of the box's far end — and it is not probabilities. For each of the $V$ tokens the box reports one raw score: any number at all, positive or negative, no rules. The name for one of these scores is a logit (say it "LOH-jit"). A big logit means the box likes that candidate; the absolute numbers mean nothing on their own — only how they compare.

Let's set up the worked example we will ride for the whole chapter, and let's be honest about the cheat inside it. The context is Chapter 6's old friend, "the cat sat on the ___", and we will watch just three candidates — mat, couch, keyboard — pretending, for arithmetic's sake, that these three are the whole vocabulary. Everything on this page scales to fifty thousand candidates without changing a single step; we shrink $V$ to $3$ only so the sums fit on one line. The box reports three logits — call the letter $z$, the traditional symbol for a logit, with the bold vector $\mathbf{z}$ collecting all of them:

In words: the box rates mat highest at 2.0, couch in the middle at 1.0, and keyboard lowest at 0.0. (One quiet note for the attentive: back in Chapter 6 these same three words carried 0.60 / 0.25 / 0.15 from counting storybooks. The model's learned opinion needn't match our little shelf of storybooks — different source, different numbers, same three words.)

Why won't the raw scores do as they stand? Because probabilities, in Chapter 6's language, must obey two house rules: each must sit between 0 and 1, and together they must share exactly one unit of belief. Logits break both — 2.0 is no probability, and a score of $-3$ would be nonsense as one. Notice that last point: negative scores are perfectly legal logits, and any converter we build has to swallow them without complaint. So we need a machine that turns any list of scores into an honest distribution — and does it without scrambling the ranking the box worked so hard to compute.

We already built the tool for step one. Chapter 4 handed us a function that turns any number, positive or negative, into a positive one, with no ties at zero. Time to cash that check.

Softmax: scores become shares of belief

The recipe has two moves. Move one — exponentiate. Run every logit through $e^z$, the exponential of Chapter 4. From that chapter's table, reused not rederived: $e^{2.0} \approx 7.39$, $e^{1.0} \approx 2.72$, and $e^{0.0} = 1$. All positive, order intact, no ties at zero. (A logit of $-1$ would have contributed $e^{-1} \approx 0.37$ — a smaller share, still positive, still legal — which is exactly why negatives cause the machine no trouble.) Move two — normalize. Divide each by the total, $7.39 + 2.72 + 1 = 11.11$, so the three shares sum to exactly 1. That is Chapter 6's conservation of belief, engineered on purpose.

Here is the winner worked in full:

In words: exponentiate mat's score, then divide by the sum of everyone's exponentiated scores — mat gets 66.5 percent of the belief. The other two follow the same recipe: $p(\text{couch}) = 2.72/11.11 = 0.245$ and $p(\text{keyboard}) = 1/11.11 = 0.090$. Check the conservation law: $0.665 + 0.245 + 0.090 = 1.000$. A real, legal distribution — the bars of Chapter 6.

That machine has a name: softmax. In general form, for candidate $i$ out of a whole vector of logits $\mathbf{z}$:

In words: candidate $i$'s probability is $e$ to its own score, divided by the sum of $e$ to everyone's score. That $\sum_j$ is Chapter 6's add-them-all-up, and the $e$ is Chapter 4's exponential — two old tools meeting inside one new machine.

Two properties make softmax feel inevitable, and both survive a hand-check. First: only the gaps between logits matter. Shift every score up by 1 — scores $[3, 2, 1]$ — and the distribution comes out identical.

Second: each $+1$ of logit multiplies a candidate's share of belief by $e \approx 2.72$. Mat sits exactly one logit above couch, and indeed $0.665 \div 0.245 \approx 2.71$ — that is $e$, give or take a rounding crumb. Hold onto that fact; it is where most people's intuition goes wrong.

Notice, finally, the shapes of Chapter 6 hiding in here: softmax output can be peaked or flat depending on how spread out the logits are — big gaps make spikes, small gaps make plains. Which raises an irresistible question: could we reshape that spread ourselves? Yes. That is the next section.

Temperature: the promised knob

Back in Chapter 6 we wrote an IOU: "a knob that reshapes peaked toward flat and back." Here it is, and it is one line of arithmetic. Before softmax, divide every logit by a number $T$ — the temperature:

In words: shrink or stretch all the scores by the same factor $T$, then convert to probabilities exactly as before. Everything downstream is untouched; only the logits going in are rescaled.

Work the cold case, $T = 0.5$, in full. Dividing by 0.5 doubles every logit, so $\mathbf{z}/T = [4.0, 2.0, 0.0]$. Now $e^{4.0} \approx 54.6$ — hand-checkable by Chapter 4's rule, $e^4 = e^2 \cdot e^2 = 7.39 \times 7.39$ — while $e^{2.0} \approx 7.39$ and $e^{0} = 1$, for a total of $63.0$:

In words: at half temperature, mat's share climbs from 0.665 to 0.867; $p(\text{couch}) = 0.117$ and $p(\text{keyboard}) = 0.016$ get squeezed. The gaps between the scores grew, so the exponentials pulled apart and the distribution sharpened onto its favorite. Cold means confident.

Now the hot case, $T = 2$. Dividing by 2 halves every logit: $\mathbf{z}/T = [1.0, 0.5, 0.0]$. The $e$-values are $e^{1} \approx 2.72$, then $e^{0.5} \approx 1.65$ (the number whose square is $e$ — check: $1.65 \times 1.65 \approx 2.72$), and $e^{0} = 1$, summing to $5.37$. The shares come out $p(\text{mat}) \approx 0.506$, $p(\text{couch}) \approx 0.307$, $p(\text{keyboard}) \approx 0.186$. (They add to 0.999 — the missing 0.001 is rounding, not leaked belief.) The gaps shrank, the exponentials huddled together, and the distribution flattened. Hot means adventurous — keyboard just doubled its odds.

Push the dial to its ends, stated as limits in plain language. As $T$ shrinks toward 0, the tallest bar takes everything — that is Chapter 6's "greedy," the same word, now with a mechanism behind it. As $T$ grows huge, every candidate drifts toward an equal $1/3$ (or, in the real model, $1/50{,}000$) — pure uniform chance, the model's opinion washed out entirely. Real systems ship with $T$ around 0.7–1.0 as typical practice; coders drop it low for predictable answers, writers raise it for surprise.

Play the game

Enough arithmetic on paper — here are the chapter's exact numbers, live and under your thumb. At $T = 1$ the three bars for "the cat sat on the ___" stand at 0.665 / 0.245 / 0.090, just as we computed. Drag the temperature slider and watch them reshape; press Sample to roll Chapter 6's weighted die and append the winner to the history strip.

One sentence on what you are holding: this widget is the entire output end of a real LLM. Production systems bolt refinements onto the die roll — you will hear the names top-k and top-p — but the spine is exactly this: logits, then $\mathrm{softmax}(\mathbf{z}/T)$, then a sample.

Autocomplete — true, and wildly misleading

Grant the true half first, and generously. Mechanically, an LLM does what your phone keyboard does: it proposes the next piece of text. Your phone does it by counting — which word usually followed the last one or two you typed? That is literally Chapter 6's storybook-shelf arithmetic (12 mats out of 20), running on your own texting history. Small context, simple counts, an honest little machine. So far the dismissal holds.

Now the misleading half — and the whole thesis of this site in one contrast. The LLM is not counting recent pairs. It computes its fifty thousand scores fresh every single time, from the entire context, through one enormous composed function (Chapter 4's machines-feeding-machines) whose billions of dials $\theta$ — theta, the collective name for every tunable number — were tuned on a large fraction of everything humans have written. One word placed early — Chapter 6's "wet" — lawfully reshapes the scores at the far end of the paragraph. Your phone cannot do that. The difference is not one of degree but of kind.

Here is the deeper point, stated carefully. To autocomplete well — at every position, across law briefs and lullabies and Python — the scores must reflect grammar, facts, style, and the logic-shaped regularities of how ideas follow ideas. The training pressure is humble ("guess the next token"); what it takes to satisfy that pressure is not. The job description is autocomplete; the qualifications are most of human knowledge. That gap between a humble objective and the machinery it demands is the single most important idea on this page.

Why the LLM cares

This chapter is not one gear of the machine — it is the machine's contract with the world, and the rest of the site fills in its two blanks. Blank one: how does the box turn "the wet cat sat on the" into logits that put towel on top? That machinery is attention and the transformer, Chapters 10 and 11 — and you can already see the puzzle from here: "wet" sits five tokens back, yet it must reach forward and bend the scores at the blank.

Blank two: where do those billions of dial-settings $\theta$ come from? That is all of Part III. One seed, planted exactly as Chapter 4 promised: training grades the model by looking at the probability it gave the correct next token — and if the model gave the right answer only $p = 0.1$, the natural logarithm of Chapter 4 converts that to a penalty of 2.3. (You were told back then that this number "returns wearing a suit"; here it is trying the suit on. The full tailoring is Chapter 12 — no formula today.)

And every knob you have met in a real product now has a mechanism behind it. The "temperature" slider in an API is this chapter's $T$; "deterministic mode" is greedy; answers changing between runs is sampling; and pricing per token (Chapter 7) is charged per turn of this loop — one full fifty-thousand-score computation per generated token, which is exactly why long answers cost real money. (Chapter 18 makes that loop fast; for now, just the name.)

Let's close Part II-so-far warmly. Text becomes IDs (Chapter 7); IDs become arrows (Chapter 8); and the machine's one move is to turn today's arrows into tomorrow's token (this chapter). You can now state, without hand-waving, what an LLM is: a function from a sequence of token IDs to a probability distribution over the next ID, sampled with a temperature knob, in a loop. Two chapters remain in Part II to open the box that makes those scores worth trusting.