One job: guess the next token
A language model predicts what comes next. That compact job description leaves a lot to explain. The context arrives as embeddings, and the model produces one score per vocabulary token.
In words: the probability of each possible next token, given all the tokens so far. That conditional bar "$\mid$" is Chapter 6's "given"; read it as "the next token, given the context."
Autoregressive generation follows this loop.
An early choice changes every later prediction. Here we take the scores as given; Chapters 10 and 11 build the part that computes them.
The box's raw scores: logits
A logit is an unrestricted score. The vector $\mathbf{z}$ collects all the logits. For hand calculation, pretend our vocabulary contains only mat, couch, keyboard.
$$z_{\text{mat}} = 2.0, \qquad z_{\text{couch}} = 1.0, \qquad z_{\text{keyboard}} = 0.0$$In words: the box rates mat highest at 2.0, couch in the middle at 1.0, and keyboard lowest at 0.0. In Chapter 6 these same three words carried 0.60 / 0.25 / 0.15 from counting storybooks. A model's learned opinion need not match that shelf of storybooks: same three words, different source, different numbers.
Negative logits are legal too. Scores need not total one, so they are not yet probabilities.
Softmax: scores become shares of belief
Softmax converts the scores. First exponentiate; then normalize by the total.
One three-candidate prediction
$$p(\text{mat}) = \frac{e^{2.0}}{e^{2.0} + e^{1.0} + e^{0.0}} = \frac{7.39}{7.39 + 2.72 + 1} = \frac{7.39}{11.11} = 0.665$$In words: exponentiate mat's score, then divide by the sum of everyone's exponentiated scores, and mat gets 66.5 percent of the belief. The other two follow the same recipe: $p(\text{couch}) = 2.72/11.11 = 0.245$ and $p(\text{keyboard}) = 1/11.11 = 0.090$. The three add to $0.665 + 0.245 + 0.090 = 1.000$, so this is a legal distribution, the kind Chapter 6 drew as bars.
In words: candidate $i$'s probability is $e$ to its own score, divided by the sum of $e$ to everyone's score. That $\sum_j$ is Chapter 6's add-them-all-up, and the $e$ is Chapter 4's exponential.
| Candidate | Score | Exponential | Share |
|---|---|---|---|
| mat | 2 | 7.39 | 0.665 |
| couch | 1 | 2.72 | 0.245 |
| keyboard | 0 | 1 | 0.090 |
A score of −1 would contribute $e^{-1}\approx0.37$, still positive. Adding the same constant to all scores leaves the shares unchanged:
A one-point gap gives a probability ratio of e, about 2.72. Raising just one score changes the shared denominator too; its normalized probability does not itself multiply by e.
Temperature: the promised knob
Temperature T rescales the score gaps before conversion:
In words: shrink or stretch all the scores by the same factor $T$, then convert to probabilities as before. Everything downstream is untouched; only the logits going in are rescaled.
Cold and hot on the same scores
At T=0.5, divide [2,1,0] by one half to get [4,2,0]. Since $e^4=(e^2)^2\approx54.6$, the exponential total is about 63.0.
$$p(\text{mat}) = \frac{e^{4.0}}{e^{4.0} + e^{2.0} + e^{0.0}} = \frac{54.6}{63.0} = 0.867 \qquad (T = 0.5)$$In words: at half temperature, mat's share climbs from 0.665 to 0.867; $p(\text{couch}) = 0.117$ and $p(\text{keyboard}) = 0.016$ get squeezed. The gaps between the scores grew, so the exponentials pulled apart and the distribution sharpened onto its favorite. A cold setting concentrates belief. It does not make the favorite correct.
At T=2, scores become [1,0.5,0]. The exponentials are about 2.72, 1.65, 1, totaling 5.37. Divide to get 0.506, 0.307, 0.186. Their displayed total is 0.999 because of rounding.
| Setting | Behavior |
|---|---|
| T below 1 | Sharpen toward the favorite. |
| T=1 | Use the original score gaps. |
| T above 1 | Flatten toward a tie. |
| T approaches zero | Concentrate on maximum-score tokens; tied maxima remain tied. |
| T grows very large | Approach equal shares across this finite vocabulary. |
Play the game
Sampling rolls the reshaped die. Greedy selection takes a maximum directly; it avoids dividing by T=0.
Other sampling filters may keep only the k highest candidates (top-k) or the smallest leading set reaching a chosen total probability (top-p). The score-to-share calculation remains the foundation.
Autocomplete: true and misleading
Compare a simple word-count predictor with a learned context model. A count table estimates what followed a short phrase; an LLM computes scores through many learned layers. The phone drawing illustrates the simpler method, not every modern keyboard.
Predicting across many kinds of text pressures the model to learn grammar, facts, style and useful reasoning patterns. A narrow training objective can require rich internal structure.
A hallucination exposes the gap between a plausible continuation and a verified fact.
See it move
Why the LLM cares
We have described what goes in and what comes out: context in, scores out, probabilities, then a choice. Attention and the transformer explain how the scores depend on context.
Training will evaluate the probability of the correct token: a correct answer assigned p=0.1 incurs about 2.3 units of loss. Part III shows how that score changes the parameters $\theta$, “theta.”
What you now know
- Generation repeatedly predicts, chooses, and appends.
- Logits are scores; softmax converts them into probabilities.
- Only score gaps affect softmax.
- Temperature changes those gaps before sampling.
- Predicting likely text requires learned structure but does not guarantee truth.
Where we're headed
How does “wet” affect a later prediction? Next we'll build attention, using dot products to decide which token information to mix.