Finish the sentence
“The cat sat on the ___.” We can imagine mat, couch, or keyboard, with different expectations for each. A language model also weighs possibilities before selecting one. Both steps can be made precise.
One unit of belief, spread out
A probability measures a chance. Draw blindly from ten equally likely marbles: five mint, three amber, two violet.
In words: out of every ten equally likely draws, five are mint, so the chance of mint is one half.
$$ p(\text{mint}) + p(\text{amber}) + p(\text{violet}) = 0.5 + 0.3 + 0.2 = 1 $$In words: add up the chances of every marble color and you get one whole.
A probability distribution assigns a share to every outcome. Like a function, p takes an outcome and returns its share. Each share is between zero and one; together they total one.
$$ \sum_{i=1}^{n} p(x_i) = 1 $$In words: add up the probability of every possible outcome and the total is always 1.
The sum symbol $\sum$ means “add them all up.” Here $x_i$ names outcome i and n counts the outcomes.
The shapes of belief
| Shape | Example shares | Meaning |
|---|---|---|
| Peaked | 4: 0.98; 5: 0.01; 22: 0.01. | One strong favorite for “2+2=___.” |
| Flat | Blue 0.24, green 0.22, red 0.20, purple 0.18, orange 0.16. | Several plausible favorite colors for a stranger. |
The bars report the model's confidence, not a guarantee that its favorite is correct. Both examples total one.
Belief with strings attached
Conditional probability depends on what we know. Read $p(\text{next}\mid\text{context})$ as “probability of next, given context.” The context goes to the right of the bar.
Worked example
The corpus below is made up. The numbers are invented so the arithmetic stays checkable, and they are not statistics about English. Suppose that across a shelf of storybooks the phrase "the cat sat on the ___" appears 20 times, followed by mat 12 times, couch 5 times and keyboard 3 times. The conditional probability of mat is those 12 hits out of 20:
$$ p(\text{mat} \mid \text{the cat sat on the}) = \frac{12}{20} = 0.6 $$In words: of the 20 times that phrase showed up, 12 were followed by mat, so given that context, mat carries 60% of the belief.
The other two come out the same way: $p(\text{couch} \mid \dots) = 5/20 = 0.25$ and $p(\text{keyboard} \mid \dots) = 3/20 = 0.15$. That is counting and dividing, the same as with the marbles.
In another invented sample, “the wet cat sat on the” occurs ten times: seven towels, two mats, one couch.
$$ p(\text{towel} \mid \text{the wet cat sat on the}) = \frac{7}{10} = 0.7 $$In words: once we know the cat is wet, towel jumps to 70% of the belief.
Mat falls from 0.60 to 0.20; towel rises to 0.70. We changed the question by changing the known information.
Rolling the die: from belief to choice
Sampling turns the distribution into one choice. Random does not mean equally likely.
Give each candidate its share of a die
- Assign faces 1–12 to mat: 12/20=0.60.
- Assign 13–17 to couch: 5/20=0.25.
- Assign 18–20 to keyboard: 3/20=0.15.
- Roll once. A 16 chooses couch even though mat has the largest share.
In 100 rolls we expect 60, 25, and 15 on average. The illustrative run below gives 57, 29, and 14. In ten rolls, 8, 1, and 1 is possible too.
With many independent draws, proportions tend toward the assigned probabilities. For independent zero-mean noise with finite variance, averaging also reduces typical error; ten coin flips need not give exactly five heads.
| Rule | What happens |
|---|---|
| Greedy | Choose the tallest bar, here mat. |
| Sampling | Choose using the whole distribution, allowing less likely outcomes. |
See it move
Why the LLM cares
$$ p(\text{next} \mid \text{context}) $$In words: the probability of each possible next word, given everything written so far.
The model builds a distribution over word-pieces, chooses one, appends it to the context, and repeats. Sampling explains why repeated prompts can produce different answers. Changing context explains why adding “wet” changes what comes next.
Training will score the probability assigned to the correct piece. The logarithm from Chapter 4 will turn that probability into a penalty.
What you now know
- Probabilities lie between zero and one and total one.
- Conditioning changes the distribution when known information changes.
- Sampling follows the shares; greedy selection picks a maximum.
- Small samples can differ greatly from expected counts.
- Softmax converts scores to shares; temperature reshapes them.
- A model repeats prediction, selection, and appending.
Where we're headed
Part I is complete: vectors, similarity, matrices, curves, slopes, and probability. Now we'll turn text into tokens so these tools can work on language.