Chance and Choice

Finish the sentence

Read this and let your mind fill the gap: "The cat sat on the ___." Notice what your brain just did — it did not hand you one word. It weighed several at once: mat, strongly (the nursery-rhyme groove runs deep); couch, plausibly; keyboard, if the cat in question is anything like a real cat. You held all three in the air at different strengths, at the same time, and only then leaned toward one.

That move — weighing many possibilities at once, then committing to a single one — is the last tool of Part I, and it happens to be the LLM's native language. This chapter makes "weighing" precise (that is a probability distribution), shows how the weights shift the instant new information arrives (conditional probability), and shows how a committee of maybes becomes one spoken word (sampling). Three ideas, all of which you already use every time you finish someone else's sentence.

One unit of belief, spread out

Start with something you can hold. A bag holds 10 marbles — 5 mint-green, 3 amber, and 2 violet. You reach in without looking and draw one. What are the chances it comes out mint? Five of the ten marbles are mint, so the probability of mint is five out of ten:

In words: out of every ten equally likely draws, five are mint, so the chance of mint is one half.

The same counting handles the rest: $p(\text{amber}) = 3/10 = 0.3$ and $p(\text{violet}) = 2/10 = 0.2$. Counting the favorable marbles and dividing by the total is the probability; nothing more mysterious than a fraction is happening yet.

Now the two house rules of probability, and they are the whole grammar of this chapter. First, every probability sits between 0 and 1 — you cannot be more certain than certain, or less certain than impossible. Second, across all the possible outcomes the values add to exactly 1:

In words: add up the chances of every marble color and you get exactly one whole.

Read that second rule as a conservation law, the kind every engineer respects on instinct: you own exactly one unit of belief, and all you may ever do is divide it among the outcomes. You cannot mint more belief to make a favorite likelier without taking that same amount from somewhere else. The books must balance.

The full list of outcomes, each with its share of belief, has a name: a probability distribution. Its natural portrait is the bar chart above — one bar per outcome, the heights summing to 1. And here is a quiet payoff from Chapter 4: a distribution is a function in the machine sense. Feed it an outcome, it returns that outcome's share of belief. "$p$ of mint" is exactly a machine taking mint in and handing 0.5 back.

We can write the balancing rule in general, and it is friendlier than it looks. Label the outcomes $x_1, x_2, \dots, x_n$. Then:

In words: add up the probability of every possible outcome and you always get exactly 1.

That leaning capital sigma, $\sum$, is new on the page but old in spirit — it means nothing more than "add them all up," met here in its most natural habitat. Every time you see $\sum$ for the rest of the site, translate it to those four words and it loses all its menace.

The shapes of belief

A distribution answers two questions in one breath: what we expect, and how sure we are. The heights tell you the first. The overall shape of the bar chart tells you the second — and once you can read shape, you can read a distribution's confidence at a glance. Two prompts mark the ends of the spectrum.

The peaked end: "2 + 2 = ___". Almost all the belief piles onto a single bar — $p(4) = 0.98$ — with only crumbs left over for slips of the pen, like $p(5) = 0.01$ and $p(22) = 0.01$. A tall spike flanked by dust. The question all but answers itself, and the distribution shows it.

The flat end: "My favorite color is ___," asked about a stranger you know nothing about. Now belief has nowhere obvious to pile up, so it spreads: blue 0.24, green 0.22, red 0.20, purple 0.18, orange 0.16 (add them: $0.24 + 0.22 + 0.20 + 0.18 + 0.16 = 1.00$, the books still balance). No bar dominates. This is the honest portrait of genuine uncertainty — not a failure to decide, but an accurate report that many answers are equally alive.

So take the two words as tools you now own: a peaked distribution is confident, a flat one is unsure, and most real distributions live somewhere between. When an LLM writes, its next-word distribution is sometimes a spike ("The capital of France is ___") and sometimes a plain ("My favorite color is ___") — and a few chapters from now you will get a knob that deliberately reshapes peaked toward flat and back. For now it is enough to see that shape carries meaning.

Belief with strings attached

The probability of rain on some random day is one number. The probability of rain given that you can already see dark clouds stacking up on the horizon is a different, larger number. Same sky, two answers — because probabilities are not carved into the world. They are indexed to what you happen to know.

Notation makes this exact with one small symbol: the vertical bar. We write $p(\text{next} \mid \text{context})$ and read it "the probability of the next word, given the context." Everything to the right of the bar is information already in hand; everything to the left is what we are still weighing. Wrap it up as conditional probability — a tiny bar carrying the entire chapter.

Now change the context by a single word and watch the whole picture reorganize. On the same shelf, the phrase "the wet cat sat on the ___" appears 10 times — followed by towel 7 times, mat 2 times, and couch once (again, invented for teaching). The leading candidate is now a word that was not even on the podium a moment ago:

In words: once we know the cat is wet, towel jumps to 70% of the belief.

One added word — wet — and the distribution is unrecognizable. Mat crashed from 0.60 to 0.20. Towel, which had 0% before, now owns 70%. The bar in $p(\text{next} \mid \text{context})$ is not decoration; it is the entire reason a model can respond to wet at all.

So do not think of conditioning as a correction bolted onto probability. Conditioning is probability doing its actual job. Belief that refuses to move when new information arrives is not rigor — it is stubbornness.

Rolling the die: from belief to choice

The committee has to speak eventually. A distribution holds three candidates in the air at once, but a sentence can only take one next word. Turning a spread-out distribution into a single concrete outcome has a name: sampling — drawing one outcome at random, with each outcome's chance set by its bar.

Make it physical, reusing the very same numbers. Our cat-distribution is 12 parts mat, 5 parts couch, 3 parts keyboard — twenty parts in all. So take a twenty-sided die (every board-gamer owns one) and hand out its faces: 1–12 mean mat, 13–17 mean couch, 18–20 mean keyboard. Roll it and say whatever it says. The corpus counts became die faces with no arithmetic at all — 12, 5, and 3 faces, straight across. An equivalent picture is a spinner wheel whose slices take up 60%, 25%, and 15% of the circle: flick it and read where it stops.

What does doing this over and over look like? Roll the die 100 times and you would expect roughly 60 / 25 / 15. One actual run of ours (invented, and I will say so) came up 57 / 29 / 14 — close, but not exact. Then a short run of only 10 rolls came up 8 / 1 / 1 — nothing like the "expected" 6 / 2.5 / 1.5. That is the law worth carrying out of this chapter: chance is lumpy up close and lawful in the long run. Small samples wobble all over the place; long runs settle down onto the bars.

There is an alternative to sampling: skip the die and always pick the tallest bar. That is called greedy selection, and it is fully deterministic — same input, same word, every single time. For "2 + 2 = ___" that is exactly what you want. For prose it is deadly: always reaching for the safest word produces text with all the flavor of oatmeal. Spreading belief and occasionally letting an underdog win is a feature, not sloppiness — it is where surprise, variety, and voice come from.

Why the LLM cares

Now the thesis of this entire site, stated plainly at last. A large language model is a machine that, given the words so far, produces a probability distribution over every word-piece it knows — tens of thousands of bars, recomputed from scratch — then samples one, appends it to the context, and does the whole thing again for the next word. That is the loop. The star of it is a single expression you can already read:

In words: the probability of each possible next word, given everything written so far.

("Word-piece" is deliberately informal here; the precise unit — the token — is the very next chapter's whole job, and the official count of them arrives there too.) Everything in this chapter is that expression at scale. The distribution is the committee of maybes. The bar is why context matters. The sample is the die roll that turns a committee into one spoken word.

This cashes out directly against your own experience. Ask an LLM the same question twice and the answers come back different — that is not a glitch, it is sampling: our d20 rolled against a fresh distribution at every word. And the conditioning bar is why one added word can swing an entire continuation, exactly the way wet swung mat into towel a few paragraphs ago. You have felt both of these a hundred times; now you know their names.

One seed planted for much later: training will need to score how much belief the model put on the right next word, and the tool that turns a probability like 0.1 into a penalty is the logarithm you met back in Chapter 4. You have now touched every ingredient of that future formula, without our having to build it yet.

And with that, Part I is complete — so here is the whole toolkit, in one warm inventory. You have arrows that carry meaning (Chapter 1), a single-number similarity score between them (Chapter 2), machines that transform them (Chapter 3), the handful of curve shapes everything is built from (Chapter 4), the downhill walk that does all the learning (Chapter 5), and belief you can spread, update, and sample (this one). Six chapters, zero hand-waving. The toolkit is complete — it is time to open the machine.

Where we're headed

That completes the toolkit — and Part I. You now own arrows that carry meaning, a one-number similarity score, matrices that transform, the four curve shapes, the downhill walk, and belief you can spread, update, and sample. Part II opens the actual machine, and it starts with an embarrassment: a computer cannot read. Not letters, not words — nothing. Before any of our mathematics can touch language, text has to be chopped into countable pieces and turned into numbers. Those pieces are called tokens, and they are stranger and more interesting than you would guess.