Inference, Efficiently

The pause and the stream

Paste three pages into a chatbot and hit send. First there is a pause — a second or two of nothing, a held breath. Then the reply arrives at a steady clip: word, then word, then word, marching out at a pace you can almost read along with. Two completely different speeds, and you get them every single time. By the end of this chapter you'll know exactly what happens in that pause, why the streaming afterward is so steady, and why neither one is an accident.

Everything so far has been about how the machine is built (Part II) and how it learns (Part III). Today is about running it — inference. During inference nothing learns and no weight moves so much as a hair; the entire game is doing one fixed computation fast and cheap. There is no new math in this chapter — it is engineering, built entirely out of tools you already own: the generation loop, the dot product, softmax, the causal mask.

Here is the itinerary in one sentence: what a single token costs to produce, the cache that refuses to compute anything twice, what a context window actually is (and where its teeth are), and how a weight like 0.34 survives being stored as "shelf 3" in four bits.

What one token costs

Recall the loop from Chapter 9: generation is sample → append → run again. Every generated token means one full trip through the tower — embeddings in, all $L$ layers of attention-and-MLP, unembedding out. And here is the sentence that should sting a little: for every one of those trips, essentially every learned number in the model gets read and used. Billions of weights, touched per token. That is the baseline bill, and it is exactly why "tokens per second" is the universal speed metric for serving an LLM — the whole machine, once, per word.

Now let's find the waste. Walk the site's running sentence one step. The model has processed "the cat sat" and predicted "on" — this is Chapter 9's very sentence, caught mid-flight. We append "on." Naively, we now re-run the whole stack on all four tokens, recomputing everything for "the," "cat," and "sat" from scratch. But look at what the causal mask (Chapter 10) guarantees: "the," "cat," and "sat" were never allowed to peek forward, so nothing about the new arrival can change any of their computed vectors. Their queries, their keys, their values — everything, in every layer — come out bit-for-bit identical to last time.

Say the conclusion plainly, because a whole industry rests on it: at each generation step, the only genuinely new work is the newest token's. Everything computed for earlier tokens is still valid, sitting there unchanged. Recomputing it is pure, provable waste — and the fix is the most-used piece of engineering in all of LLM serving.

The KV cache: never compute twice

The fix is one sentence long, and then we'll work it by hand. Keep a running store of every past token's key vector and value vector, in every layer — the KV cache — so a new token computes only its own three role-vectors and reads everyone else's from the store. Why keys and values but not queries? A token's query is used once, the moment it asks its question; its key and value are what every future token will need to score it and mix it in. Past questions are spent. Past answers stay on file.

Let's work one full step with Chapter 10's exact cast. After "the cat sat," the cache holds six little vectors (one toy head, one layer): the keys $\mathbf{k}_{\text{the}} = [1, -1]$, $\mathbf{k}_{\text{cat}} = [1, 1]$, $\mathbf{k}_{\text{sat}} = [0, 1]$, and the values $\mathbf{v}_{\text{the}} = [0, 2]$, $\mathbf{v}_{\text{cat}} = [3, 1]$, $\mathbf{v}_{\text{sat}} = [1, 0]$ — Chapter 10's numbers, unchanged. Now "on" arrives with its own freshly computed trio, invented here for this chapter's toy (same pretend mini-model as Chapter 10): $\mathbf{q}_{\text{on}} = [2, 1]$, $\mathbf{k}_{\text{on}} = [1, 0]$, $\mathbf{v}_{\text{on}} = [0, 1]$. The only attention arithmetic needed this step is on's query against the four keys on file. Here is the loudest one worked in full:

In words: line on's question up against cat's key, multiply matching slots, add — a score of 3.

Now soften the scores into shares with Chapter 9's softmax — unscaled, the same convention as Chapter 10's spine. Exponentiate each score, then divide by the total. Every exponential comes from Chapter 4's table or its multiply rule ($e^{3} = e^{1} \cdot e^{2} = 2.72 \times 7.39$): $e^{1} \approx 2.72$, $e^{3} \approx 20.09$, $e^{1} \approx 2.72$, $e^{2} \approx 7.39$, totaling $\approx 32.92$:

In words: turn the four scores into four shares of on's attention. We reuse Chapter 10's symbol $a_i$ for an attention share — "a," not $w$, because these are computed on the fly, not learned. So "on" leans hard on "cat" (0.61), gives a fifth to itself (0.22), and spares a little for "the" and "sat" (0.08 each). One honest note, said once: rounded to two decimals these shares read $0.08 + 0.61 + 0.08 + 0.22 = 0.99$ — the familiar two-decimal artifact from Chapter 10, not a bug.

Count what the cache just saved. Without it, this step recomputes the whole $4 \times 4$ score grid — and re-derives every query, key, and value behind it, in every layer. With it, attention does one new row: on's query against four keys. The three earlier rows were never touched today.

At toy scale the saving looks modest. At real scale it is brutal: at position 1,000, the cache turns roughly 1,000 rows of redone work into 1.

And now we can cash in the hook. The pause before the first word is prefill: your entire prompt sweeps through the stack in one parallel pass — all positions at once, the transformer's native talent (Chapter 11) — filing keys and values for every prompt token into the cache. The stream afterward is generation: one token, one new row, against a full cache — steady, predictable work per token. Two phases, two speeds.

What a context window really is

Define it with the tools already on the table. The context window is the largest $n$ — $n$, the number of tokens in context, as always — that the machine can hold and attend over at once. It is a workspace, not a memory and not knowledge. The model's knowledge lives in its frozen weights (Chapter 11); the window holds the tokens of the current job, plus their cached keys and values. A bigger window is a bigger desk, not a smarter worker.

Why is it finite? Two honest reasons. Reason one — training territory: the model only ever practiced on sequences up to some length, and its seat stamps (the positional information from Chapter 11) beyond that range describe terrain training never visited; models behave unreliably off the map. Reason two — cost, and here your own numbers bite. Attention compares every asker with every earlier token, so the prefill score work grows with the square of the length. Chapter 10 planted this seed; let's work it:

In words: ten times the context means a hundred times the score grid — per layer, per head. That is the quadratic tooth.

The cache is gentler, but still relentless: it grows linearly with $n$, because every token adds its one key and one value, in every layer. One real-scale computation, every factor from Chapter 11's public GPT-3 numbers (embedding width 12,288; 96 layers): $2 \text{ vectors} \times 12{,}288 \text{ numbers} \times 96 \text{ layers} \approx 2.4$ million numbers per token — about 4.7 MB at 2 bytes per number. A full 2,048-token context is therefore roughly 10 GB of cache, before you store the model itself. When you hear that long-context serving is a memory problem, this is the memory.

What happens when a conversation outgrows the window? A practical truth, no new mechanism: the app — not the model — must drop or summarize older turns before re-sending them. The model has no idea anything is missing, because for the model there is no "anything" outside its context. Real windows have grown from 2,048 tokens in the GPT-3 era to hundreds of thousands today — hard engineering on every front of this chapter — but every window still ends somewhere, and the quadratic teeth still bite.

Quantization: shrinking the numbers

Third pressure point: the weights themselves. Computers store each number with a fixed budget of on/off switches — bits — and the budget sets how fine a ruler you get. 32 bits distinguishes billions of values; 4 bits distinguishes exactly 16. Standard training uses 32- or 16-bit weights, so a 7-billion-weight model is 28 or 14 GB before it computes a single thing. The size problem is not the count of weights — it is the luxury of each one.

Quantization is the decision that weights don't need luxury. Here is the chapter's worked spine. Take four real weights from one row of a grid: $[0.34,\, -0.12,\, 0.7,\, 0.23]$. Now build the 4-bit ruler, and notice where its one magic number comes from. Four bits give us sixteen patterns to spend. Spend them as the whole numbers $-7$ through $+7$ — that is fifteen shelves, sign included, with one pattern left over — and the biggest shelf number available to us is 7. So we size the ruler by finding the biggest magnitude among our weights (0.7) and dividing it by that 7, which gives the shelf spacing — the scale. Shelves land at $0, \pm 0.1, \pm 0.2, \dots, \pm 0.7$, and the largest weight sits exactly on the top shelf, which is the point: stretch the ruler to fit the widest number we actually have:

In words: store each weight as its nearest shelf number — a small whole number that fits in 4 bits — and to use it later, multiply the shelf number back by the scale. So 0.34 rounds to shelf 3, and rebuilding gives $3 \times 0.1 = 0.30$, a little shy of the original.

Why doesn't the model fall apart? Test it on the operation that actually matters — Chapter 2's dot product, the machine's heartbeat. Dot both versions of our row against $\mathbf{x} = [1, 1, 1, 1]$: the true weights give $0.34 - 0.12 + 0.7 + 0.23 = 1.15$; the shelf weights give $0.3 - 0.1 + 0.7 + 0.2 = 1.10$. Off by about 4% — and this is a four-number sum. Real sums run over thousands of weights whose rounding errors point up and down at random, and many small random errors in a big sum behave like Chapter 6's coin flips: some land high, some land low, and the total hugs the middle rather than running away. The model never needed the third decimal of any single weight; it needs the big blended sums to come out right — and they do.

One more honest reason it works: training itself was noisy — every batch of examples jiggled the weights a little (Chapter 15) — so the final function settles into a comfortably wide valley in the loss landscape (Chapter 13's language), not balanced on a knife's edge. In a wide valley, small nudges to the weights barely move the outputs. Quantization is one more small nudge.

The fine print, one beat, honestly. Real schemes don't share a single ruler across a whole grid — they store one full-precision scale per small group of weights (each block of, say, 32 or 64 gets its own ruler), which keeps the shelves snug everywhere. And quantization is not free: push down to 4 bits and quality dips a little; push to 2 and models get noticeably worse. The sweet spot in the wild sits around 4 to 8 bits. Then the payoff, which is pure arithmetic — bits times count:

In words: four bits instead of thirty-two is an eight-times smaller model — the difference between a server rack and the phone in your pocket.

Why the LLM cares

This chapter is the decoder ring for every model card and pricing page you will ever read. Practice on a real-looking spec string: "7B-Q4, 8K context." You can now read every piece of it — 7 billion parameters (Chapter 11's counting), quantized to 4 bits (~3.5 GB, small enough to run locally), and a context window of 8,192 tokens (its whole workspace, cache costs and all). Nothing in the string is mysterious anymore.

Now decode the everyday experiences, one clause each. The pause before the first word is prefill filling the cache. The steady streaming is one new row per token. Long conversations slowing down and costing more is the row lengthening and the cache swelling with $n$. Input tokens priced cheaper than output tokens is typical practice with an honest mechanism: prefill digests your whole prompt in one parallel pass, while output tokens are minted one serial trip at a time. And "this model runs on a phone" — quantization did that.

One breadth beat, names only, no mechanics. Serving stacks pile on further tricks in the exact same spirit as today. Batching pushes many users' requests through the machine at once (it is why shared APIs are cheap). Speculative decoding has a small, fast model draft several tokens and lets the big model check the whole draft in one pass. Every one of them obeys the same rule: never do expensive work you can avoid, skip, or share.

Connect it backward and forward. The KV cache is Chapter 10's grid made affordable; quantization is Chapter 2's dot products proving robust; the context window is the honest boundary around it all. And that boundary sets up a real problem: your documents, your company wiki, your lab notebooks will never fit in any window. The fix is not a bigger desk — it is a library card. That's next.