The Big Picture

Apple, static, apple

Here is the whole series in one picture. We start with an apple. We slowly turn it into random static — the gray snow an old television showed when it had nothing to display. And then we turn that static back into an apple. Not the apple we started with: a brand-new one, a fruit that has never existed, photographed by nobody, and yet completely believable.

Every arrow in that little diagram is a piece we will unpack over the next seven chapters. Today we build the big picture, from nothing, and by the end of it you will know exactly what question the rest of the book is answering. There is one small piece of linear algebra in the middle, and I promise it is nothing scary — it is the one idea the entire field rests on, and once you have it, everything else is machinery.

What we're actually asking for

Start with the goal, stated plainly. We want a machine that hands us a brand-new apple: not one it memorized, but an apple that never existed and still looks completely real. A machine that produces fresh examples of a thing, rather than sorting or labeling things it is shown, is called a generative model. Each thing it produces is a fresh sample, and in this book that sample is always an apple.

To learn what an apple is, it has to be shown apples. Lots of them, and — this is the part that matters far more than it sounds — all different. Red, green, golden. Big, small, spotted. Photographed from the side and from above. From all that variety it can extract what an apple actually is: the round body, the dimple at the top, the stem. If instead every training photo were the same apple on the same table in the same light, the model would have no way to tell which parts are "apple" and which parts are just "that Tuesday afternoon".

An image is a point

Now the one idea. To a computer, an image is not a picture — it is a list of numbers. A pixel in a black-and-white image is a single number for brightness, running from 0 (black) to 1 (white). Take a tiny image, two pixels by two pixels, holding these four brightnesses:

In words: a two-by-two grid of brightnesses — a near-white pixel in the top-left, a dark one in the top-right, a darker one in the bottom-left, and a light one in the bottom-right.

To hand that grid to a computer we flatten it: read the pixels in order, left to right along the top row, then left to right along the bottom row, and write them out in one line. Top-left first, so $0.9$. Then top-right, $0.2$. Then bottom-left, $0.1$. Then bottom-right, $0.8$. That gives us our clean image, which we will call $\mathbf{x}_0$ — "x-zero, the clean image", and that subscript zero will earn its keep in the next chapter:

In words: the same four brightnesses, in reading order, written as one list of four numbers. Nothing was lost and nothing was added — only the shape of the container changed.

A list written out like that has a name, and the name is a vector. That is the whole definition: an ordered list of numbers, and the order matters. Swap the last two entries and you have a different image. I will use "list" and "vector" interchangeably from here, and in a moment a third word for the same object joins them.

Here comes the leap, and it is a small one. A list of numbers is also a point. Two numbers are a point on a flat map: go $0.9$ along, $0.2$ up. Three numbers are a point in the room you are sitting in: along, up, and back. Four numbers are a point in a four-dimensional space, which nobody can picture and which behaves, arithmetically, exactly like the first two cases. That word dimension is worth pinning down, because it gets used loosely everywhere; here it means one thing only. One dimension is one number in the list, which is one independent direction you are free to move in. Nothing mysterious happens in four dimensions — you needed four numbers to say where the point is, so the space it lives in has four. The list and the point are not similar — they are the same object, written two ways. (If lists-as-arrows is new, the math toolkit in the AI/ML book builds it from scratch, and numbers that point does the same for text.)

Real images are bigger. A modest square image is 512 pixels across and 512 pixels down, and each pixel in color carries three numbers — how much red, how much green, how much blue. The reason there are three of them is the human eye: it has three kinds of color-sensing cell in it, so three numbers at every dot is enough to reproduce anything you can actually see. A screen works the same way, with three tiny lamps at every dot. Count them:

In words: the grid holds 262,144 pixels, and each one needs three numbers for its color, so the whole image is 786,432 numbers.

So one photograph of one apple is one list of 786,432 numbers — which is to say, one single point in a space with 786,432 dimensions. Mathematicians write "the space of all lists of $d$ numbers" as $\mathbb{R}^d$, so our apple is a point in $\mathbb{R}^{786{,}432}$. Do not try to visualize that. Nobody can, and nobody does. What we do instead — throughout this book, and in every diagram you will ever see of this subject — is draw two of those axes and quietly ignore the other 786,430. The picture is a cartoon; the reasoning behind it is exact.

How far apart are two images?

Once images are points, "these two pictures look alike" stops being a feeling and starts being a measurement: similar images sit near each other, different images sit far apart. And distance between points is something you already know how to compute — it is the Pythagorean theorem, in as many dimensions as you like.

Nothing about that calculation cared that we used two coordinates. With 786,432 of them you would subtract 786,432 pairs, square 786,432 differences, add them all up, and take one square root. It is the same formula, run longer. That is the reader's proof that "similar images are nearby points" is a real, computable statement and not a metaphor.

The apple neighborhood

Now the beautiful part. Scatter every real apple photograph you can find into that enormous space, one point each. They do not land at random. They clump — hard — into one connected region, because real apples share structure: round-ish outlines, smooth skin, a stem at the top, light falling from somewhere. Any two apple photos are far more alike than either is to a photo of a bicycle, and "more alike" now means "closer".

That clump is called the data distribution, and because saying that eight times a chapter is tiring, we will mostly call it the apple neighborhood. It is the same thing: the territory where apples live.

This gives us a precise definition of the job, which is the first real progress we have made. Generating a new apple means finding a fresh point inside the apple neighborhood. Land inside it and you get a believable apple. Land outside it and you get garbage — a wash of noise, a smear, something with no name. There is no third outcome, and there is nothing else to the problem.

Why a lucky guess never works

So: pick a point inside the apple neighborhood. That sounds like it should be easy. It is brutally hard, and it is worth seeing exactly why, because the reason dictates everything the rest of the book does.

The neighborhood is a tiny, intricately shaped region floating in a space of hundreds of thousands of dimensions. High-dimensional spaces are far emptier than our intuition allows, and here is a small calculation that shows it. Think of a $d$-dimensional cube and ask: what fraction of it lies in the middle half of every axis at once? Along one axis, the middle half is a fraction $1/2$. The axes are independent, so along two axes at once it is $\tfrac12 \times \tfrac12 = \tfrac14$, along three it is $\tfrac12\times\tfrac12\times\tfrac12 = \tfrac18$, and along $d$ of them:

In words: the share of the cube that is central in all $d$ directions at once is one half, multiplied by itself $d$ times.

Put a number in. At $d = 10$ that fraction is $1/2^{10}$, and $2^{10} = 1{,}024$, so about one part in a thousand. At $d = 20$ it is $1/2^{20}$, and $2^{20} = 1{,}024 \times 1{,}024 = 1{,}048{,}576$ — about one part in a million. Twenty dimensions. We have 786,432 of them, and the pattern only steepens.

Sean's picture for this: picking a good apple point by luck is a blindfolded pin drop into a stadium, hoping to hit one particular seat. And that image, honestly, flatters the problem. A big stadium holds fifty thousand seats, so a blind drop finds your seat about one time in fifty thousand — better odds than the twenty-dimensional cube we just computed, and unimaginably better than the real thing. Random guessing is not a slow method here. It is not a method at all.

A path instead of a jump

So we stop trying to jump. Here is the trick the whole field is built on, and it is the kind of idea that looks obvious only after somebody has said it.

Instead of leaping straight to a good apple, we teach a model to walk. Take a real apple — a point sitting comfortably inside the neighborhood — and add a little bit of random noise to every one of its numbers. The point shifts slightly. The picture looks like an apple with some grain on it. Now do it again. And again, and again, hundreds of times, each time nudging the point a little further out, until the apple is completely buried in static and the point is nowhere near the neighborhood at all. That deliberate march outward has a name we will use for the rest of the book: the forward process — adding noise on purpose.

Why on earth would destroying our data help? Because it hands us something we did not have before: a path. We now know a route from inside the neighborhood all the way out into the randomness, and we know it step by step, because we built it ourselves, one small nudge at a time. Every intermediate point along that route is a fact we can write down.

A model, in this book, means a function with a great many adjustable numbers inside — numbers that get tuned until the function does what we want. That is the whole definition we need here; if you want to see one built from a single unit upward, the AI/ML book does it in the neuron.

And if a model can learn to walk that path backward — if, shown a slightly noisy apple, it can produce a slightly less noisy one, over and over — then something remarkable follows. It does not need to start from a real apple. It can start from anywhere in the randomness, from a fresh field of static that belongs to no apple at all, and take the same kind of small step, again and again, and calmly stroll into the good neighborhood. That walk back is the reverse process, also called the denoising process, and it is the second arrow on our opening diagram.

That is the answer to "why add noise". It converts one impossible jump into a long series of small, learnable steps. Each individual step is an easy question — this picture is almost an apple with a bit of grain; what would it look like with slightly less grain? — and a few hundred easy questions, answered in a row, accomplish what no single guess ever could.

One word to take with you: latent

One flag before we wrap, and it is a promise rather than a detour. Those points are enormous — 786,432 numbers each — and the reverse walk visits a point hundreds of times on the way in. Doing that arithmetic on 786,432 numbers, over and over, is punishingly expensive.

So in practice these systems do not walk in pixel space at all. They first squeeze each image down into a much smaller summary called a latent, do all their walking in that compact space, and expand the result back into an image only at the very end. You do not need any of the details yet — Chapter 4 is entirely about how that squeeze works and what it costs. For now, just take the word with you. That is the "latent" in latent diffusion, which is the name of the family of models this entire book is about.

See it move

Where you'll meet this

Open any image-generation workflow — ComfyUI, or the settings panel of a hosted tool — and the first thing it asks for is a width and a height. That is not a display preference. You are choosing $d$: how many numbers the thing you are about to generate will contain, and therefore which space the whole computation happens in. Ask for 512 by 512 in color and you have asked for a point in 786,432 dimensions, by the multiplication we did above. Doubling each side quadruples the count, and the cost follows it.

The second thing you will notice is that the word latent is written all over those interfaces — a node that makes an empty one, a wire colored differently from the image wires, a preview that looks like a smudged thumbnail. Every one of those refers to the compact summary we just named, and now you know why it exists: nobody wants to do the walk in 786,432 dimensions if a much smaller space will do.

And the third is a habit worth building now. When a generated image comes out wrong — melted, smeared, uncanny — the honest description is not "the model made a mistake". It is "the walk ended just outside the neighborhood". That framing will diagnose more problems for you than any list of settings.