The Big Picture

Apple, static, apple

The whole series fits in one picture. We start with an apple. We slowly turn it into random static, the gray snow an old television showed when it had nothing to display. Then we turn that static back into an apple. It isn't the apple we started with. It's a brand-new one, a fruit that never existed and that nobody photographed, and it still looks believable.

This chapter builds that picture from nothing, so that by the end you can state the question the rest of the book answers. There's one piece of linear algebra in the middle. It's the idea the whole field rests on, and it's easier than its reputation.

What we're asking for

We want a machine that hands us a brand-new apple, one that never existed and still looks real. A machine that produces fresh examples of a thing, rather than sorting or labeling things it is shown, is called a generative model. Each thing it produces is a fresh sample, and in this book that sample is always an apple.

To learn what an apple is, it has to be shown apples, and they have to be all different. Red, green, golden. Big, small, spotted. Photographed from the side and from above. From that variety it can pull out what an apple is: the round body, the dimple at the top, the stem. If every training photo were the same apple on the same table in the same light, the model could not tell which parts are "apple" and which parts are just "that Tuesday afternoon".

An image is a point

To a computer, an image is a list of numbers. A pixel in a black-and-white image is a single number for brightness, running from 0 (black) to 1 (white). Take a tiny image, two pixels by two pixels, holding these four brightnesses:

In words: a two-by-two grid of brightnesses. A near-white pixel in the top-left, a dark one in the top-right, a darker one in the bottom-left, and a light one in the bottom-right.

To hand that grid to a computer we flatten it. Read the pixels left to right along the top row, then left to right along the bottom row, and write them out in one line. Top-left first, so $0.9$, then $0.2$, then $0.1$, then $0.8$. That gives us our clean image, which we will call $\mathbf{x}_0$, read "x-zero, the clean image". The subscript zero will earn its keep in the next chapter:

In words: the same four brightnesses, in reading order, written as one list of four numbers. Nothing was lost and nothing was added. Only the shape of the container changed.

A list written out like that has a name, and the name is a vector. That is the whole definition: an ordered list of numbers, and the order matters. Swap the last two entries and you have a different image. I'll use "list" and "vector" interchangeably from here.

A list of numbers is also a point. Two numbers are a point on a flat map: go $0.9$ along, $0.2$ up. Three numbers are a point in the room you are sitting in: along, up, and back. Four numbers are a point in a four-dimensional space, which nobody can picture and which behaves arithmetically like the first two cases. The word dimension gets used loosely everywhere, so pin it down here. One dimension is one number in the list, which is one independent direction you are free to move in. You needed four numbers to say where the point is, so the space it lives in has four. The list and the point are the same object, written two ways. (If lists-as-arrows is new, the math toolkit in the AI/ML book builds it from scratch, and numbers that point does the same for text.)

Real images are bigger. A modest square image is 512 pixels across and 512 pixels down, and each pixel in color carries three numbers: how much red, how much green, how much blue. There are three because the human eye has three kinds of color-sensing cell in it, so three numbers at every dot reproduce anything you can see. A screen works the same way, with three tiny lamps at every dot. Count them:

In words: the grid holds 262,144 pixels, and each one needs three numbers for its color, so the whole image is 786,432 numbers.

So one photograph of one apple is one list of 786,432 numbers, which is one point in a space with 786,432 dimensions. Mathematicians write "the space of all lists of $d$ numbers" as $\mathbb{R}^d$, so our apple is a point in $\mathbb{R}^{786{,}432}$. Don't try to visualize that. Nobody can. What this book does instead, and what every diagram you will see of this subject does, is draw two of those axes and quietly ignore the other 786,430. The picture is a cartoon. The reasoning behind it holds.

How far apart are two images?

Once images are points, "these two pictures look alike" stops being a feeling and starts being a measurement. Similar images sit near each other, different images sit far apart. And you already know how to compute distance between points. It's the Pythagorean theorem, in as many dimensions as you like.

Nothing about that calculation cared that we used two coordinates. With 786,432 of them you would subtract 786,432 pairs, square 786,432 differences, add them all up, and take one square root. Same formula, run longer. So "similar images are nearby points" is a computable statement, not a figure of speech.

The apple neighborhood

Scatter every real apple photograph you can find into that enormous space, one point each. They do not land at random. They clump into one connected region, because real apples share structure: round-ish outlines, smooth skin, a stem at the top, light falling from somewhere. Any two apple photos are more alike than either is to a photo of a bicycle, and "more alike" now means "closer".

That clump is called the data distribution. Saying that eight times a chapter is tiring, so I'll mostly call it the apple neighborhood. Same thing: the territory where apples live.

That gives us a precise definition of the job. Generating a new apple means finding a fresh point inside the apple neighborhood. Land inside it and you get a believable apple. Land outside it and you get a wash of noise or a smear. There is no third outcome, and there is nothing else to the problem.

Why a lucky guess never works

Picking a point inside the apple neighborhood by guessing is hopeless, and the reason dictates everything the rest of the book does.

The neighborhood is a small, complicated region floating in a space of hundreds of thousands of dimensions. High-dimensional spaces are far emptier than our intuition allows. Take a $d$-dimensional cube and ask what fraction of it lies in the middle half of every axis at once. Along one axis, the middle half is a fraction $1/2$. The axes are independent, so along two axes at once it is $\tfrac12 \times \tfrac12 = \tfrac14$, along three it is $\tfrac12\times\tfrac12\times\tfrac12 = \tfrac18$, and along $d$ of them:

In words: the share of the cube that is central in all $d$ directions at once is one half, multiplied by itself $d$ times.

Put a number in. At $d = 10$ that fraction is $1/2^{10}$, and $2^{10} = 1{,}024$, so about one part in a thousand. At $d = 20$ it is $1/2^{20}$, and $2^{20} = 1{,}024 \times 1{,}024 = 1{,}048{,}576$, about one part in a million. That is at twenty dimensions. We have 786,432 of them, and the pattern only steepens.

Picking a good apple point by luck is a blindfolded pin drop into a stadium, hoping to hit one particular seat, and that flatters the problem. A big stadium holds fifty thousand seats, so a blind drop finds your seat about one time in fifty thousand. Those are better odds than the twenty-dimensional cube we just computed, and far better than the real thing. Random guessing here is not a slow method. It is no method at all.

A path instead of a jump

So we stop trying to jump. The trick the whole field is built on is to move in small steps instead.

We teach a model to walk. Take a real apple — a point sitting comfortably inside the neighborhood — and add a little bit of random noise to every one of its numbers. The point shifts slightly. The picture looks like an apple with some grain on it. Do it again. And again, hundreds of times, each time nudging the point a little further out, until the apple is buried in static and the point is nowhere near the neighborhood. That march outward has a name we will use for the rest of the book: the forward process, which means adding noise on purpose.

Why would destroying our data help? Because it hands us a path. We now know a route from inside the neighborhood all the way out into the randomness, and we know it step by step, because we built it ourselves one small nudge at a time. Every intermediate point along that route is a fact we can write down.

A model, in this book, means a function with a great many adjustable numbers inside, tuned until the function does what we want. That is the whole definition we need here. If you want to see one built from a single unit upward, the AI/ML book does it in the neuron.

Now suppose a model can learn to walk that path backward: shown a slightly noisy apple, it produces a slightly less noisy one, over and over. It no longer needs to start from a real apple. It can start anywhere in the randomness, from a fresh field of static that belongs to no apple at all, take the same kind of small step again and again, and walk into the good neighborhood. That walk back is the reverse process, also called the denoising process, and it is the second arrow on our opening diagram.

That is the answer to "why add noise". It converts one impossible jump into a long series of small, learnable steps. Each step is an easy question: this picture is almost an apple with a bit of grain, so what would it look like with slightly less grain? A few hundred easy questions, answered in a row, accomplish what no single guess could.

One word to take with you: latent

Those points are enormous, 786,432 numbers each, and the reverse walk visits a point hundreds of times on the way in. Doing that arithmetic over and over costs far more than anyone wants to pay.

So in practice these systems do not walk in pixel space at all. They first squeeze each image down into a much smaller summary called a latent, do all their walking in that compact space, and expand the result back into an image only at the end. You don't need the details yet. Chapter 4 is about how that squeeze works and what it costs. For now, take the word with you. That is the "latent" in latent diffusion, the name of the family of models this book is about.

See it move

Where you'll meet this

Open any image-generation workflow, ComfyUI or the settings panel of a hosted tool, and the first thing it asks for is a width and a height. That is not a display preference. You are choosing $d$, the number of numbers in the thing you are about to generate, and therefore the space the whole computation happens in. Ask for 512 by 512 in color and you have asked for a point in 786,432 dimensions. Doubling each side quadruples the count, and the cost follows it.

The word latent is written all over those interfaces too: a node that makes an empty one, a wire colored differently from the image wires, a preview that looks like a smudged thumbnail. Every one of those refers to the compact summary we just named, and now you know why it exists. Nobody wants to do the walk in 786,432 dimensions if a much smaller space will do.

There's also a habit worth building now. When a generated image comes out melted or smeared, the useful description isn't "the model made a mistake". It's "the walk ended just outside the neighborhood". That framing will diagnose more problems for you than any list of settings.