Making Things Up

Sorters and makers

Every model we have built so far is a sorter. The line-fitter in fitting a line sorted houses into prices; the classifier in drawing boundaries sorted photos into cat-or-dog; the little network in the neuron sorted a curved region into inside-or-out. Point any of them at a photo and it answers a question about the photo. This chapter builds a different kind of machine entirely — one that does not answer questions about pictures but paints pictures of cats that have never existed, writes faces that belong to no one, invents landscapes no one has ever stood in. How does a pile of arithmetic learn to make things up?

Let us name the two jobs precisely, because the distinction is the spine of the whole chapter. A discriminative model learns to tell classes apart: given an input $x$, it returns an answer $y$. That is every model in the book so far. A generative model learns something deeper and stranger: it learns where the data itself lives — the whole spread of what real cat-photos look like — so that it can reach into that spread and pull out a fresh one. Two jobs, two kinds of probability:

In words: a sorter models the probability of a label $y$ given an input $x$ — "shown this photo, how likely is 'cat'?" A maker models $p(x)$, the probability of the input itself — "how likely is this arrangement of pixels to be a real photo at all?" Learn that second thing and you can grade any image for realness, and — the whole point — generate new ones that score high.

There are three great families of generative model — VAEs, GANs, and diffusion — and they are three different answers to one hard question. We will build the intuition for each, land on diffusion (the family behind today's image generators) in the most detail, and finish by tracing exactly how a sentence of text becomes a picture.

The one trick: learn the distribution, then sample

Underneath all three families is a single idea, and it is worth stating plainly before we meet any of them: to make new data, first learn the distribution of the real data, then draw a random sample from it. Recall probability from the math toolkit — a distribution is just belief spread across outcomes, telling you which values are common and which are rare. For a dataset of face-photos, the distribution says two-eyes-above-a-nose is overwhelmingly common; static-snow is essentially never a face. Learn that, and generating is nothing more than reaching in and grabbing.

That reaching-in has a notation, and it is new, so we will introduce it gently and use it exactly one way for the rest of the chapter:

In words: read the little tilde $\sim$ as "drawn from." We pick an $x$ at random — but not evenly. We pick it according to the distribution $p(x)$, so common things come up often and rare things almost never. This is the very same move the LLM made when it sampled a next token from a probability distribution (the sister site's prediction game) — except here the "token" is an entire image.

Now confront why this is genuinely hard, so the three families feel earned rather than arbitrary. The distribution of real face-photos lives in a space of millions of pixel-dimensions, and the region where real faces actually sit is a vanishingly thin, wildly curved sliver of that space. You cannot write it down as a formula, and you cannot fill in a table. Each of the three families is a different clever trick for learning that impossible-to-write-down distribution well enough to sample from it. That is the whole plot.

And here is the payoff, so you are not kept waiting for it: once a model holds the distribution, you can do more than sample blindly — you can steer the sampling with a condition. "A face, but smiling." "A cat, in the style of Van Gogh." "A harbour at dusk, from the air." Conditioning is what turns a random-face machine into a type-a-sentence-get-a-picture machine, and we return to it at the very end.

VAEs: a map you can wander

The first family is the gentlest, and you already own its ancestor. Recall PCA from the classical toolbox: it squashed many features down to a few directions that held most of the variation — a compression. A variational autoencoder (VAE) is that idea supercharged into a generator, with two trainable networks in place of a fixed formula.

Walk the two halves. The encoder takes a full image and squeezes it down to a tiny handful of numbers — its latent code, a single point on a compact map. The decoder takes that point and rebuilds the full image from it. Train the pair together so the rebuilt image matches the original: encode-then-decode should return roughly what went in.

In words: the image $x$ is encoded down to the latent code $z$, then decoded back up to $\hat{x}$ — "x-hat," the model's rebuilt version of the image. The hat means "the model's version," the same convention we used for the prediction $\hat{y}$ in earlier chapters; here the model is rebuilding an input rather than guessing a label.

Now the twist that makes it generative rather than merely a compressor. A VAE is trained with a second pressure alongside "rebuild faithfully": the latent space must be smooth and packed near the origin — no empty gaps, no far-flung islands — so that every point on the map decodes to something plausible. Use the map. A particular face encodes to the two-dimensional code $z = (0.3, -0.8)$. Nudge the first coordinate from $0.3$ up toward $0.9$ and the decoded face's smile smoothly widens; the map has a "smile direction" you can slide along. And because the map has no holes, you can also pick a code you have never seen — say $z = (-0.5, 0.4)$, drawn at random near the origin — hand it to the decoder, and out comes a coherent face nobody ever photographed. That is generation in one sentence: sample a latent point, decode it.

One honest tradeoff, stated once. VAEs are stable to train and give you that beautifully organized, steerable map — but because they average over possibilities to keep the map smooth, their outputs tend to come out a touch blurry, like a photo through frosted glass. That softness is exactly the flaw the next family was invented to beat.

GANs: the forger and the detective

The second family is the most theatrical. A generative adversarial network (GAN) throws out the idea of measuring image quality with a fixed formula and instead sets up a contest between two networks.

Name the two players with the analogy that makes it stick. The generator is a forger: it takes a handful of random numbers and paints a fake image, trying to pass it off as real. The discriminator is a detective: it is shown a mix of real training images and the forger's fakes, and its only job is to call each one real or fake. Both learn by gradient descent — the same downhill nudging from teaching the network — but they pull in opposite directions: the forger wants the detective fooled, the detective wants the forger caught.

The dynamics are the lesson, so watch the escalation. Early on, the forger's images are laughable and the detective spots them instantly — a strong error signal that teaches the forger exactly what gave it away. So the forger improves; so the detective must sharpen; so the forger improves again. Round after round, an arms race, until the forger's fakes are so convincing the detective can do no better than a coin flip — it outputs about $0.5$, "I genuinely cannot tell." At that point the generator has learned to produce samples indistinguishable from real data, and here is the punchline of the whole scheme: you keep the forger and throw the detective away.

One honest tradeoff, once. When a GAN works, its images are razor-sharp — no VAE blur, because nothing here rewards averaging; a blurry fake is an easy catch for the detective. But the arms race is delicate. It can collapse — the forger discovers one image that always fools the detective and refuses to make anything else, a failure called "mode collapse" — or the two players can chase each other forever without settling. GANs gave us the first genuinely photorealistic faces, and then a third family arrived that was both sharp and stable.

Diffusion: learning to un-blur static

The third family is the one behind today's image generators, and it is worth building carefully. The idea sounds almost too plain to work: teach a network to remove a little noise from an image, then run it over and over to carve a picture out of pure static. One strip of pictures carries the whole idea.

Build the forward process first, because it is the easy half and it makes the training data for free. Take a clean training image and add a small dose of random noise. Add a little more. And more. After enough steps the image is unrecognizable — indistinguishable from pure TV static. This destruction is cheap and easy to do, and you can stop at any step, so you can manufacture unlimited pairs of "slightly-noisier" and "slightly-cleaner" images. That is the muted "add noise" arrow running left to right across the strip.

Now the training job, in one line: teach a network to undo a single step of noising — shown a noisy image, predict the noise that was added, so it can be subtracted off. Do it on a single pixel to keep it checkable by hand.

Land the generative magic — running the trained denoiser backward, from nothing. Start with a fresh page of pure random static, no image at all. Ask the network: "if this were a slightly-noisy picture, what noise would I remove?" Subtract that guess. Now it is a hair less random. Ask again. Subtract again. Repeat for a few hundred tiny steps, and a coherent image condenses out of the static — a face, a landscape, a cat — that was never there to begin with. That is the mint "learned denoising" arrow running right-to-left across the strip, the exact opposite of the noising arrow.

Say why diffusion won, briefly and honestly. It is stable to train — no GAN arms race, just an ordinary predict-the-noise loss graded by gradient descent like everything else in this book — and it produces sharp, varied images, with no VAE blur and no mode collapse, because it builds each picture through hundreds of small, correctable steps instead of one giant leap. That combination is why the image generators you have heard of are diffusion models.

What you just read is the idea in miniature. The full build — the noise schedule that decides how much static each step adds, the network that predicts it, the compression trick that makes running the whole thing affordable, and how a sentence steers the result — is its own site: How AI Makes Images, the third and last of these three. It starts from the same apple-to-static-to-apple arrow drawn above and does every step in front of you, with the numbers on the page. If this section is the trailer, that is the film.

Where you'll meet this

Time to cash the opening promise: how does a sentence become a picture? Recall that on the sister LLM site, text became vectors — a sentence gets encoded into an embedding that captures its meaning as a direction (the sister site's chapter on meaning as direction). A text-to-image generator is a diffusion model with one addition: at every denoising step, it is also handed that sentence's embedding and told to steer its noise-removal toward images that match that meaning. Same predict-the-noise engine, now conditioned. "A red bicycle on the moon" becomes an embedding, and the denoiser carves static into exactly that.

Here is the family-picker, so the chapter leaves you with judgment and not just three stories. Want a smooth, steerable map of your data and don't mind some softness? Reach for a VAE. Want maximum sharpness on a narrow domain like faces, and can babysit a delicate training run? A GAN. Want state-of-the-art, stable, controllable image or audio generation? Diffusion — the current default. All three share the one spine: learn the distribution, then sample.

Widen past pretty pictures, because the reach is enormous. The same "learn a distribution and sample from it" idea designs protein structures and drug candidates — a diffusion model can denoise its way to a novel structure the way it denoises to a novel face — synthesizes speech and music, and fills in missing data. And here is the thread back to the whole site: an LLM is itself a generative model. It learned the distribution of text and samples the next token from it — the machine that writes and the machine that paints are cousins.

One measured word on the tension this technology carries. Because these models learn the distribution of whatever they are fed, they inherit its patterns and its biases; and because they invent rather than retrieve, they can produce convincing things that are entirely false — which is exactly why the next chapter's idea, learning from feedback about what we actually want, matters so much.