Making Things Up

Sorters and makers

The line-fitter predicted prices, the classifier chose categories, and our small network learned inside from outside a curved region. This chapter builds a machine that makes a new picture. How can arithmetic produce a face nobody photographed?

A discriminative model predicts a label or value from an input. A generative model learns the distribution of the data itself so it can produce new samples; write input $x$ and answer $y$:

In words: a sorter models the probability of a label $y$ given an input $x$: "shown this photo, how likely is 'cat'?" A maker models $p(x)$, the probability of the input itself: "how likely is this arrangement of pixels to be a real photo at all?" Learn that second one and you can grade any image for realness, and generate new ones that score high.

We will compare three ways to learn that distribution: VAEs, GANs, and diffusion.

Learn the distribution, then sample

To make new data, learn its distribution, then sample. In the math toolkit, a distribution spread belief across outcomes: common things get more, rare things less.

We write that draw with a tilde:

In words: read the little tilde $\sim$ as "drawn from." We pick an $x$ at random, according to the distribution $p(x)$ rather than evenly, so common things come up often and rare things almost never. An LLM does the same thing when it samples a next token from a probability distribution (the sister site's prediction game). Here the "token" is an entire image.

For faces, two eyes above a nose are common; random static rarely resembles a face. Learning that pattern across millions of pixel dimensions is difficult: plausible images occupy a thin, curved region, not a table we can fill by hand.

We can also condition sampling, steering it with extra information: a smiling face, a cat in Van Gogh's style, or a harbor at dusk from the air.

VAEs: a map you can wander

PCA compressed many features into a few useful directions. A variational autoencoder (VAE) turns compression and reconstruction into a generator using two trainable networks.

The encoder compresses an image into a latent code, which is a small vector on a compact map. The decoder rebuilds the image from that code. Train both so encode-then-decode approximately restores the input.

In words: the image $x$ is encoded down to the latent code $z$, then decoded back up to $\hat{x}$, read "x-hat," the model's rebuilt version of the image. The hat means "the model's version," the same convention we used for the prediction $\hat{y}$ in earlier chapters; here the model is rebuilding an input rather than guessing a label.

A second training pressure keeps the latent codes near a smooth distribution around the origin, so nearby samples tend to decode plausibly. In our illustration, a face has code $z=(0.3,-0.8)$. Moving its first coordinate from $0.3$ toward $0.9$ widens the smile; a new point $z=(-0.5,0.4)$ decodes to a new face. These are illustrative coordinates, not universal smile controls.

That is how a VAE generates: sample a latent point and decode it. VAEs are stable to train and give an organized map, though averaging over possible reconstructions can produce blurry outputs.

GANs: the forger and the detective

A generative adversarial network (GAN) trains two networks against each other.

The generator is the forger: random numbers go in, a fake image comes out. The discriminator is the detective: it sees real images mixed with fakes and learns which is which. Both use gradient descent, with opposing goals.

1. Detect: early fakes are easy to recognize.
2. Improve: the discriminator’s error signal teaches the generator what gave it away.
3. Repeat: better fakes demand a sharper discriminator, which supplies harder feedback.
4. Generate: keep the trained generator; generation no longer needs the discriminator.

In the ideal balanced outcome, the detective can do no better than a coin flip, outputting about $0.5$. That is the goal, not a guarantee that every training run reaches it.

GAN images can be sharp because blur gives the detective a clue. But the contest is delicate: in mode collapse, the generator repeats one successful image or a narrow set instead of covering the distribution; the players can also keep chasing each other without settling.

Diffusion: learning to un-blur static

Teach a network to predict noise, then use it repeatedly to turn fresh static into an image.

This additive pixel example isolates the arithmetic; a full sampler also uses a noise schedule and scaling. Repeating small steps, a few hundred of them in this illustration, allows gradual refinement without the GAN's adversarial contest.

1. Start clean: choose a training image.
2. Add noise: gradually obscure it until it resembles static.
3. Learn: use noisy examples and their known noise to train a predictor.
4. Start fresh: draw new random static for generation.
5. Denoise: repeatedly apply the trained predictor, building a new image.

For text-to-image, encode “a red bicycle on the moon” into an embedding and supply it at every denoising step to steer the result toward the prompt. Meaning as Direction explains the text vectors. Conditioning guides a sample; it does not guarantee an exact picture.

How AI Makes Images gives the full derivation: noise schedules, prediction, compression, and text guidance.

Choosing a generator

Choose by the kind of control and the training tradeoff the task needs. All three families share one aim, learning the distribution and then sampling from it, and reach it by different routes: a smooth latent map, an adversarial contest, or repeated refinement.

Where you'll meet this