The Compression Trick

Pixels are the problem

Our noise predictor works on pixels, and pixels are the problem. In Chapter 3 we built a network that looks at a noisy apple and points at the noise inside it, and everything about that network is correct. It is also, run the way we have described it so far, ruinously expensive — because every single thing it touches is a list of 786,432 numbers, and it has to touch that list over and over before an apple appears.

This is the chapter where we collect on a promise. Back in Chapter 1 we said that images are big and expensive, that in practice we squeeze each one into a small summary first, and that you should just remember the word: latent. That was the whole explanation you were given, and it was deliberately incomplete. Here is the rest of it. By the end of this chapter you will know exactly what a latent is, where it comes from, why the entire field is named after it, and what that VAE Decode box at the end of every workflow graph is actually doing.

The plan is short enough to state in one breath: train a separate little pair of networks that can shrink an apple down and blow it back up again, then run all of Chapter 2 and all of Chapter 3 on the shrunken version. Not one equation changes. Only the meaning of the numbers changes. That is the trick, and it is worth doing the arithmetic to see how much it buys.

Counting the cost honestly

Chapter 1 counted an image out: a 512 × 512 color photograph of our apple is $512 \times 512 \times 3 = 786{,}432$ numbers, one apple at one moment in pixel form. That number is the entire motivation for this chapter, so carry it forward as it stands.

786,432 is arithmetic, not a configuration choice — any 512 × 512 color image has exactly that many numbers. Now put it in motion. The reverse process does not run the noise predictor once. It runs it many times over, each pass reading the whole array and writing a whole new array of the same size. Take fifty passes as an illustrative figure — the exact count is a knob, and Chapter 5 is where we turn it — and the numbers moving through the machine come to:

In words: fifty passes over 786,432 numbers means about 39.3 million numbers read and rewritten, just to produce one picture — and that counts only the inputs and outputs, not the far larger pile of arithmetic happening inside the network on each pass.

Every one of those numbers costs memory to hold and arithmetic to update. Worse, the cost grows with the square of the image's width: doubling to 1024 × 1024 quadruples the count to 3,145,728. Generating at any respectable size becomes something you rent a datacenter for, not something that runs on the graphics card in your desk. That is the wall, and hitting it is what sent people looking for a smaller place to work.

A learned squeeze and a learned expand

Here is the observation that opens the door. Almost none of the 786,432-number lists that exist are pictures of anything. Chapter 1 made this point geometrically — real apples occupy a vanishingly thin sliver of the enormous space of possible images. Turn that around and it becomes an opportunity: if real images only ever live in a thin sliver, we do not need enough room to describe every possible list of numbers. We only need enough room to describe the ones that actually occur.

So we train a pair of networks to find that smaller description. The first is the VAE encoder, which eats an image and produces a small array of numbers. The second is the VAE decoder, which eats that small array and produces a full-size image again. Together the pair is called the VAE, and the small array in the middle is a latent. The room those small arrays live in is latent space.

They are trained together, and their training goal is the plainest one imaginable: squeeze an image, expand it again, and make what comes out match what went in. Nothing about noise, nothing about diffusion, no apples in particular — a great many varied images, squeezed and expanded, with the pair adjusted until the round trip stops losing anything you would notice. Written down, that is three short statements:

In words: run the clean apple $\mathbf{x}_0$ through the encoder and you get $\mathbf{z}_0$ — "z-zero, the latent". Run that latent through the decoder and you get $\hat{\mathbf{x}}_0$ — "x-zero-hat", our reconstruction of the apple. The wavy equals sign says the reconstruction is very close to the original, not identical to it.

That approximate sign is honest and it matters. The round trip is lossy: some information is genuinely thrown away and cannot be recovered. What makes the trick work is which information gets thrown away. The encoder has seen enough real photographs to have learned what real photographs are like — that neighboring pixels tend to agree, that edges continue, that a shadow on one side of an apple implies a highlight on the other. It keeps the structure that real images have, and it spends nothing on describing the wild pixel-by-pixel combinations that never occur in any photograph anyone has ever taken. Feed it a photo and the round trip is nearly perfect. Feed it pure static and the round trip is terrible — which is exactly the trade we wanted.

How small is the latent?

Now the numbers that name the field. A latent is not a flat list — it keeps a grid shape, so that position in the latent still corresponds, loosely, to position in the picture. For a Stable-Diffusion-style model, which is the configuration we will quote throughout this book, two choices set the size. The first is the compression factor $f$ — "the compression factor" — which is how many pixels along each side collapse into one latent cell. Typically $f = 8$, and eight is not a law of nature: it is where the people building these models landed, because squeezing harder costs you detail you can see, and squeezing more gently does not save enough to be worth it. The second is how many numbers each latent cell carries, called its channels, in the same sense as the picture's color channels: how many separate numbers are stacked up at each position. Typically 4, in place of the picture's 3.

Both of those are configuration, not law — a different model may use a different factor or a different channel count. With the typical values, the grid arithmetic runs like this:

In words: 512 pixels along a side, collapsed 8 to 1, leaves 64 latent cells along that side. A 64-by-64 grid holds 4,096 cells, and 4 numbers per cell gives 16,384 numbers for the whole latent.

Set the two counts side by side and divide. This is the number that justifies the entire chapter, so let us do the division rather than announce it:

In words: a latent holds 48 times fewer numbers than the picture it came from. You can check it by multiplying back: $16{,}384 \times 48 = 16{,}384 \times 50 - 16{,}384 \times 2 = 819{,}200 - 32{,}768 = 786{,}432$.

There is a second route to the same 48, and it is worth seeing because it shows where the saving comes from. Each side shrinks by 8, so the grid shrinks by $8 \times 8 = 64$. But the channels go the other way, from 3 up to 4, which costs us a factor of $\tfrac{4}{3}$. Net: $64 \times \tfrac{3}{4} = 48$. The spatial squeeze does all the work; a little of it is handed back to buy the extra channels, and those extra channels are what let the encoder store richer information per cell than a color ever could.

Rerun the cost from the top of the chapter with those numbers. Fifty passes over a latent is $50 \times 16{,}384 = 819{,}200$ numbers moved, against 39,321,600 in pixel space. The saving is the same factor of 48, and it is the difference between a rented datacenter and the card in your desk.

A toy compression you can check by hand

"Lossy but structure-preserving" is the kind of phrase that sounds like an excuse until you watch it happen on numbers small enough to hold. So here is a miniature encoder and decoder, with a rule fixed by hand instead of learned, on an "image" of four numbers.

That is the whole idea in miniature. The compression is perfect when the data has the structure the rule expects (neighbors agreeing) and imperfect when it does not. Our toy rule was written by hand, so it only knows one kind of structure. A real VAE encoder has millions of adjustable numbers inside and learns its own notion of what image structure looks like, from a great many photographs — which is why it can afford to throw away 47 numbers out of every 48 and still hand you back an apple you would not think twice about.

The latent is not the U-Net's bottleneck

This section exists to head off one specific confusion, because two different small things sit in the middle of two different pictures and they are not the same thing.

Chapter 3 described the U-Net as a shape that narrows and then widens — shrink the input to see the big picture, grow it back to place the fine detail. The narrow middle of that U is called the bottleneck. It is tempting to say "ah, the bottleneck is the latent." It is not, and the difference is worth stating precisely.

The latent is what the diffusion machinery operates on. It is produced once by the VAE encoder, it persists from one denoising pass to the next, and it is still sitting there when the whole process finishes, waiting to be decoded. The U-Net's bottleneck is an internal activation inside a single pass of the noise predictor: it comes into existence halfway through one forward run, it is consumed by the widening arm of the same run, and it is gone before the pass ends. Every one of our fifty passes creates and discards its own bottleneck, and all fifty of them are working on the same latent.

The substitution that names the field

Now the payoff, and it is almost anticlimactic, which is the point. Take Chapter 2's forward process in its closed form, and Chapter 3's loss. Replace $\mathbf{x}$ with $\mathbf{z}$ everywhere. Stop. That is the whole modification.

Here they are as Chapter 2 and Chapter 3 wrote them, in pixels:

In words: the noisy image at noise level $t$ is the clean image faded by $\sqrt{\bar{\alpha}_t}$ — "the square root of alpha-bar-t", the fraction of the original signal still surviving — plus one blob of noise $\boldsymbol{\epsilon}$ scaled by the rest. And the loss is how far the noise predictor's guess falls from the noise we actually added.

And here they are in latent space:

In words: the noisy latent at noise level $t$ is the clean latent faded by the same $\sqrt{\bar{\alpha}_t}$, plus a blob of noise scaled by the same $\sqrt{1-\bar{\alpha}_t}$. And the loss is measured on the noise predictor's guess about the noise in a latent instead of the noise in a picture.

Read the two pairs against each other and notice what did not change. Not the noise schedule, not $\bar{\alpha}_t$, not the square roots, not the shape of the loss, not the training procedure, not a single coefficient. Every derivation from Chapter 2 stands untouched. The variance-preservation argument still holds, because it was an argument about the coefficients, and the coefficients are the same. The only difference is what the letters point at: $\boldsymbol{\epsilon}$ is now a blob of 16,384 random numbers shaped like a latent instead of 786,432 shaped like a picture, and the noise predictor is a network whose input and output are latent-shaped.

Adding noise to a latent, and taking it back out again, is what "latent diffusion" means. Two words, and you have now earned both of them.

See it move

Where you'll meet this

Open any image-generation workflow graph — ComfyUI is the common one — and you will find a box near the right-hand end labeled VAE Decode. It has two inputs: a wire coming in from the node that runs the denoising — in ComfyUI it is labeled KSampler — carrying something the interface labels LATENT, and a second wire carrying the VAE itself, loaded from a file. It has one output, labeled IMAGE, which usually runs straight into Save Image.

That box is $\mathrm{Dec}(\cdot)$. Not an analogy for it — that is literally the network this chapter has been describing, sitting at the very end of the pipeline. Everything to its left is operating on 16,384 numbers. Everything to its right is a picture.

Three practical things fall straight out of this, and they are the sort of thing that otherwise takes a month of confused forum reading to piece together.

First, why nothing appears until the end. People often expect to watch an image sharpen up while it generates, and instead the interface sits there and then produces a finished picture all at once. That is because there is no image until the decoder runs. What exists during generation is a latent, and a latent is not viewable — hence those approximate previews we flagged earlier, which are a translation of the latent, not the thing itself.

Second, why the VAE is a separate file you can swap. Encoder and decoder are their own small networks, trained separately from the noise predictor, so a model release ships them as their own weights. Swapping in a different VAE decoder changes the fine texture and color of your output while leaving the composition alone — the latent was the same; only the expansion of it changed. And if you load a VAE that was trained against a different latent layout than your model produces, you get a decoder reading a code it does not speak, which comes out as colorful garbage rather than an error message.

Third, why there is a matching VAE Encode box, and when you need it. Everything so far has run left to right from noise. But if you hand the system an existing photograph and ask it to modify that — the workflow usually called image-to-image — the picture has to become a latent before any of the machinery can touch it. That is the encoder, $\mathrm{Enc}(\cdot)$, doing the other half of this chapter's job.