Generating from Static · How AI Makes Images

Starting from nowhere

We have an encoder, a decoder, and a trained noise predictor. Time to use them. Everything in the last three chapters was preparation: Chapter 2 built the forward process — a path from a real apple out to pure static, Chapter 3 trained a network to look at any point on that path and name the noise sitting on it, and Chapter 4 moved the whole apparatus into a smaller space so it is affordable to run. Not one of those chapters made a new apple.

This one does. And the starting material is going to feel like a trick, because it is nothing at all: a grid of random numbers.

$$\mathbf{z}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$$

In words: our starting latent — the compact grid of numbers that Chapter 4 called a latent — is drawn from the standard bell curve, mean zero, spread one. Every one of its 16,384 numbers (the typical $64 \times 64 \times 4$ latent) is an independent random draw. The little tilde $\sim$ is read "drawn from", and $\mathcal{N}(\mathbf{0}, \mathbf{I})$ is standard Gaussian noise: each number pulled independently from the bell curve centered at 0 with spread 1.

Notice what $\mathbf{z}_T$ is not. It is not an apple that has been noised. No real photograph was harmed to produce it; there is no $\mathbf{z}_0$ hiding underneath it waiting to be recovered. It came from a random number generator. That is the honest answer to the question people always ask — where does the new apple come from? — and it is the reason the apple we are about to make is genuinely new rather than remembered.

Here is the plan, in three moves that we then repeat. Ask the noise predictor what noise it sees. Remove a controlled fraction of that noise. Add back a little fresh noise. Repeat, working the timestep downward, and at the end hand the result to the VAE decoder. The loop is called the sampler, and each pass around it is one of its sampler steps.

One loop, run a few dozen times, then one decode. The $K$ on the return arrow is just how many times round we go — a couple of dozen, typically, and a later section pins the number down. The latent never leaves the loop until the very end — which is why nothing that looks like a picture exists until Dec runs, exactly as the pipeline figure in Chapter 4 promised.

One sampler step, term by term

This is the equation the whole chapter is built on. It is the reverse process written as arithmetic — one rung of the ladder, from timestep $t$ down to $t-1$.

One honest word about it first, because everything up to this point in the book has been derived in front of you and this line will not be. It comes out of the same variance bookkeeping you did in Chapter 2, run in reverse, and the papers that first wrote it down spend several pages getting there. We are not going to spend those pages. What we will do instead is take the line apart term by term until every piece is something you can check, and then run the whole thing on one real number:

$$\mathbf{z}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{z}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,\boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t)\right) + \sigma_t \mathbf{w}, \qquad \mathbf{w}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$$

In words: take the current noisy latent, subtract a small multiple of the noise the network thinks is on it, divide the whole thing by the square root of alpha-t to undo the shrink the forward process applied, and finally sprinkle a little fresh noise back on. The symbols are all old friends: $\beta_t$ — beta-t — is how much noise the noise schedule added at timestep $t$; $\alpha_t = 1 - \beta_t$ is how much signal it kept; $\bar{\alpha}_t$ — alpha-bar-t — is how much of the original signal survives all the way from the start to timestep $t$; and $\boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t)$ is Chapter 3's noise predictor, its parameters $\theta$ now frozen — training is over, and not one of those numbers changes again while we generate. The two new symbols are $\sigma_t$ — sigma-t, the sampler's noise scale — and $\mathbf{w}$, a fresh noise draw.

Three moves are packed in there, and each deserves its own paragraph.

(a) Subtract a fraction of the predicted noise. The network hands us $\boldsymbol{\epsilon}_\theta$, a grid the same shape as the latent, holding its best guess at the noise. We do not subtract all of it. We subtract it scaled by $\beta_t/\sqrt{1-\bar{\alpha}_t}$, a number that is small whenever $\beta_t$ is small. Hold that thought — the next section is entirely about why.

(b) Rescale by $1/\sqrt{\alpha_t}$. The forward process did two things at every step: it shrank the signal by $\sqrt{\alpha_t}$ and then added noise. We have just dealt with the noise, so the shrink is what remains to undo, and dividing by $\sqrt{\alpha_t}$ undoes it exactly. This is bookkeeping, not intelligence — it keeps the numbers at the scale the network expects to see at timestep $t-1$.

(c) Add fresh noise, $\sigma_t\mathbf{w}$. This is the move that surprises everyone, so it gets its own section below. On the very last sampler step we set $\sigma_t = 0$ and the term vanishes, because we want to hand the decoder a clean latent, not a slightly speckled one.

One sampler step, fully worked

Let us run that equation on a single number. Real latents hold 16,384 of them and every one goes through exactly this arithmetic, independently, so watching one is watching all of them. We round to four significant figures in the middle of the derivation, here and everywhere else in this chapter.

Worked example: one sampler step

Our ingredients. The schedule at this timestep says $\beta_t = 0.2$, so $\alpha_t = 1 - 0.2 = 0.8$. The cumulative survival is $\bar{\alpha}_t = 0.5$: half the original signal strength is left, which is a sensible place to stand and watch a sampler step happen — far enough up the ladder that the latent is genuinely messy, far enough down that there is still something to work with. The current latent number is $z_t = 1.2$, and the noise predictor, shown this latent and this timestep, returns $\epsilon_\theta = 0.9$.

1. The denominator. $\sqrt{1-\bar{\alpha}_t} = \sqrt{1 - 0.5} = \sqrt{0.5} = 0.7071$.

2. The removal coefficient. $\dfrac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} = \dfrac{0.2}{0.7071} = 0.2828$.

3. How much noise we actually take off. $0.2828 \times 0.9 = 0.2546$.

4. The subtraction. $1.2 - 0.2546 = 0.9454$.

5. The rescale. $\sqrt{\alpha_t} = \sqrt{0.8} = 0.8944$, and $\dfrac{0.9454}{0.8944} = 1.0570$.

6. The fresh noise. For $\sigma_t$ we need a choice, because nothing so far has fixed one. The common choice, and the one we will use, is $\sigma_t = \sqrt{\beta_t}$: this rung's forward step added noise of variance $\beta_t$ on the way up, so on the way down we put back a helping of the same size and let the next prediction organize it. Here that is $\sqrt{0.2} = 0.4472$. Suppose the draw comes back $w = -0.3$. Then $0.4472 \times (-0.3) = -0.1342$, and $z_{t-1} = 1.0570 + (-0.1342) = 0.9228$.

The final step instead. If this were the last sampler step we would set $\sigma_t = 0$, the whole term would be $0$, and we would stop at $z_{t-1} = 1.0570$ — the same number, minus the sprinkle.

The one-step update equation drawn large across a dark panel, with each term pinned to its computed value by a thin muted leader line. Under the beta-over-root term sits the number 0.2828 in amber, labeled 'removal coefficient'. Under epsilon-theta sits 0.9 in violet, labeled 'predicted noise'. Under z-t sits 1.2 in mint. A chain of mint boxes beneath reads 1.2 minus 0.2546 equals 0.9454, then divided by 0.8944 equals 1.0570, then plus sigma-t w equals 0.9228. A small gray note reads 'last step: sigma-t equals 0, stop at 1.0570'. — **Every intermediate, pinned to the term that produced it.** Nothing in a sampler step is hidden; it is six pieces of arithmetic repeated once per latent number, per sampler step.

Why we remove only a fraction

Compare what we just did with what Chapter 3 could already do. Chapter 3 rearranged the forward process to estimate the clean latent in one leap:

$$\hat{\mathbf{z}}_0 = \frac{\mathbf{z}_t - \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t)}{\sqrt{\bar{\alpha}_t}}$$

In words: subtract all of the predicted noise, scaled by the full $\sqrt{1-\bar{\alpha}_t}$, then divide by $\sqrt{\bar{\alpha}_t}$ to restore the faded signal. That gives an estimate of the clean latent — right now, in one move.

So why not do that and go home? Look at the two coefficients side by side on our numbers. The one-leap version removes $\sqrt{1-\bar{\alpha}_t} = 0.7071$ times the predicted noise. The sampler step removes $\beta_t/\sqrt{1-\bar{\alpha}_t} = 0.2828$ times it. The ratio is $0.2828 / 0.7071 = 0.4$ — the sampler takes off exactly two fifths of what the leap would take off. (That $0.4$ is no accident: it is $\beta_t/(1-\bar{\alpha}_t) = 0.2/0.5 = 0.4$ — and it is worth reading those two numbers with Chapter 2's eyes. $1-\bar{\alpha}_t$ is the total noise variance the whole climb has piled onto the latent, and $\beta_t$ is the part that this one rung put there. So the fraction is this rung's share of the damage. The sampler undoes that share and leaves the rest for the rungs below it, which is exactly what taking one step down a ladder ought to mean.)

And the leap itself, worked, so you can see what we are declining: $0.7071 \times 0.9 = 0.6364$; $1.2 - 0.6364 = 0.5636$; $\sqrt{0.5} = 0.7071$; $0.5636 / 0.7071 = 0.7970$. One move, and we have a candidate clean latent of $0.7970$.

The trouble is what that number means at a high timestep. When the latent is mostly static, an enormous number of different apples could have produced it, and the network was trained to minimize average squared error across all of them — and when a great many answers are plausible, the setting that scores best on average is the average of those answers. Its best answer is therefore an average of all of them — which is to say, a blur. Committing to it fully bakes that blur into the result. Removing two fifths instead leaves the latent still noisy, still ambiguous, and hands it back to the network at a lower timestep where the field of possible apples has narrowed. Each pass, the network gets to look again with more evidence. Many small commitments beat one large one.

Why we add noise back

Now the strange term. Having worked to remove noise, we deliberately put some back: $\sigma_t\mathbf{w}$, with $\mathbf{w}$ a fresh draw from the same standard bell curve. Every reader's first instinct is that this must be a mistake.

It is not, and the reason is that the update rule in step (a) is deterministic. Feed it the same latent twice and it produces the same output twice. If we ran a purely deterministic loop from a fixed start, the sampler would follow one rigid path, and at high timesteps that path leads toward the average apple — the safe, smooth, slightly waxy apple that sits in the middle of everything the model has learned. The fresh noise keeps the walk live. It nudges the latent off the average path and into one specific, committed apple, and it does so at every rung, which is why the model has room to invent detail that the average apple would never have.

There is a second, more mechanical way to say the same thing. The forward process of Chapter 2 threw away information at every step. A reverse step cannot conjure that information back; the honest thing is to re-supply a matching amount of randomness and let the next prediction organize it. That is what $\sigma_t\mathbf{w}$ is: not a mistake, but the missing information, replaced with something the network can shape — and it is why we sized it at $\sqrt{\beta_t}$, matching the amount that rung took away.

It is also why the last step is the one exception. Every other rung hands its result back to the network, which can absorb a little jostling and will tidy it up on the next pass. The last rung hands its result to the decoder, which cannot. So on that step, and only that step, we set $\sigma_t = 0$ and keep the sprinkle to ourselves.

One sentence on the alternatives, and then we move on: several well-known samplers — DDIM, Euler, DPM++ among them — make different choices about this noise term and about how to take the step, and they trade sharpness, speed, and variety against one another. The update rule above is the one to understand first; the others are variations on it.

Training steps are not sampler steps

This is the single most-confused pair of numbers in the whole field, and this chapter owns it. There are two entirely different step counts in play, and they are not the same quantity measured twice.

Note: $T$ and $K$ are different numbers

$T$ is the length of the noise schedule — the fixed ladder of rungs defined once, before training, with a $\beta_t$ value on each. Typically $T = 1000$. It is a property of the model, and nothing you do at generation time changes it. Why a thousand? Because of the assumption in the "Slow down" note above: a reverse step is only a good approximation when each rung is a tiny nudge, and putting a thousand rungs between a clean picture and pure static is what makes every rung tiny.

$K$ is the number of sampler steps: how many of those rungs the sampler actually visits on the way down. Typically $K = 20$ to $50$. With $T = 1000$ and $K = 20$, the sampler strides $1000/20 = 50$ rungs at a time and visits $t = 1000,\, 950,\, 900,\, \ldots,\, 100,\, 50$ — twenty timesteps out of a thousand.

So why twenty, and not the full thousand? Cost. Every sampler step is one complete pass of the network over the latent, so running a thousand of them takes fifty times as long and produces a picture most people cannot tell apart from the twenty-step one. Striding does cost a little accuracy, because each stride is a bigger jump than the approximation was designed for. Twenty to fifty is where that trade sits comfortably.

So: $K \le T$, always. When a ComfyUI workflow shows a box labeled steps: 20, that number is $K$. "The model was trained for 20 steps" is wrong. "It denoises a thousand times when you generate" is also wrong. Both numbers are real; they are just answers to different questions.

The deeper distinction is what training and generation each do, and they could hardly be less alike. Training, from Chapter 3: take a real apple, encode it, pick one random timestep, manufacture the noisy latent with the closed form, ask the network for the noise, compare against the noise we drew ourselves, and nudge $\theta$ downhill. One timestep. No loop. An answer key. A loss.

Generation, this chapter: start from noise that belongs to no apple, walk $t$ downward through $K$ rungs, and at no point compute a loss, because there is no right answer to compare against. The parameters $\theta$ are frozen; nothing learns; nothing is stored. The network is a fixed function being evaluated $K$ times.

Two columns side by side on a dark panel. The left column, headed TRAINING in amber, lists: real apple in, one random timestep t, noise drawn by us as the answer key, one prediction, a loss computed in red, theta updated. The right column, headed SAMPLING in mint, lists: pure noise in, timesteps walked downward from T, no answer key, K predictions, no loss, theta frozen. A muted divider runs between them and a footer note reads 'the same network, used two completely different ways'. — **Same network, two jobs.** Training has an answer key and never runs the loop; sampling runs the loop and never has an answer key. Nearly every confusion about diffusion dissolves once these two columns stay separate in your head.

What emerges, and when

Run the loop and watch the decoded latent along the way, and the apple does not fade in evenly. Broad structure arrives first — a mass here, a background there. Then color settles across those regions. Then edges sharpen, and only in the last few sampler steps do the small things appear: the speck of highlight, the stem, the faint mottling on the skin.

That ordering is not a stylistic choice; it falls out of the schedule. Early sampler steps sit at high $t$, where $\bar{\alpha}_t$ is tiny and almost nothing of the original signal survives — so the only thing the network can meaningfully commit to is coarse layout. Late sampler steps sit at low $t$, where $\bar{\alpha}_t$ is close to 1, the latent is nearly clean, and the only remaining work is fine detail. The apple is built the way the forward process tore it down, in reverse order.

The mirror image of Chapter 2's noise ladder. Where Chapter 2 showed one apple dissolving from clean to static, this shows a different apple assembling from static to clean — and the two strips run in opposite directions along the same schedule. This figure is currently a hand-drawn placeholder; the real strip needs five panels decoded from actual latents.

At $t = 0$ the loop exits with a clean latent, $\mathbf{z}_0$. It is still not an image: it is 16,384 numbers in a learned code. The last act is the one node Chapter 4 named:

$$\hat{\mathbf{x}}_0 = \mathrm{Dec}(\mathbf{z}_0)$$

In words: hand the finished latent to the VAE decoder, and it expands those 16,384 numbers back into $512 \times 512 \times 3 = 786{,}432$ pixel numbers. That is the moment the apple exists as something you can look at.

And it is a new apple. Not the apple from Chapter 2, not any apple in the training data — a fresh sample that has never existed before, assembled out of a random grid by a function that learned what apples look like in general.

Where the seed lives

We have been saying "random numbers" as though they arrived from nowhere. They arrive from a pseudorandom generator: a deterministic machine that takes one integer — the seed — and produces a stream of numbers that behave like random draws but are completely reproducible. (The sister book's chapter on chance and choice covers what "random" means to a computer, if you want the longer version.) The chain is short and worth naming end to end:

seed $\rightarrow$ pseudorandom stream $\rightarrow$ the starting latent $\mathbf{z}_T$ $\rightarrow$ (through $K$ sampler steps) $\rightarrow$ $\mathbf{z}_0$ $\rightarrow$ $\mathrm{Dec}(\mathbf{z}_0)$ $\rightarrow$ the image.

Everything after the first arrow is fixed arithmetic. So if you keep the seed, the model, the schedule, $K$, and the sampler identical, you get a pixel-identical apple back, today and next year. That is reproducibility, not memory — the machine is not recalling the apple, it is recomputing it. (The fresh-noise draws $\mathbf{w}$ come from the same seeded stream, so they are reproducible too; with $\sigma_t = 0$ throughout there is no randomness after the start at all.) Change the seed by one, and every one of those 16,384 starting numbers changes, the loop walks a different route, and a different — equally valid — apple comes out.

Same model, same settings, different starting static. Nothing about the model changed between these two rows — only the integer that seeded the noise. Variety in diffusion comes from where you start, not from anything stored.

See it move

Watch (18:28): track the removal coefficient in the corner as $t$ falls. It starts tiny — the sampler barely commits when the latent is mostly static — and the apple's outline is settled long before any of its detail is.

Where you'll meet this

Open any image-generation interface and you are looking at the contents of this chapter, relabelled. The box marked steps is $K$, the number of sampler steps — set it to 4 and the sampler takes enormous, over-committed strides and gives you the smeared, over-committed result the annealing callout predicted; set it to 150 and you mostly buy diminishing returns at 150 times the cost, since each sampler step is one full pass through the noise predictor. The box marked seed, with its little "randomize" toggle, is the integer at the head of the chain above; locking it is how people iterate on one apple instead of a new one every time. The dropdown of sampler names — Euler, DDIM, DPM++ — is a choice of update rule and of what to do with $\sigma_t$. And the VAE Decode node sitting downstream of it all is $\mathrm{Dec}$, which is why the preview stays an abstract smear until the very last moment.

Two failure modes you can now diagnose rather than guess at. If your images look soft and averaged, you are likely committing too hard per step — too few sampler steps. And if you change "nothing" between two runs and get a different apple, check whether the seed was on randomize: everything downstream of that integer is deterministic, so a changed image means a changed input somewhere.

What you now know

Generation starts from $\mathbf{z}_T \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ — a latent of pure random numbers that belongs to no apple and was retrieved from nowhere.
One sampler step subtracts a fraction $\beta_t/\sqrt{1-\bar{\alpha}_t}$ of the predicted noise, rescales by $1/\sqrt{\alpha_t}$, and adds $\sigma_t\mathbf{w}$ back on: with $\beta_t = 0.2$, $\bar{\alpha}_t = 0.5$, $z_t = 1.2$ and $\epsilon_\theta = 0.9$, that ran $1.2 - 0.2546 = 0.9454$, then $0.9454/0.8944 = 1.0570$.
We remove only $0.2828$ of the predicted noise rather than the full $0.7071$ because a prediction made at a high timestep is an average over every apple that could have produced the latent, and averages are blurry — many small sampler steps let the network look again.
The fresh noise term is not a mistake: without it the loop is deterministic and drifts toward the average apple, and it is set to zero only on the final sampler step.
$T$ and $K$ are different numbers: the schedule has $T = 1000$ rungs, a sampler typically visits $K = 20$ to $50$ of them, and the "steps" box in a workflow is always $K$.
The seed picks the starting noise field, not a stored picture — same seed and same settings reproduce the same apple by recomputation, and $\mathrm{Dec}(\mathbf{z}_0)$ turns the finished 16,384-number latent into 786,432 pixel numbers at the very end.

Where we're headed. You can now run the whole machine end to end: noise in, a brand-new apple out, every number along the way accounted for. But notice what never once entered the loop — any statement of what we wanted. The noise predictor took exactly two inputs, $\mathbf{z}_t$ and $t$, and it produced an apple only because apples are what it was trained on; had it been trained on something else, the same loop would have produced that instead, and we would have had no say in the matter. Next chapter introduces conditioning — the machinery that turns a prompt into a third input the denoiser can follow — because right now we can make an apple, but we cannot ask for a green one.