Generating from Static · How AI Makes Images

Starting from nowhere

We have an encoder, a decoder, and a trained noise predictor, and this chapter uses them to make an apple. The last three chapters were preparation. Chapter 2 built the forward process, a path from a real apple out to pure static. Chapter 3 trained a network to look at any point on that path and name the noise sitting on it. Chapter 4 moved the whole thing into a smaller space so it is affordable to run. None of them made a new apple.

The starting material will feel like a trick, because it is nothing at all: a grid of random numbers.

$$\mathbf{z}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$$

In words: our starting latent, the compact grid of numbers that Chapter 4 called a latent, is drawn from the standard bell curve, mean zero, spread one. Every one of its 16,384 numbers (the typical $64 \times 64 \times 4$ latent) is an independent random draw. The tilde $\sim$ is read "drawn from", and $\mathcal{N}(\mathbf{0}, \mathbf{I})$ is standard Gaussian noise: each number pulled independently from the bell curve centered at 0 with spread 1.

$\mathbf{z}_T$ is not an apple that has been noised. No real photograph went into it, and there is no $\mathbf{z}_0$ hiding underneath waiting to be recovered. It came from a random number generator. That is the answer to the question people always ask, where does the new apple come from?, and it is why the apple we are about to make is new rather than remembered.

The plan is three steps, repeated. Ask the noise predictor what noise it sees. Remove a controlled fraction of that noise. Add back a little fresh noise. Repeat, working the timestep downward, and at the end hand the result to the VAE decoder. That loop is the sampler, and each pass around it is one of its sampler steps.

One loop, run a few dozen times, then one decode. The $K$ on the return arrow is how many times round we go, a couple of dozen typically, and a later section pins the number down. The latent never leaves the loop until the end, which is why nothing that looks like a picture exists until Dec runs, as the pipeline figure in Chapter 4 promised.

One sampler step, term by term

This is the equation the whole chapter is built on. It is the reverse process written as arithmetic: one rung of the ladder, from timestep $t$ down to $t-1$.

One word first. Everything up to this point in the book has been derived in front of you, and this line will not be. It comes out of the same variance bookkeeping you did in Chapter 2, run in reverse, over several pages in the papers that first wrote it down. This chapter takes the line apart term by term instead, then runs it on one real number:

$$\mathbf{z}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{z}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,\boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t)\right) + \sigma_t \mathbf{w}, \qquad \mathbf{w}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$$

In words: take the current noisy latent, subtract a small multiple of the noise the network thinks is on it, divide the whole thing by the square root of alpha-t to undo the shrink the forward process applied, and finally sprinkle a little fresh noise back on. The symbols are the familiar ones. $\beta_t$, beta-t, is how much noise the noise schedule added at timestep $t$. $\alpha_t = 1 - \beta_t$ is how much signal it kept. $\bar{\alpha}_t$, alpha-bar-t, is how much of the original signal survives from the start to timestep $t$. And $\boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t)$ is Chapter 3's noise predictor, its parameters $\theta$ now frozen, because training is over and none of those numbers changes again while we generate. The two new symbols are $\sigma_t$, sigma-t, the sampler's noise scale, and $\mathbf{w}$, a fresh noise draw.

Three steps are packed in there, one paragraph each.

(a) Subtract a fraction of the predicted noise. The network hands us $\boldsymbol{\epsilon}_\theta$, a grid the same shape as the latent, holding its best guess at the noise. We do not subtract all of it. We subtract it scaled by $\beta_t/\sqrt{1-\bar{\alpha}_t}$, a number that is small whenever $\beta_t$ is small. The next section is about why.

(b) Rescale by $1/\sqrt{\alpha_t}$. The forward process shrank the signal by $\sqrt{\alpha_t}$ and then added noise. We have dealt with the noise, so the shrink is what remains to undo, and dividing by $\sqrt{\alpha_t}$ undoes it. That keeps the numbers at the scale the network expects at timestep $t-1$.

(c) Add fresh noise, $\sigma_t\mathbf{w}$. This is the move that surprises everyone, so it gets its own section below. On the last sampler step we set $\sigma_t = 0$ and the term vanishes, because the decoder should get a clean latent rather than a slightly speckled one.

One sampler step, fully worked

Run that equation on a single number. Real latents hold 16,384 of them and every one goes through this same arithmetic independently, so watching one is watching all of them. We round to four significant figures throughout.

Worked example: one sampler step

Our ingredients. The schedule at this timestep says $\beta_t = 0.2$, so $\alpha_t = 1 - 0.2 = 0.8$. The cumulative survival is $\bar{\alpha}_t = 0.5$, so half the original signal strength is left: far enough up the ladder that the latent is messy, far enough down that there is still something to work with. The current latent number is $z_t = 1.2$, and the noise predictor, shown this latent and this timestep, returns $\epsilon_\theta = 0.9$.

1. The denominator. $\sqrt{1-\bar{\alpha}_t} = \sqrt{1 - 0.5} = \sqrt{0.5} = 0.7071$.

2. The removal coefficient. $\dfrac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} = \dfrac{0.2}{0.7071} = 0.2828$.

3. How much noise we take off. $0.2828 \times 0.9 = 0.2546$.

4. The subtraction. $1.2 - 0.2546 = 0.9454$.

5. The rescale. $\sqrt{\alpha_t} = \sqrt{0.8} = 0.8944$, and $\dfrac{0.9454}{0.8944} = 1.0570$.

6. The fresh noise. Nothing so far has fixed $\sigma_t$. The common choice, used here, is $\sigma_t = \sqrt{\beta_t}$: this rung's forward step added noise of variance $\beta_t$ on the way up, so on the way down we put back a helping of the same size and let the next prediction organize it. Here that is $\sqrt{0.2} = 0.4472$. Say the draw comes back $w = -0.3$. Then $0.4472 \times (-0.3) = -0.1342$, and $z_{t-1} = 1.0570 + (-0.1342) = 0.9228$.

The final step instead. If this were the last sampler step we would set $\sigma_t = 0$, the whole term would be $0$, and we would stop at $z_{t-1} = 1.0570$, the same number without the sprinkle.

The one-step update equation drawn large across a dark panel, with each term pinned to its computed value by a thin muted leader line. Under the beta-over-root term sits the number 0.2828 in amber, labeled 'removal coefficient'. Under epsilon-theta sits 0.9 in violet, labeled 'predicted noise'. Under z-t sits 1.2 in mint. A chain of mint boxes beneath reads 1.2 minus 0.2546 equals 0.9454, then divided by 0.8944 equals 1.0570, then plus sigma-t w equals 0.9228. A small gray note reads 'last step: sigma-t equals 0, stop at 1.0570'. — **Every intermediate, pinned to the term that produced it.** Nothing in a sampler step is hidden. It is six pieces of arithmetic, repeated once per latent number per sampler step.

Why we remove only a fraction

Compare what we just did with what Chapter 3 could already do. Chapter 3 rearranged the forward process to estimate the clean latent in one leap:

$$\hat{\mathbf{z}}_0 = \frac{\mathbf{z}_t - \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t)}{\sqrt{\bar{\alpha}_t}}$$

In words: subtract all of the predicted noise, scaled by the full $\sqrt{1-\bar{\alpha}_t}$, then divide by $\sqrt{\bar{\alpha}_t}$ to restore the faded signal. That gives an estimate of the clean latent — right now, in one move.

So why not do that and go home? Put the two coefficients side by side on our numbers. The one-leap version removes $\sqrt{1-\bar{\alpha}_t} = 0.7071$ times the predicted noise. The sampler step removes $\beta_t/\sqrt{1-\bar{\alpha}_t} = 0.2828$ times it. The ratio is $0.2828 / 0.7071 = 0.4$, so the sampler takes off two fifths of what the leap would take off.

That $0.4$ is not a coincidence. It is $\beta_t/(1-\bar{\alpha}_t) = 0.2/0.5 = 0.4$, and both numbers should be read with Chapter 2's eyes. $1-\bar{\alpha}_t$ is the total noise variance the whole climb piled onto the latent, and $\beta_t$ is the part this one rung put there. The fraction is this rung's share of the damage. The sampler undoes that share and leaves the rest for the rungs below it, which is what taking one step down a ladder should mean.

The leap itself, worked, so you can see what we are declining: $0.7071 \times 0.9 = 0.6364$; $1.2 - 0.6364 = 0.5636$; $\sqrt{0.5} = 0.7071$; $0.5636 / 0.7071 = 0.7970$. One move, and we have a candidate clean latent of $0.7970$.

The trouble is what that number means at a high timestep. When the latent is mostly static, a great many different apples could have produced it, and the network was trained to minimize average squared error across all of them. When many answers are plausible, the setting that scores best on average is the average of those answers, which is a blur, and committing to it bakes that blur into the result. Removing two fifths instead leaves the latent noisy, and hands it back to the network at a lower timestep where the field of possible apples has narrowed.

Why we add noise back

Having worked to remove noise, we deliberately put some back: $\sigma_t\mathbf{w}$, with $\mathbf{w}$ a fresh draw from the same standard bell curve. Most readers' first instinct is that this must be a mistake.

The reason it is there is that the update rule in step (a) is deterministic. Feed it the same latent twice and it produces the same output twice, so a purely deterministic loop from a fixed start follows one rigid path, and at high timesteps that path leads toward the average apple: the safe, smooth, slightly waxy apple in the middle of everything the model has learned. The fresh noise nudges the latent off that path and into one specific apple, at every rung, which gives the model room to invent detail the average apple would never have.

There is a second, more mechanical way to say the same thing. The forward process of Chapter 2 threw away information at every step. A reverse step cannot bring that information back, so it re-supplies a matching amount of randomness and lets the next prediction organize it. That is what $\sigma_t\mathbf{w}$ is: the missing information, replaced with something the network can shape. It is why we sized it at $\sqrt{\beta_t}$, matching the amount that rung took away.

It is also why the last step is the exception. Every other rung hands its result back to the network, which tidies up a little jostling on the next pass. The last rung hands its result to the decoder, which cannot. So on that step alone we set $\sigma_t = 0$.

Several well-known samplers, DDIM and Euler and DPM++ among them, make different choices about this noise term and about how to take the step, trading sharpness, speed and variety against one another. The update rule above is the one to understand first. The others are variations on it.

Training steps are not sampler steps

Two different step counts are in play here, and they are not the same quantity measured twice. This is the most-confused pair of numbers in the field.

Note: $T$ and $K$ are different numbers

$T$ is the length of the noise schedule: the fixed ladder of rungs defined once, before training, with a $\beta_t$ value on each. Typically $T = 1000$. It is a property of the model, and nothing you do at generation time changes it. Why a thousand? Because of the assumption in the "Slow down" note above. A reverse step is only a good approximation when each rung is a tiny nudge, and putting a thousand rungs between a clean picture and pure static is what makes every rung tiny.

$K$ is the number of sampler steps: how many of those rungs the sampler actually visits on the way down. Typically $K = 20$ to $50$. With $T = 1000$ and $K = 20$, the sampler strides $1000/20 = 50$ rungs at a time and visits $t = 1000,\, 950,\, 900,\, \ldots,\, 100,\, 50$: twenty timesteps out of a thousand.

Why twenty, and not the full thousand? Cost. Every sampler step is one complete pass of the network over the latent, so a thousand of them takes fifty times as long and produces a picture most people cannot tell apart from the twenty-step one. Striding costs a little accuracy, because each stride is a bigger jump than the approximation was designed for. Twenty to fifty is where that trade sits.

So $K \le T$, always. When a ComfyUI workflow shows a box labeled steps: 20, that number is $K$. "The model was trained for 20 steps" is wrong. "It denoises a thousand times when you generate" is also wrong. Both numbers are real, and they answer different questions.

The bigger distinction is what training and generation each do. Training, from Chapter 3: take a real apple, encode it, pick one random timestep, manufacture the noisy latent with the closed form, ask the network for the noise, compare against the noise we drew ourselves, and nudge $\theta$ downhill. One timestep, no loop, an answer key and a loss.

Generation, this chapter: start from noise that belongs to no apple, walk $t$ downward through $K$ rungs, and never compute a loss, because there is no right answer to compare against. The parameters $\theta$ are frozen, nothing learns and nothing is stored. The network is a fixed function evaluated $K$ times.

Two columns side by side on a dark panel. The left column, headed TRAINING in amber, lists: real apple in, one random timestep t, noise drawn by us as the answer key, one prediction, a loss computed in red, theta updated. The right column, headed SAMPLING in mint, lists: pure noise in, timesteps walked downward from T, no answer key, K predictions, no loss, theta frozen. A muted divider runs between them and a footer note reads 'the same network, used two completely different ways'. — **Same network, two jobs.** Training has an answer key and never runs the loop. Sampling runs the loop and never has an answer key. Most confusion about diffusion clears up once these two columns stay separate in your head.

What emerges, and when

Run the loop and watch the decoded latent along the way, and the apple does not fade in evenly. Broad structure arrives first, a mass here and a background there. Then color settles across those regions. Then edges sharpen, and only in the last few sampler steps do the small things appear: the speck of highlight, the stem, the faint mottling on the skin.

That ordering falls out of the schedule rather than out of any stylistic choice. Early sampler steps sit at high $t$, where $\bar{\alpha}_t$ is tiny and almost nothing of the original signal survives, so the only thing the network can commit to is coarse layout. Late sampler steps sit at low $t$, where $\bar{\alpha}_t$ is close to 1, the latent is nearly clean, and the only remaining work is fine detail. The apple is built the way the forward process tore it down, in reverse order.

A horizontal strip of five square frames on a dark background, all from one generation. The first is pure multicolored static. The second is static with a faint red smudge near the middle. The third shows a red apple shape half buried in the static. The fourth shows a clearer red apple with a stem on a gray wooden surface, with static still over the background. The fifth is a clean photograph of a red apple on a weathered wooden table. Under each frame a mint label reads 20, 15, 10, 5 and 1 sampler steps remaining. — **The mirror image of Chapter 2's noise ladder.** Where Chapter 2 showed one apple dissolving from clean to static, this shows a *different* apple assembling from static to clean, and the two strips run in opposite directions along the same schedule. Each frame is the latent at that point passed through the VAE decoder. Stable Diffusion 1.5, the prompt "a photo of a ripe red apple on a wooden table", seed 42, 20 DDIM sampler steps at guidance scale 7.5.

At $t = 0$ the loop exits with a clean latent, $\mathbf{z}_0$. It is still not an image. It is 16,384 numbers in a learned code, and the last step is the node Chapter 4 named:

$$\hat{\mathbf{x}}_0 = \mathrm{Dec}(\mathbf{z}_0)$$

In words: hand the finished latent to the VAE decoder, and it expands those 16,384 numbers back into $512 \times 512 \times 3 = 786{,}432$ pixel numbers. That is the moment the apple exists as something you can look at.

And it is a new apple, not the apple from Chapter 2 and not any apple in the training data. It is a fresh sample, assembled out of a random grid by a function that learned what apples look like in general.

Where the seed lives

Those "random numbers" arrive from a pseudorandom generator: a deterministic machine that takes one integer, the seed, and produces a stream of numbers that behave like random draws but are reproducible. The sister book's chapter on chance and choice covers what "random" means to a computer if you want the longer version. The chain is short enough to name end to end:

seed $\rightarrow$ pseudorandom stream $\rightarrow$ the starting latent $\mathbf{z}_T$ $\rightarrow$ (through $K$ sampler steps) $\rightarrow$ $\mathbf{z}_0$ $\rightarrow$ $\mathrm{Dec}(\mathbf{z}_0)$ $\rightarrow$ the image.

Everything after the first arrow is fixed arithmetic. Keep the seed, the model, the schedule, $K$ and the sampler identical and you get a pixel-identical apple back, today and next year. That is reproducibility rather than memory. The machine is recomputing the apple, not recalling it. The fresh-noise draws $\mathbf{w}$ come from the same seeded stream, so they are reproducible too, and with $\sigma_t = 0$ throughout there is no randomness after the start at all. Change the seed by one and every one of those 16,384 starting numbers changes, the loop walks a different route, and a different, equally valid apple comes out.

Same model, same settings, different starting static. Nothing about the model changed between these two rows — only the integer that seeded the noise. Variety in diffusion comes from where you start, not from anything stored.

See it move

Watch (18:28): track the removal coefficient in the corner as $t$ falls. It starts tiny, because the sampler barely commits when the latent is mostly static, and the apple's outline is settled long before any of its detail is.

Where you'll meet this

Open any image-generation interface and you are looking at the contents of this chapter, relabeled. The box marked steps is $K$, the number of sampler steps. Set it to 4 and the sampler takes huge, over-committed strides and gives you the smeared result the annealing callout predicted. Set it to 150 and you mostly buy diminishing returns at 150 times the cost, since each sampler step is one full pass through the noise predictor. The box marked seed, with its "randomize" toggle, is the integer at the head of the chain above, and locking it is how people iterate on one apple instead of a new one every time. The dropdown of sampler names, Euler and DDIM and DPM++, is a choice of update rule and of what to do with $\sigma_t$. The VAE Decode node downstream of it all is $\mathrm{Dec}$, which is why the preview stays an abstract smear until the last moment.

Two failure modes you can now diagnose rather than guess at. If your images look soft and averaged, you are probably committing too hard per step, which means too few sampler steps. And if you change "nothing" between two runs and get a different apple, check whether the seed was on randomize. Everything downstream of that integer is deterministic, so a changed image means a changed input somewhere.

What you now know

Generation starts from $\mathbf{z}_T \sim \mathcal{N}(\mathbf{0},\mathbf{I})$, a latent of pure random numbers that belongs to no apple and was retrieved from nowhere.
One sampler step subtracts a fraction $\beta_t/\sqrt{1-\bar{\alpha}_t}$ of the predicted noise, rescales by $1/\sqrt{\alpha_t}$, and adds $\sigma_t\mathbf{w}$ back on: with $\beta_t = 0.2$, $\bar{\alpha}_t = 0.5$, $z_t = 1.2$ and $\epsilon_\theta = 0.9$, that ran $1.2 - 0.2546 = 0.9454$, then $0.9454/0.8944 = 1.0570$.
We remove only $0.2828$ of the predicted noise rather than the full $0.7071$ because a prediction made at a high timestep is an average over every apple that could have produced the latent, and averages are blurry. Many small sampler steps let the network look again.
The fresh noise term is there because without it the loop is deterministic and drifts toward the average apple. It is set to zero only on the final sampler step.
$T$ and $K$ are different numbers. The schedule has $T = 1000$ rungs, a sampler typically visits $K = 20$ to $50$ of them, and the "steps" box in a workflow is always $K$.
The seed picks the starting noise field, not a stored picture. The same seed and settings reproduce the same apple by recomputation, and $\mathrm{Dec}(\mathbf{z}_0)$ turns the finished 16,384-number latent into 786,432 pixel numbers at the end.

Where we're headed. You can now run the whole machine end to end: noise in, a brand-new apple out, every number along the way accounted for. But nothing that entered the loop said what we wanted. The noise predictor took two inputs, $\mathbf{z}_t$ and $t$, and it produced an apple only because apples are what it was trained on. Had it been trained on something else, the same loop would have produced that instead, and we would have had no say in it. The next chapter introduces conditioning, which turns a prompt into a third input the denoiser can follow. Right now we can make an apple, but we cannot ask for a green one.