Making It Move

Every apple we have made stands perfectly still

Every apple we have made stands perfectly still, frozen in a single instant. Now we make it move. Put our apple on the wooden table and ask for two seconds of it slowly turning, so the little dimple by the stem swings from the left edge round toward the middle. That is a video, and a video is not a fundamentally new kind of object — it is image data with one more axis stapled on, the axis we call time.

So here is the honest first attempt, and I want us to take it seriously rather than skip it, because its failure is the whole chapter. We already know how to make one apple. Run Chapter 5's sampler sixteen times with the same prompt and the same weights, and we have sixteen apples. Lay them end to end at eight frames per second and play them. What do we get?

We get flicker: an apple that jumps between varieties, sizes, and shadows eight times a second. And the precise diagnosis matters, so let me state it carefully. Not one of those frames is wrong. Each one is a clean, believable apple — a fresh sample that landed squarely inside the data distribution, the apple neighborhood from Chapter 1. The failure is not quality. The failure is that the frames are independent samples, and independence is exactly what motion is not. Motion means frame 2 is a small, structured change to frame 1 — the same fruit, the same light, one instant later. Sixteen independent draws have no mechanism for "same".

One block, with a time axis

The fix follows straight from the diagnosis. If independence is the disease, stop making independent things. We stop treating a video as $F$ separate images and start treating it as a single object with a time axis: a spatiotemporal latent block, written $\mathbf{Z}$ — capital Z, because it is a stack, not a single frame's latent. One noise blob is drawn for the whole block. One noise predictor looks at the whole block. One sampler walks the whole block down to zero together.

Let me take that name apart, because it sounds worse than it is. Spatio means space, the width and height we already had. Temporal means time, the axis we are adding. Latent is Chapter 4's word for the compressed summary of a picture. And block means all of them stacked into one array.

Let us count what that block holds, because the arithmetic is the entire story of why video is expensive. From Chapter 4, a latent for a single $512\times512$ image, at the typical Stable-Diffusion-style setting of downscale factor $f = 8$ and 4 channels, is a 64-by-64 grid of 4 channels: 16,384 numbers per frame, counted out in that chapter.

One small warning about letters before we go on: in Chapter 4, $f$ was the downscale factor 8; in this chapter $f$ is a frame index, and $F$ is the total number of frames. The field reuses the letter, and we are stuck with it — from here on, $f$ counts frames.

Now stack $F = 16$ frames, an illustrative length that at 8 frames per second buys us two seconds of turning apple:

In words: sixteen frames, each of 16,384 latent numbers, is 262,144 numbers in the block — sixteen times as many as one still image.

That factor of 16 is the plain cost of video, and it is not hiding anywhere: every sampler step now denoises sixteen frames' worth of numbers instead of one. It is worth seeing what the compression trick is buying us here, because it buys the same 48× it always did: the same sixteen frames in raw pixels would be 12,582,912 numbers rather than 262,144 — the same 48-fold saving Chapter 4 found for a single image, unchanged by adding frames.

With the block defined, the forward process — adding noise on purpose — needs no new mathematics at all. Chapter 2's closed form carries over exactly, with $\mathbf{Z}$ in place of $\mathbf{z}$:

In words: the noisy block at noise level $t$ is the clean block faded by $\sqrt{\bar{\alpha}_t}$ — alpha-bar-t, the cumulative fraction of signal still kept by step $t$ — plus one blob of noise $\boldsymbol{\epsilon}$ scaled by the rest.

Two things about that line deserve saying out loud. First, $\boldsymbol{\epsilon}$ is now a blob of 262,144 random numbers drawn in one go for the entire block, not sixteen separate blobs — the block is noised as a unit. Second, $\bar{\alpha}_t$ is a single number, the same for every frame, because the noise schedule is a property of the timestep $t$, not of position or frame. All sixteen frames sit at the same noise level at all times. That is what lets them be denoised together.

Letting the frames see each other

Bundling the frames into one array is necessary but not sufficient. If the noise predictor still processed each frame in its own private lane, never comparing them, we would have rebuilt independent sampling with extra steps. Something inside the network has to let frame 2 look at frame 1.

Let us watch why that helps on the smallest possible video: three frames, one pixel each, and take the numbers as the brightness of that pixel after the reverse process — the denoising process — has finished. Denoised independently, the three frames come out as $0.81$, $0.44$, $0.77$ — picked so the middle frame is visibly darker than its neighbors, a flash you could actually see. Each value is fine on its own. The trouble is in the differences:

In words: the pixel jumps down by 0.37 from frame 1 to frame 2, then back up by 0.33 from frame 2 to frame 3. On screen, that is a flash.

Now let each frame blend itself with its neighbors instead of standing alone. Give the frame itself half the weight and each neighbor a quarter — weights $[0.25,\, 0.5,\, 0.25]$ — with the edge frames, which have only one neighbor, reusing themselves in the missing slot. Those three numbers are chosen for a reason: they add up to exactly one, so the blend can smooth the sequence but can never brighten or darken it overall. Every multiplication, printed:

The sequence still moves — 0.7175, 0.615, 0.6875 is not a flat line, and nothing here forbids change. What it forbids is unstructured change. That is the whole trick of temporal coherence: leave a channel open through which each frame's answer is pulled toward what its neighbors are doing.

Where the weights come from

We hand-picked $[0.25,\, 0.5,\, 0.25]$. A real video model learns them, per patch and per moment, using machinery we already own: attention, from Chapter 6 and, in its original form, from the LLM book's attention chapter. In Chapter 6 an image patch formed a query and aimed it at the prompt's keys. Here the axis changes and nothing else does: the patch at a given position in frame $f$ forms a query $\mathbf{q}^{(f)}$, and aims it at the keys $\mathbf{k}^{(1)}, \dots, \mathbf{k}^{(F)}$ belonging to the same patch position in every frame. That is the amber column in the block figure above. Because it looks along time rather than across the image, this is called temporal attention, and the block of network layers that houses it is usually called a motion module.

Work one softmax to see the weights appear. Say our patch in frame 2 scores its three candidates — frames 1, 2, 3 — at $[1,\, 2,\, 1]$: it matches itself best, and its two neighbors equally and less well. Exponentiate each score — that is, raise $e$, the fixed constant near $2.718$ we met in Chapter 6, to the power of each score — rounding to four significant figures as we have throughout the book:

In words: raising $e$ to each score turns the scores into positive numbers that grow fast, so the winner pulls away from the others.

In words: add the three exponentials to get the total, 12.825, which is what we divide by.

In words: dividing each exponential by the total gives three shares — 0.212, 0.576, 0.212 — which add to 1.000 and say how much of each frame this patch takes.

Now spend them the way we spent the hand-picked weights. Frame 2's three candidates carry $0.81$, $0.44$ and $0.77$, so the blended value for frame 2 is:

In words: multiply each frame's value by the share the softmax gave it and add the three products; frame 2 comes out at 0.5884, against the 0.615 the hand-picked weights produced.

One small piece of honesty about that total. Every number on screen is rounded to four figures, so adding the three rounded pieces on a calculator lands a hair under the figure I just gave you. The number printed there is the true sum, correctly rounded, and that difference is the rounding.

Those three shares play exactly the role our hand-picked $[0.25,\, 0.5,\, 0.25]$ played, and $0.5884$ against $0.615$ is how close that role is: the same smoothing, arrived at by the network instead of chosen by us. The difference is that the network computes them fresh from the content: on a patch of still tablecloth the shares spread out and the frames lock together hard, while on the fast-moving edge of the turning apple the patch can concentrate its share on itself and let that region change quickly. Coherence stops being a fixed rule and becomes something decided per patch, per frame.

Starting from a picture we already have

Everything so far generates a video from nothing but noise and a prompt. Far more often, what we actually want is this: here is my apple, this exact photograph on this exact table — now make it turn. That is image-to-video, and the block gives us a clean way to say it.

Encode the starting picture with the VAE encoder from Chapter 4, and hold the result fixed as the block's first frame:

In words: frame 1 of the clean block is not something the sampler gets to invent — it is the encoded starting image, pinned in place, while the sampler is left to fill in the other fifteen frames around it.

The reason this is easy rather than awkward is that we know frame 1's value at every noise level, not only at $t = 0$. Chapter 2's closed form hands it to us directly: $\sqrt{\bar{\alpha}_t}\,\mathbf{z}^{(1)}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}$ is the correctly-noised version of our known frame at level $t$, computable in one line. So at each sampler step the common approach re-imposes the constraint: run Chapter 5's update rule on the whole block — unchanged, there is no new sampler mathematics here — and then set frame 1's slot back to the value we know it must have.

And now temporal attention earns its keep twice over. Frame 2's patches are attending to frame 1's patches, and frame 1's patches are not a guess — they are our photograph. That fixed anchor propagates: frame 2 is pulled toward frame 1, frame 3 toward frame 2, and the apple's identity — its exact color, its blemish, the grain of the table under it — is carried down the strip instead of being re-invented sixteen times. Pin one frame, and coherence spreads the pin.

See it move

Where you'll meet this

Open any video workflow and the shapes from this chapter are printed on the wires. Where an image workflow has an Empty Latent Image node with width and height, a video workflow has one with width, height, and a frame count — that third box is $F$, and it is the number that multiplied our 16,384 up to 262,144. When it turns out that a clip is too long to fit in memory, this is the arithmetic that explains it: doubling the frame count doubles the block, and every sampler step pays.

The loading of a separate motion module on top of an image model is the second place. That file is not a new U-Net and it does not replace the base model any more than a LoRA does — it supplies the temporal layers that let the frames look at each other, and it is inert without the image model underneath it. You will also see an image-to-video node with a slot for a starting picture; that slot is $\mathbf{z}^{(1)}_0$, the pinned first frame.

Finally, the failure modes now have names. A clip whose subject morphs identity every few frames is losing the coherence channel — the frames are drifting back toward independence. A clip that is beautifully stable and utterly motionless has the opposite problem: the temporal blend is dominating, every frame is being dragged too hard toward its neighbors, and the video has collapsed into a still. Those are the two extremes the "slow down" callout above put numbers on, and good video generation lives in the band between them.

Every arrow, explained

We began this book with one picture: an apple, an arrow to a field of static, and an arrow back out to a different apple. I promised then that every arrow in it was a piece we would unpack. Let us go back and look at it now, because the picture has not changed at all — only you have.

Read it left to right. The apple on the left is $\mathbf{x}_0$, and Chapter 1 taught us that it is not really a picture at all but a single point in a space of 786,432 numbers, sitting inside a tight cloud of other apple points. Generating meant landing a fresh point inside that cloud, which was hopeless as a single blind leap.

The first arrow is the forward process, and Chapter 2 built it: fade the apple by $\sqrt{\bar{\alpha}_t}$, add noise scaled by $\sqrt{1-\bar{\alpha}_t}$, and you can reach any noise level in one shot. The static in the middle panel is the end of that road — and also, read the other way, the beginning of ours, because Chapter 5 showed that the pure noise we start a generation from comes from a seed and belongs to no apple at all.

The second arrow — the long one, the one that does the real work — took four chapters. Chapter 3 trained $\boldsymbol{\epsilon}_\theta$ to look at a noisy input and name the noise inside it, and showed why that is the learnable question: we make the training data ourselves, so we always know the answer. Chapter 4 moved the whole apparatus into latent space, cutting 786,432 numbers to 16,384 and leaving every equation otherwise untouched. Chapter 5 turned the trained predictor into the sampler, walking $K$ sampler steps down the schedule, removing a fraction of the predicted noise each time. Chapter 6 gave that walk a steering wheel, turning "a green apple on a wooden table" into conditioning $\mathbf{c}$ and pushing the prediction away from the generic answer by the guidance scale $s$. Chapter 7 taught the frozen model one new thing with a small pair of skinny matrices. And this chapter gave the whole object a time axis.

That is the entire book, and it is the entire diagram. Two arrows.