Making It Move

Every apple so far stands still

Every apple we have made stands perfectly still, frozen in a single instant. Now we make it move. Put our apple on the wooden table and ask for two seconds of it slowly turning, so the dimple by the stem swings from the left edge round toward the middle. That is a video, and a video is image data with one more axis added: time.

Start with the first attempt anybody tries, because its failure is the whole chapter. We already know how to make one apple. Run Chapter 5's sampler sixteen times with the same prompt and the same weights, and we have sixteen apples. Lay them end to end at eight frames per second and play them. What do we get?

We get flicker: an apple that jumps between varieties, sizes and shadows eight times a second. The diagnosis matters, so state it carefully. Not one of those frames is wrong. Each is a clean, believable apple, a fresh sample that landed inside the data distribution, the apple neighborhood from Chapter 1. The failure is that the frames are independent samples, and motion is the opposite of independence. Motion means frame 2 is a small, structured change to frame 1: the same fruit, the same light, one instant later. Sixteen independent draws have no mechanism for "same".

One block, with a time axis

The fix follows from the diagnosis: stop making independent things. We stop treating a video as $F$ separate images and start treating it as a single object with a time axis, a spatiotemporal latent block, written $\mathbf{Z}$ — capital Z, because it is a stack, not a single frame's latent. One noise blob is drawn for the whole block. One noise predictor looks at the whole block. One sampler walks the whole block down to zero together.

Take that name apart. Spatio means space, the width and height we already had. Temporal means time, the axis we are adding. Latent is Chapter 4's word for the compressed summary of a picture. And block means all of them stacked into one array.

Count what that block holds, because the arithmetic is the whole story of why video is expensive. From Chapter 4, a latent for a single $512\times512$ image, at the typical Stable-Diffusion-style setting of downscale factor $f = 8$ and 4 channels, is a 64-by-64 grid of 4 channels: 16,384 numbers per frame, counted out in that chapter.

A warning about letters first. In Chapter 4, $f$ was the downscale factor 8. In this chapter $f$ is a frame index and $F$ is the total number of frames. The field reuses the letter and we are stuck with it, so from here on $f$ counts frames.

Now stack $F = 16$ frames, an illustrative length that at 8 frames per second buys two seconds of turning apple:

In words: sixteen frames, each of 16,384 latent numbers, is 262,144 numbers in the block, sixteen times as many as one still image.

That factor of 16 is the cost of video, and it is not hidden anywhere. Every sampler step now denoises sixteen frames' worth of numbers instead of one. The compression trick still buys the same 48× it always did: the same sixteen frames in raw pixels would be 12,582,912 numbers rather than 262,144, the same 48-fold saving Chapter 4 found for a single image, unchanged by adding frames.

With the block defined, the forward process needs no new mathematics. Chapter 2's closed form carries over, with $\mathbf{Z}$ in place of $\mathbf{z}$:

In words: the noisy block at noise level $t$ is the clean block faded by $\sqrt{\bar{\alpha}_t}$, alpha-bar-t, the cumulative fraction of signal still kept by step $t$, plus one blob of noise $\boldsymbol{\epsilon}$ scaled by the rest.

Two things about that line. First, $\boldsymbol{\epsilon}$ is now a blob of 262,144 random numbers drawn in one go for the entire block, not sixteen separate blobs. The block is noised as a unit. Second, $\bar{\alpha}_t$ is a single number, the same for every frame, because the noise schedule is a property of the timestep $t$, not of position or frame. All sixteen frames sit at the same noise level at all times, which is what lets them be denoised together.

Letting the frames see each other

Bundling the frames into one array is necessary but not sufficient. If the noise predictor still processed each frame in its own private lane, never comparing them, we would have rebuilt independent sampling with extra steps. Something inside the network has to let frame 2 look at frame 1.

Watch why that helps on the smallest possible video: three frames, one pixel each, with the numbers as the brightness of that pixel after the reverse process has finished. Denoised independently, the three frames come out as $0.81$, $0.44$, $0.77$, picked so the middle frame is visibly darker than its neighbors, a flash you could see. Each value is fine on its own. The trouble is in the differences:

In words: the pixel jumps down by 0.37 from frame 1 to frame 2, then back up by 0.33 from frame 2 to frame 3. On screen, that is a flash.

Now let each frame blend itself with its neighbors instead of standing alone. Give the frame itself half the weight and each neighbor a quarter, so the weights are $[0.25,\, 0.5,\, 0.25]$, with the edge frames reusing themselves in the missing slot. Those three numbers add up to one, so the blend can smooth the sequence but can never brighten or darken it overall. Every multiplication, printed:

The sequence still moves. 0.7175, 0.615, 0.6875 is not a flat line, and nothing here forbids change. What it forbids is unstructured change. That is the trick of temporal coherence: leave a channel open through which each frame's answer is pulled toward what its neighbors are doing.

Where the weights come from

We hand-picked $[0.25,\, 0.5,\, 0.25]$. A real video model learns them, per patch and per moment, using attention from Chapter 6 and, in its original form, from the LLM book's attention chapter. In Chapter 6 an image patch formed a query and aimed it at the prompt's keys. Here only the axis changes. The patch at a given position in frame $f$ forms a query $\mathbf{q}^{(f)}$ and aims it at the keys $\mathbf{k}^{(1)}, \dots, \mathbf{k}^{(F)}$ belonging to the same patch position in every frame, the amber column in the block figure above. Because it looks along time rather than across the image, this is temporal attention, and the block of network layers that houses it is usually called a motion module.

Work one softmax to see the weights appear. Say our patch in frame 2 scores its three candidates, frames 1, 2 and 3, at $[1,\, 2,\, 1]$: it matches itself best and its two neighbors equally and less well. Exponentiate each score, raising $e$, the fixed constant near $2.718$ from Chapter 6, to the power of each score, rounding to four significant figures as throughout the book:

In words: raising $e$ to each score turns the scores into positive numbers that grow fast, so the winner pulls away from the others.

In words: add the three exponentials to get the total, 12.825, which is what we divide by.

In words: dividing each exponential by the total gives three shares, 0.212, 0.576 and 0.212, which add to 1.000 and say how much of each frame this patch takes.

Now spend them the way we spent the hand-picked weights. Frame 2's three candidates carry $0.81$, $0.44$ and $0.77$, so the blended value for frame 2 is:

In words: multiply each frame's value by the share the softmax gave it and add the three products. Frame 2 comes out at 0.5884, against the 0.615 the hand-picked weights produced.

One note on that total. Every number on screen is rounded to four figures, so adding the three rounded pieces on a calculator lands a hair under the figure printed there. The printed number is the true sum, correctly rounded.

Those three shares play the role our hand-picked $[0.25,\, 0.5,\, 0.25]$ played, and $0.5884$ against $0.615$ shows how close: the same smoothing, arrived at by the network instead of chosen by us. The difference is that the network computes them fresh from the content. On a patch of still tablecloth the shares spread out and the frames lock together hard, while on the fast-moving edge of the turning apple the patch can concentrate its share on itself and let that region change quickly. Coherence stops being a fixed rule and becomes something decided per patch, per frame.

Starting from a picture we already have

Everything so far generates a video from nothing but noise and a prompt. More often what we want is: here is my apple, this photograph on this table, now make it turn. That is image-to-video, and the block gives us a clean way to say it.

Encode the starting picture with the VAE encoder from Chapter 4, and hold the result fixed as the block's first frame:

In words: frame 1 of the clean block is not something the sampler invents. It is the encoded starting image, pinned in place, while the sampler fills in the other fifteen frames around it.

This is easy rather than awkward because we know frame 1's value at every noise level, not only at $t = 0$. Chapter 2's closed form hands it to us: $\sqrt{\bar{\alpha}_t}\,\mathbf{z}^{(1)}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}$ is the correctly-noised version of our known frame at level $t$, computable in one line. So at each sampler step the common approach re-imposes the constraint. Run Chapter 5's update rule on the whole block, with no new sampler mathematics, and then set frame 1's slot back to the value we know it must have.

Now temporal attention earns its keep twice over. Frame 2's patches are attending to frame 1's patches, and frame 1's patches are not a guess. They are our photograph. That anchor propagates: frame 2 is pulled toward frame 1, frame 3 toward frame 2, and the apple's identity, its color, its blemish, the grain of the table under it, is carried down the strip instead of being re-invented sixteen times.

See it move

Where you'll meet this

Open any video workflow and the shapes from this chapter are printed on the wires. Where an image workflow has an Empty Latent Image node with width and height, a video workflow has one with width, height and a frame count. That third box is $F$, the number that multiplied our 16,384 up to 262,144. When a clip turns out to be too long to fit in memory, this is the arithmetic that explains it: doubling the frame count doubles the block, and every sampler step pays.

Loading a separate motion module on top of an image model is the second place. That file is not a new U-Net, and it does not replace the base model any more than a LoRA does. It supplies the temporal layers that let the frames look at each other, and it is inert without the image model underneath it. You will also see an image-to-video node with a slot for a starting picture. That slot is $\mathbf{z}^{(1)}_0$, the pinned first frame.

The failure modes now have names. A clip whose subject changes identity every few frames is losing the coherence channel, with the frames drifting back toward independence. A clip that is stable and motionless has the opposite problem: the temporal blend is dominating, every frame is being dragged too hard toward its neighbors, and the video has collapsed into a still. Those are the two extremes the "slow down" callout above put numbers on, and good video generation lives in the band between them.

Every arrow, explained

We began this book with one picture: an apple, an arrow to a field of static, and an arrow back out to a different apple. Every arrow in it was a piece this book would unpack. Look at it again. The picture has not changed at all.

Read it left to right. The apple on the left is $\mathbf{x}_0$, and Chapter 1 taught us that it is a single point in a space of 786,432 numbers, sitting inside a tight cloud of other apple points. Generating meant landing a fresh point inside that cloud, which was hopeless as a single blind leap.

The first arrow is the forward process, and Chapter 2 built it: fade the apple by $\sqrt{\bar{\alpha}_t}$, add noise scaled by $\sqrt{1-\bar{\alpha}_t}$, and you can reach any noise level in one shot. The static in the middle panel is the end of that road. Read the other way it is the beginning of ours, because Chapter 5 showed that the pure noise we start a generation from comes from a seed and belongs to no apple at all.

The second arrow, the long one that does the real work, took four chapters. Chapter 3 trained $\boldsymbol{\epsilon}_\theta$ to look at a noisy input and name the noise inside it, and showed why that is the learnable question: we make the training data ourselves, so we always know the answer. Chapter 4 moved the whole thing into latent space, cutting 786,432 numbers to 16,384 and leaving every equation otherwise untouched. Chapter 5 turned the trained predictor into the sampler, walking $K$ sampler steps down the schedule and removing a fraction of the predicted noise each time. Chapter 6 gave that walk a way to be aimed, turning "a green apple on a wooden table" into conditioning $\mathbf{c}$ and pushing the prediction away from the generic answer by the guidance scale $s$. Chapter 7 taught the frozen model one new thing with a small pair of skinny matrices. And this chapter gave the whole object a time axis.