LoRAs and Adapters

The apple the prompt cannot name

Chapter 6 ended on a hard edge. Prompts steer the model within what it already knows, and teaching it something new is a different job. That is this chapter.

Say there is a particular apple you care about: an heirloom variety with a russet shoulder and a squat, lopsided profile. You type it into the prompt and you get an apple, a good one, and not that one. No arrangement of words reaches it, because the words the noise predictor was trained against never pointed at your apple.

The obvious fix is to keep training. Take the finished model, feed it thirty photographs of your apple, and let Chapter 3's gradient steps keep rolling. That is full fine-tuning, and it works. It also hands you a bill. A model of this kind holds billions of learned numbers, the parameters $\theta$ from Chapter 3, counting the hundreds-of-millions U-Net together with the other trained pieces packed into the same checkpoint. Full fine-tuning nudges every one of them, each needing its own gradient and its own bookkeeping while training runs, and what you get back is a second copy of the model: gigabytes, for one apple.

There is a much smaller way to write down a change, and it is the rest of this chapter: a LoRA, short for Low-Rank Adaptation. It leaves the model's own billions of numbers frozen, in Chapter 5's sense of the word: the training process is forbidden to change those numbers at all. They are read and multiplied by, and never touched. That is what makes everything else cheap. The same idea, with the same numbers, appears in the sister book's LoRA chapter, so the two books agree digit for digit. This chapter builds it from scratch anyway, then does the part that belongs to images.

Learning only the change

Inside the noise predictor, the numbers that do the work are arranged in rectangular grids called weight matrices. Pick one and call it $\mathbf{W}$, "the matrix W", a block of numbers the network multiplies things by. After a fine-tune it has become some new matrix $\mathbf{W}'$, "W-prime", and whatever fine-tuning did to it, the result can be written as the original plus a grid of changes. Write $\Delta\mathbf{W}$ for that grid of changes, using the Greek letter delta, the traditional symbol for "a change":

In words: the adapted matrix is the original matrix plus a matrix of changes, added cell by matching cell.

On its own that rewrite saves nothing. $\Delta\mathbf{W}$ has as many cells as $\mathbf{W}$, so a million-cell matrix still needs a million-cell correction. It only pays if the change can be described more cheaply than the thing it changes. So how complicated is the change a narrow fine-tune needs?

The LoRA bet is that it is very simple, and there is a precise way to say "simple" about a matrix. A matrix built by multiplying one tall skinny column by one flat wide row can only push its outputs along a single direction. It has rank one, the simplest non-zero change there is. Take a column $\mathbf{B}$ and a row $\mathbf{A}$:

In words: the whole grid of changes is one column of numbers standing up, multiplied by one row of numbers lying flat.

That bottleneck width is the knob. Let $r$ numbers through instead of one, where $r$ is the rank, and $\mathbf{B}$ becomes a matrix with $r$ columns, $\mathbf{A}$ a matrix with $r$ rows, and the change can span up to $r$ independent directions. In practice $r$ is small: 4, 8, 16 or 32, against matrices a thousand numbers wide. Small works because the change a narrow fine-tune needs is simple. You are not teaching the model to see. You are teaching it one apple. Everything else about training is unchanged. The loss is Chapter 3's, in the latent, conditioned form Chapters 4 and 6 left it in, $L(\theta) = \|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t, \mathbf{c})\|^2$, with the gradient allowed to move only $\mathbf{B}$ and $\mathbf{A}$.

The strength dial, as arithmetic

A LoRA ships with one more number: a scale that says how loudly the learned change should speak. In the update it is the fraction $\frac{\alpha}{r}$:

In words: the working matrix is the frozen matrix plus alpha-over-r times the learned low-rank change.

$r$ is the rank, the bottleneck width we just chose, and $\alpha$ is "alpha, the LoRA scale", a volume knob. Dividing by $r$ is a courtesy to your future self: double the rank and the product $\mathbf{B}\mathbf{A}$ picks up roughly twice as many contributions, so dividing by $r$ keeps the overall loudness comparable when you change rank.

Now the number you turn. In a workflow it is labeled LoRA strength, and it is the multiplier in front of $\mathbf{B}\mathbf{A}$. Sweep it across the same input $[1, 1]$, remembering that $\mathbf{B}\mathbf{A}\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 8 \\ 4 \end{bmatrix}$ from the worked example above, and the base path contributes $[2, 2]$ every time:

Where the adapter attaches

Which matrices get the treatment? Almost always the ones inside the cross-attention blocks of the U-Net, from Chapter 6, where each patch of the latent asks the conditioning $\mathbf{c}$ what it should be. Those are the matrices that decide how strongly a word in your prompt pulls on a region of the image, so a small correction there is a large change in how the model reads language.

That explains trigger words. Whoever trains an adapter captions the training images with a rare token, "sks apple" or something else the base model has no strong opinion about, and the adapter learns to attach a large correction to that token's pull, because there was nothing there before to interfere with. Leave the trigger word out of your prompt and such an adapter can look inert. Put it in and the whole thing switches on. You are choosing whether the corrected path ever gets a strong query to answer.

Everything else is as you left it. Picture Chapter 4's latent-diffusion-pipeline map again: encoder, loop in latent space, decoder. A LoRA changes nothing about it, not the VAE, not the sampler, and no extra stage. It edits a handful of matrices inside the noise predictor, and Chapter 5's loop runs as before, calling a slightly different function at every sampler step.

Why varied apples matter

Back in Chapter 1 we laid out nine deliberately different apple thumbnails, red and green and golden, spotted, large, small, photographed from two angles, and the caption promised that the difference would matter when we got to LoRAs. This is where that gets paid.

Training an adapter is Chapter 3's procedure on a very small dataset. Encode one of your photographs to a latent, pick a random timestep $t$, add the matching noise using Chapter 2's closed form, ask the noise predictor for that noise, compare, and push $\mathbf{B}$ and $\mathbf{A}$ downhill. Repeat over your twenty or thirty images. The learned change is whatever consistently reduces that loss across your set.

That last sentence contains the trap. The adapter learns whatever is consistent, and it has no idea which parts of your photographs are the concept and which are the circumstances. Say all thirty photographs of your heirloom apple sit on the same wooden table, in the same warm afternoon light, shot from the same height. "Russet shoulder" is consistent across your set, and so is "wooden table", and "warm light", and "shot from above". The gradient cannot tell them apart, so the adapter learns the whole bundle as one concept. Load it, prompt for your apple in a snowfield, and the wooden table comes to the snowfield with it. That is the method working as specified on data that specified the wrong thing.

The fix is variety, and you apply it with a camera rather than a setting. Photograph the apple on a table, on grass, in a hand, against a white wall; morning light, lamp light, flat overcast; close, far, above, side. Now the only thing consistent across the set is the apple, so the only thing the adapter can learn is the apple. Variety isolates the concept from its accidents, and twenty varied photographs beat two hundred from one sitting.

Two neighboring failures deserve names. Overfitting is training for too many passes over too few images: the adapter stops learning "this apple" and starts memorizing "these thirty pictures", and the giveaway is a generation that reproduces a training photo's composition whatever else you ask for. And data quality beats quantity: five sharp, well-lit, correctly captioned photographs teach a cleaner concept than fifty blurry ones, because every blurry image quietly teaches "blurry" as part of the target.

Counting the savings

Now the arithmetic that made the technique famous. Take a weight matrix that is $1{,}000$ numbers wide and $1{,}000$ numbers tall, a typical size inside a model like this. A full change $\Delta\mathbf{W}$ to it has one entry per cell:

In words: changing that matrix the ordinary way means writing down and training one million numbers.

Now the low-rank version at a typical rank of $r = 8$. The column block $\mathbf{B}$ is $1{,}000$ tall and $8$ wide; the row block $\mathbf{A}$ is $8$ tall and $1{,}000$ wide:

In words: sixteen thousand trained numbers stand in for a million.

The ratio, worked: $16{,}000 \div 1{,}000{,}000 = 0.016$, which is $1.6\%$. Multiply that across the handful of cross-attention matrices a LoRA touches and the file lands in the low tens of megabytes, beside a base model of several gigabytes. Those figures are typical of models of this kind rather than laws. It is why a shelf of adapters is practical to own: one frozen base model and a stack of small files, each teaching it one thing.

Because the corrections are additions, they compose. Load two adapters and the corrections add up on each shared matrix, each with its own strength:

In words: the adapted matrix is the frozen matrix plus the first adapter's change at its own strength, plus the second adapter's change at its own strength. These strengths $s_1$ and $s_2$ are LoRA strengths, not Chapter 6's guidance scale $s$, which lives in a different equation.

Composition works right up until it does not. Each adapter was trained separately, assuming the frozen matrix underneath it was untouched and knowing nothing of the other. If they push the same matrices in conflicting directions, say a style adapter that flattens everything against a subject adapter that insists on photographic texture, the sum is a fight rather than a compromise, and the usual repair is turning both strengths down.

See it move

Where you'll meet this

In a workflow graph, this chapter is one node, usually called Load LoRA, sitting after the checkpoint loader, taking the base model in on one wire and handing a modified model out on another. That wiring is the picture of the chapter: the adapter never stands alone in the graph, because it has nothing to do without a model flowing through it.

The number in that node's box is $\frac{\alpha}{r}$ from the sweep table, the same multiplier with the same arithmetic behind it. Some interfaces give you two of them, one for the correction applied inside the U-Net and one for the correction applied to the text encoder, because a LoRA can carry strips for both. And when a model page tells you to include a strange token like "sks apple" in your prompt, you know what you are doing: handing the corrected cross-attention path the one query it was trained to answer loudly.

Three habits fall out of the math. If a generation looks untouched, check the trigger word first. If it looks scorched, drop the strength before you rewrite the prompt, because you are turning down the multiplier in front of $\mathbf{B}\mathbf{A}$ and nothing more. And if a LoRA does nothing at all, check which base model it was trained against. LoRA is the famous member of a larger family of adaptation methods, and the others are a topic for another day.