LoRAs and Adapters

The apple the prompt cannot name

Chapter 6 ended on a hard edge: the model will follow any prompt it already understands, and not one it has never seen. Prompts steer the model within what it knows. Teaching it something new is a different job — and that is this chapter.

Here is the situation, in apples. There is a particular apple you care about: an heirloom variety with a russet shoulder and a squat, lopsided profile. You type it into the prompt. You get an apple — a good one, a plausible one, and not that one. No arrangement of words reaches it, because the words the noise predictor was trained against never pointed at your apple in the first place.

The obvious fix is to keep training. Take the finished model, feed it thirty photographs of your apple, and let Chapter 3's gradient steps keep rolling. That is full fine-tuning, and it works. It also hands you a bill. A model of this kind typically holds billions of learned numbers — the parameters $\theta$ from Chapter 3, whose hundreds-of-millions U-Net is counted here together with the other trained pieces packed into the same checkpoint — and full fine-tuning nudges every one of them, each needing its own gradient worked out and its own bookkeeping kept while training runs. What you get back at the end is an entire second copy of the model: gigabytes, for one apple.

There is a much smaller way to write down a change, and it is the subject of the rest of this chapter: a LoRA, short for Low-Rank Adaptation. It leaves the model's own billions of numbers frozen, in exactly Chapter 5's sense of the word: the training process is forbidden to change those numbers at all. They are read, they are multiplied by, and they are never touched. That one sentence is what makes everything else cheap. The same idea, with the same numbers, appears in the sister book's LoRA chapter — deliberately, so the two books agree digit for digit. We will build it from scratch here anyway, and then do the part that belongs to images.

Learn the change, not the weights

Inside the noise predictor, the numbers that do the work are arranged in rectangular grids called weight matrices. Pick any one of them and call it $\mathbf{W}$ — "the matrix W", a block of numbers the network multiplies things by. After a fine-tune it has become some new matrix $\mathbf{W}'$ — "W-prime" — and whatever fine-tuning did to it, we can always write the result as the original plus a grid of changes. Writing $\Delta\mathbf{W}$ for that grid of changes (the Greek letter delta, the traditional symbol for "a change"):

In words: the adapted matrix is the original matrix plus a matrix of changes, added cell by matching cell.

On its own that reframe saves nothing, and it would be dishonest to pretend otherwise. $\Delta\mathbf{W}$ has exactly as many cells as $\mathbf{W}$; a million-cell matrix still needs a million-cell correction. The reframe only pays if the change can be described more cheaply than the thing it changes. So: how complicated is the change a narrow fine-tune actually needs?

The LoRA bet is that it is very simple, and there is a precise way to say "simple" about a matrix. A matrix built by multiplying one tall skinny column by one flat wide row can only ever push its outputs along a single direction — it has rank one, the simplest non-zero change there is. Watch it happen with the smallest example that shows anything. Take a column $\mathbf{B}$ and a row $\mathbf{A}$:

In words: the whole grid of changes is one column of numbers standing up, multiplied by one row of numbers lying flat.

That bottleneck width is the knob. Let $r$ numbers through instead of one — "r, the rank" — and $\mathbf{B}$ becomes a matrix with $r$ columns, $\mathbf{A}$ a matrix with $r$ rows, and the change can span up to $r$ independent directions. In practice $r$ is small: 4, 8, 16, 32, against matrices a thousand numbers wide. Small works because the change a narrow fine-tune needs really is simple. You are not teaching the model to see. You are teaching it one apple. Everything else about training is unchanged — the loss is Chapter 3's, in the latent, conditioned form Chapters 4 and 6 left it in, $L(\theta) = \|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t, \mathbf{c})\|^2$, with the gradient allowed to move only $\mathbf{B}$ and $\mathbf{A}$.

The strength dial, as arithmetic

A LoRA ships with one more number attached: a scale that says how loudly the learned change should speak. Written into the update, it is the fraction $\frac{\alpha}{r}$:

In words: the working matrix is the frozen matrix plus alpha-over-r times the learned low-rank change.

Two symbols, both re-introduced in words. $r$ is the rank — the bottleneck width we just chose. And $\alpha$ is "alpha, the LoRA scale", a volume knob. Dividing by $r$ is a courtesy to your future self: double the rank and the product $\mathbf{B}\mathbf{A}$ picks up roughly twice as many contributions, so dividing by $r$ keeps the overall loudness comparable when you change rank.

Now the number the user actually turns. In a workflow it is labeled LoRA strength, and mathematically it is nothing more exotic than the multiplier in front of $\mathbf{B}\mathbf{A}$. Sweep it across the same input $[1, 1]$, remembering that $\mathbf{B}\mathbf{A}\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 8 \\ 4 \end{bmatrix}$ from the worked example above, and the base path contributes $[2, 2]$ every time:

Where the adapter attaches

Which matrices get the treatment? Almost always the ones inside the cross-attention blocks of the U-Net — the machinery from Chapter 6 where each patch of the latent asks the conditioning $\mathbf{c}$ what it should be. We are not re-deriving that here; Chapter 6 owns it. What matters is the consequence. Those are precisely the matrices that decide how strongly a word in your prompt pulls on a region of the image, so a small correction there is a large change in how the model reads language.

That is the honest explanation for trigger words. Whoever trains an adapter captions the training images with a rare token — "sks apple", something the base model has no strong opinion about — and the adapter learns to attach a large correction to that token's pull, precisely because there was nothing there before to interfere with. Leave the trigger word out of your prompt and such an adapter can look almost inert; put it in and the whole thing switches on. You are choosing whether the corrected path ever gets a strong query to answer.

Everything else is exactly as you left it. Re-picture Chapter 4's latent-diffusion-pipeline map: encoder, loop in latent space, decoder. A LoRA changes nothing about it — not the VAE, not the sampler, no extra stage. It edits a handful of matrices inside the noise predictor, and Chapter 5's loop then runs as before, calling a very slightly different function at every sampler step.

Why varied apples matter

Back in Chapter 1 we laid out nine apple thumbnails — deliberately different: red, green, golden, spotted, large, small, photographed from two angles — and the caption made a promise: these training apples are all different, and that is going to matter a lot when we get to LoRAs. This is where that debt gets paid.

Training an adapter is Chapter 3's procedure on a very small dataset. Encode one of your photographs to a latent, pick a random timestep $t$, add the matching noise using Chapter 2's closed form, ask the noise predictor for that noise, compare, and push $\mathbf{B}$ and $\mathbf{A}$ downhill. Repeat over your twenty or thirty images. The learned change is whatever consistently reduces that loss across your set.

Read that last sentence again, because it contains the whole trap. The adapter learns whatever is consistent. It has no idea which parts of your photographs are the concept and which are the circumstances. Suppose all thirty photographs of your heirloom apple sit on the same wooden table, in the same warm afternoon light, shot from the same height. "Russet shoulder" is consistent across your set — and so is "wooden table", and so is "warm light", and so is "shot from above". The gradient cannot tell them apart, so the adapter learns the whole bundle as one concept. Load it up, prompt for your apple in a snowfield, and the wooden table comes to the snowfield with it. That is not a bug in the method; it is the method working exactly as specified on data that specified the wrong thing.

The fix is variety, and you apply it with a camera, not a setting. Photograph the apple on a table, on grass, in a hand, against a white wall; morning light, lamp light, flat overcast; close, far, above, side. Now the only thing consistent across the set is the apple, so the only thing the adapter can learn is the apple. Variety is not "more data" — it is what isolates the concept from its accidents, and twenty varied photographs beat two hundred from one sitting.

Two neighboring failures deserve names. Overfitting is training for too many passes over too few images: the adapter stops learning "this apple" and starts memorising "these thirty pictures", and the giveaway is a generation that reproduces a training photo's exact composition whatever else you ask for. And data quality beats quantity outright: five sharp, well-lit, correctly captioned photographs teach a cleaner concept than fifty blurry ones, because every blurry image quietly teaches "blurry" as part of the target.

Counting the savings

Now the arithmetic that made the technique famous, on one honest matrix. Take a weight matrix that is $1{,}000$ numbers wide and $1{,}000$ numbers tall — a typical size for a matrix inside a model like this. A full change $\Delta\mathbf{W}$ to it has one entry per cell:

In words: changing that matrix the ordinary way means writing down and training one million numbers.

Now the low-rank version at a typical rank of $r = 8$. The column block $\mathbf{B}$ is $1{,}000$ tall and $8$ wide; the row block $\mathbf{A}$ is $8$ tall and $1{,}000$ wide:

In words: sixteen thousand trained numbers stand in for a million.

The ratio, worked: $16{,}000 \div 1{,}000{,}000 = 0.016$, which is $1.6\%$. Multiply that across the handful of cross-attention matrices a LoRA touches and the file lands in the low tens of megabytes, sitting beside a base model of several gigabytes — figures typical of models of this kind, not laws. That is the whole reason a shelf of adapters is a practical thing to own: one frozen base model, and a stack of small files, each teaching it one thing.

Because the corrections are additions, they compose. Load two adapters and the corrections add up on each shared matrix, each with its own strength:

In words: the adapted matrix is the frozen matrix plus the first adapter's change at its own strength, plus the second adapter's change at its own strength. (These strengths $s_1$ and $s_2$ are LoRA strengths — they are not Chapter 6's guidance scale $s$, which lives in an entirely different equation.)

Composition works right up until it does not. Each adapter was trained separately, assuming the frozen matrix underneath it was untouched and knowing nothing of the other. If they push the same matrices in conflicting directions — a style adapter that flattens everything against a subject adapter that insists on photographic texture — the sum is not a compromise but a fight, and the usual repair is turning both strengths down.

See it move

Where you'll meet this

In a workflow graph, this chapter is one node — usually called something like Load LoRA, sitting after the checkpoint loader, taking the base model in on one wire and handing a modified model out on another. That wiring is the picture of the whole chapter: the adapter never stands alone in the graph, because there is nothing for it to do without a model flowing through it.

The number in that node's box is $\frac{\alpha}{r}$ from the sweep table — the same multiplier, with the same arithmetic behind it. Some interfaces give you two of them, one for the correction applied inside the U-Net and one for the correction applied to the text encoder, because a LoRA can carry strips for both. And when a model page tells you to include a strange token like "sks apple" in your prompt, you now know exactly what you are doing: handing the corrected cross-attention path the one query it was trained to answer loudly.

Three habits fall straight out of the math. If a generation looks untouched, check the trigger word first. If it looks scorched, drop the strength before you rewrite the prompt — you are turning down the multiplier in front of $\mathbf{B}\mathbf{A}$, nothing more. And if a LoRA does nothing at all, check which base model it was trained against. (LoRA is the famous member of a larger family of adaptation methods, and the others are a topic for another day.)