LoRA

The price of a tweak

Chapter 16 ended with a tempting recipe. Want a radar-systems tutor instead of a generic assistant? Fine-tune it: keep training the model on your own examples. It sounds cheap — a little extra practice on top of a finished model. Now do the accounting the recipe quietly skipped. A mid-size open model holds about 7 billion learned numbers, and full fine-tuning nudges every single one of them.

Here is the bill, in three plain lines:

• Memory. Every trainable weight needs its own gradient — the backward blame signal from Chapter 14 — plus the optimizer's running bookkeeping for each weight. The training footprint balloons to several times the size of the model itself.
• Storage. The result of your fine-tune is a complete second copy of all 7 billion numbers — roughly 14 GB at standard 16-bit storage. That is one full copy per project, per experiment.
• Hardware. This is rack-of-GPUs territory, not laptop territory.

Fine-tuning for a narrow job — a tone, a format, a specialty — feels like it should be a small change. So here is the question the rest of this chapter answers: is there a way to make the change literally small — to write it down in far fewer numbers than the model itself? There is. It is called LoRA, low-rank adaptation, and by the last paragraph its one formula will read like arithmetic.

Don't rewrite — learn the change

One move unlocks everything: fine-tuning never needs to replace a weight grid — it can learn a correction that gets added on. Pick any one grid $\mathbf{W}$ in the model (one attention lens from Chapter 10, say). After fine-tuning it has become some new grid $\mathbf{W}'$ — read that "W-prime, the updated grid" — and we can always write the update as the old grid plus a grid of changes. Writing $\Delta$ for that change (the Greek letter delta, the traditional symbol for "a change"):

In words: the new grid is the old grid plus a grid of changes, added cell by cell.

And adding two grids is a move you already own. We have added vectors component by component since Chapter 1; adding two grids is the same move — matching cells add.

On its own, though, this reframe saves nothing, and honesty demands we say so. The change $\Delta\mathbf{W}$ has exactly as many cells as $\mathbf{W}$ — a million-cell grid still needs a million-cell correction. Writing the update as "old plus change" only pays off if the change can be described more cheaply than the original. So the real question becomes: how complicated is the change that fine-tuning actually needs?

Picture what you do when you personalize a textbook. You don't reprint the book — you write in the margins. The book (billions of numbers, the pretrained $\mathbf{W}$) stays exactly as it was; your notes are tiny, because what you add is narrow and consistent. LoRA is the mathematical version of writing in the margins — and "how complicated is a change" turns out to have a precise mathematical name: rank.

Rank: how complicated is a change?

Picture before symbols. Chapter 3 taught that a $2\times 2$ grid takes in any arrow on the plane and hands back another arrow — and in general its outputs can land anywhere on the plane. Now meet a grid that can't. Here is the chapter's toy change-grid, $\begin{bmatrix} 6 & 2 \\ 3 & 1 \end{bmatrix}$, and we'll feed it three inputs by Chapter 3's recipe (each output column is a column of the grid; a general input has the rows dot it):

• Feed it $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$: out comes the first column, $\begin{bmatrix} 6 \\ 3 \end{bmatrix}$.
• Feed it $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$: out comes the second column, $\begin{bmatrix} 2 \\ 1 \end{bmatrix}$.
• Feed it $\begin{bmatrix} 1 \\ 1 \end{bmatrix}$: the rows dot the input — $6+2 = 8$ on top, $3+1 = 4$ below — out comes $\begin{bmatrix} 8 \\ 4 \end{bmatrix}$.

Now look at the three answers together: $[6, 3]$, $[2, 1]$, $[8, 4]$. Every one of them lies on the same line through the origin — the line of the direction $[2, 1]$. Each is just some number of copies of $[2, 1]$: three copies, one copy, four copies. And this is true for every possible input. This grid's whole world of outputs is one single line.

That is the idea we needed a name for. The rank of a grid is how many independent directions its outputs can span. A generic $2\times 2$ grid fills the plane: rank 2. Our toy only ever draws one line: rank 1 — the simplest possible non-zero change. Low rank means a change with few independent moves in it. The grid still holds four numbers; it is the behavior that is simple.

And here is the secret that makes rank useful: a rank-1 grid factors. It is a skinny column times a flat row.

In words: the whole change-grid is one column, $\mathbf{B} = [2, 1]$ standing up, multiplied by one row, $\mathbf{A} = [3, 1]$ lying flat.

Check a cell so the product isn't magic — grid times grid is rows-dotting-columns from Chapter 3. Cell in row 1, column 1 is $2\times 3 = 6$; cell in row 2, column 2 is $1\times 1 = 1$. (The other two are yours to verify.) These two factors keep their names all chapter: $\mathbf{B}$, the column, $d$ numbers tall, and $\mathbf{A}$, the row, $d$ numbers wide.

Now watch why the factored form is stuck on one line — the bottleneck, worked with the chapter's spine numbers. Take the input $[1, 1]$. The row $\mathbf{A}$ hits it first and squashes it to a single number; then the column $\mathbf{B}$ stretches that number back into an arrow:

In words: $\mathbf{A}$ squashes the whole input down to a single number, and $\mathbf{B}$ turns that number into some amount of one fixed direction. Everything must squeeze through a one-number bottleneck — and that is why every output sits on one line.

Want a slightly richer change? Give the bottleneck more lanes. Let $r$ numbers pass through instead of 1: $\mathbf{B}$ becomes $d \times r$ (that many skinny columns), $\mathbf{A}$ becomes $r \times d$ (that many flat rows), and now the outputs can span up to $r$ independent directions. That dial $r$ — "r, the rank you choose" — is LoRA's central knob: 1, 4, 8, 64… always tiny next to $d$.

The LoRA move: a skinny detour

Now assemble the method in one breath, then spend the section proving it works. Freeze the pretrained grid $\mathbf{W}$ — training never touches it again. Bolt a detour onto it: the input also flows through $\mathbf{A}$ (down to $r$ numbers), then through $\mathbf{B}$ (back up to full size), and that detour's answer — scaled by a volume knob — is added onto $\mathbf{W}$'s answer. Train only $\mathbf{B}$ and $\mathbf{A}$.

Here is the formula, with every symbol re-introduced in words. The rank you chose is $r$, and $\alpha$ — "alpha, the LoRA scale" — is a volume knob for how loudly the detour speaks:

In words: the working grid is the frozen grid plus alpha-over-r times the learned detour.

Why divide by $r$? One honest sentence: if you later double the rank, the product $\mathbf{B}\mathbf{A}$ picks up roughly twice as many contributions, so dividing by $r$ keeps the detour's overall volume comparable — and your other settings survive a change of rank.

Now the payoff. Instead of running the detour live, pre-add the two grids once — $\mathbf{W}' = \mathbf{W} + 2\cdot\mathbf{B}\mathbf{A}$, where $2\cdot\mathbf{B}\mathbf{A} = \begin{bmatrix} 12 & 4 \\ 6 & 2 \end{bmatrix}$ — and push the same input through the single merged grid:

In words: you can run the detour live, or fold it into the grid once and throw the scaffolding away — the merged grid gives identical outputs, $[18, 10]$ both ways.

That equality is why a merged LoRA costs nothing extra at inference: once you have added the detour into $\mathbf{W}$, the model is just… a model.

One detail practitioners rely on, and it is elegant. At the very start of training, $\mathbf{B}$ is set to all zeros — so $\mathbf{B}\mathbf{A} = 0$, the change $\Delta\mathbf{W}$ is zero, and the fine-tune begins exactly at the pretrained model's behavior. Training then grows the change outward from "no change at all." Chapter 13's downhill walk starts from the best base camp there is.

A closing note on our toy's volume, because it matters. Our detour shouts: it changes the doubler's output from $[2, 2]$ to $[18, 10]$. That is on purpose — every digit stays visible. Real fine-tuning updates are gentle nudges, not shouts. And, pleasingly, our knob is honest anyway: $\alpha = 2$ with $r = 1$ gives a multiplier of 2, and the field's everyday default of $\alpha = 16$ with $r = 8$ gives… the same 2.

Why so little is enough

A careful reader should be uneasy right now. A rank-1 change only draws a line — surely a real fine-tune needs the full plane (well, the full 4,096-dimensional space)? The empirical answer, honestly attributed: the 2021 paper that introduced LoRA (Microsoft researchers) found that for adapting a big pretrained model, ranks as small as 1 to 8 matched full fine-tuning on many tasks. Not "almost as good on a budget" — matched.

The intuition for why, in this site's own language: a fine-tune for a narrow job is a consistent, repeated adjustment. "Lean the outputs toward radar-systems vocabulary." "Always answer in the house format." The same few directions of change, applied across many different situations — and that is precisely what a low-rank grid is: a few directions, applied everywhere. The pretraining already built the library (Chapter 15); the fine-tune is margin notes, and margin notes are low-rank.

Say the limit out loud, because this is a bet, not a law. Low rank fits changes of style, format, tone, and light specialization. Teaching the model a genuinely new body of knowledge — a language it never saw, a private field of facts — is not a few consistent directions, and low rank can genuinely fall short there. Then you crank $r$ up, or full fine-tune, or don't put facts in the weights at all (retrieval, two chapters ahead). A method's honest boundary is part of understanding it.

Counting the savings

Now the arithmetic that made LoRA famous, on one honest mid-size grid: a $d \times d$ weight grid with $d = 1{,}000$. A full change $\Delta\mathbf{W}$ is $1{,}000 \times 1{,}000 = 1{,}000{,}000$ numbers. LoRA at $r = 8$ replaces it with two slivers: $\mathbf{B}$ is $1{,}000 \times 8 = 8{,}000$ numbers, $\mathbf{A}$ is $8 \times 1{,}000 = 8{,}000$ numbers — 16,000 in all.

In words: sixteen thousand trainable numbers stand in for a million — 1.6 percent.

Scale that to a real model. In practice you attach detours to a subset of grids — classically the attention lenses $\mathbf{W}_Q$ and $\mathbf{W}_V$ (Chapter 10's violet grids) in every layer — and a 7-billion-parameter model ends up with a few million trainable numbers: well under 1% of the model. The saved training memory — no gradients, no optimizer bookkeeping for the frozen 99%-and-then-some — is exactly what moves fine-tuning off a rack of data-center GPUs and onto one consumer graphics card.

And the storage win becomes a workflow win. Your finished fine-tune is just $\mathbf{B}$ and $\mathbf{A}$ for each adapted grid — an adapter — a file measured in megabytes, riding on a base model measured in gigabytes. Keep a shelf of them: a radar-systems tutor, a contract summarizer, a pirate-speak toy — one frozen base model, many cheap personalities.

That leaves one honest choice at serving time. Either run the detour live — a tiny bit of extra arithmetic per token, and adapters can be hot-swapped between requests, which is how one provider serves thousands of customers' fine-tunes on shared hardware — or merge, folding $\frac{\alpha}{r}\mathbf{B}\mathbf{A}$ into $\mathbf{W}$ once (exactly as our worked example proved is lossless) for zero added cost, at the price of baking that one personality in.

See it move

You've built this on paper; now watch it move. The video takes the change-grid $\begin{bmatrix} 6 & 2 \\ 3 & 1 \end{bmatrix}$, eats three different inputs, and drops every answer on one line; splits the grid into a column times a row; and shrinks a million-number square down to two 16,000-number slivers — the very same numbers as this chapter.

Why the LLM cares

Here is where this shows up in the wild. When you hear "I fine-tuned a 7B model on my gaming PC overnight," you are hearing LoRA (or a close cousin). When an API offers cheap fine-tunes with instant switching between them, you are hearing adapters being hot-swapped on a shared base model. This one technique is a large part of why customized models went from a corporate luxury to a hobbyist weekend.

Tie it back to the machine we built. Chapter 11 said everything a model knows lives in how its violet grids transform vectors. LoRA is surgical editing of that behavior — a few learned directions added to chosen grids — without disturbing the library underneath. It is the gentlest possible answer to "change what the machine does."

One beat of breadth, names only: the same trick escaped LLMs entirely. Image-generation models take LoRA adapters too — an adapter that draws in your art style is a few megabytes riding a frozen giant — and "parameter-efficient fine-tuning" is now a whole family of methods, of which LoRA is the famous member.

You can now decode a sentence like "a rank-16 LoRA on the attention layers, alpha 32" the way you decode arithmetic — and you know exactly which numbers move, which stay frozen, and why the file is small.