The price of a tweak
We want to adapt a model to our examples. Full fine-tuning trains every weight and keeps its gradients and optimizer state. LoRA, low-rank adaptation, makes the trainable change smaller.
| Resource | Cost |
|---|---|
| Weights | 7 billion numbers at two bytes each: about 14 GB. |
| Training state | Gradients, optimizer state, and forward activations add memory. |
| Saved variants | A full new parameter set per fine-tuned model. |
Learn the change
Any updated matrix W-prime can be written as W plus a change. The symbol $\Delta$, “delta,” means change.
$$\mathbf{W}' = \mathbf{W} + \Delta\mathbf{W}$$In words: the new grid is the old grid plus a grid of changes, added cell by cell.
An unrestricted change has as many entries as W, so this identity alone saves nothing. We need a compact description of the change, like notes that modify how we use a book rather than a second full copy.
Rank: how complicated is a change?
![A quadrant plot. Three thin input arrows leave the origin to (1,0), (0,1) and (1,1), labeled in [1,0], in [0,1], in [1,1]. Three thicker output arrows also leave the origin, to (2,1), (6,3) and (8,4), labeled out [2,1], out [6,3], out [8,4]. A dashed line runs from the origin through the direction [2,1], and all three arrowheads sit exactly on it. An annotation reads: every output lands on this one line — rank 1.](media/lora/rank-one-line.svg){kind=icon}
Rank counts independent output directions. Our change sends [1,0] to [6,3], [0,1] to [2,1], and [1,1] to [8,4]. All are multiples of [2,1], so its rank is one.
$$\Delta\mathbf{W} = \begin{bmatrix} 6 & 2 \\ 3 & 1 \end{bmatrix} = \underbrace{\begin{bmatrix} 2 \\ 1 \end{bmatrix}}_{\mathbf{B}} \underbrace{\begin{bmatrix} 3 & 1 \end{bmatrix}}_{\mathbf{A}}$$In words: the whole change-grid is one column, $\mathbf{B} = [2, 1]$ standing up, multiplied by one row, $\mathbf{A} = [3, 1]$ lying flat.
This factorization uses a column B and row A. Cell (1,1) is 2×3=6; cell (2,2) is 1×1=1.
Pass through the bottleneck
$$\mathbf{A}\begin{bmatrix} 1 \\ 1 \end{bmatrix} = 3 \times 1 + 1 \times 1 = 4 \qquad \mathbf{B} \cdot 4 = \begin{bmatrix} 8 \\ 4 \end{bmatrix}$$In words: $\mathbf{A}$ squashes the whole input down to a single number, and $\mathbf{B}$ turns that number into some amount of one fixed direction. Everything must squeeze through a one-number bottleneck, which is why every output sits on one line.
The bottleneck is one number wide. With r lanes, B has d×r entries and A has r×d entries for a square d×d base matrix. The product has rank at most r, not necessarily exactly r.
The LoRA detour
Frozen W still transforms the input. A second route applies A then B, scales its output, and adds it to the frozen route.
Choose rank r and scale $\alpha$, “alpha.” Standard LoRA uses the multiplier α/r:
$$\mathbf{W}' = \mathbf{W} + \frac{\alpha}{r}\,\mathbf{B}\mathbf{A}$$In words: the working grid is the frozen grid plus alpha-over-r times the learned detour.
Worked example
Work every digit on the chapter's full toy. The frozen grid is a doubling machine from Chapter 3's family, $\mathbf{W} = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}$; our column and row are $\mathbf{B} = [2, 1]$ and $\mathbf{A} = [3, 1]$ from the last section; and we set $\alpha = 2$, $r = 1$, so $\alpha/r = 2$. Push $\mathbf{x} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$ through the detour route and add:
- Frozen path: $\mathbf{W}\mathbf{x} = \begin{bmatrix} 2 \\ 2 \end{bmatrix}$.
- Detour path: $\mathbf{A}\mathbf{x} = 4$, then $\mathbf{B}\cdot 4 = \begin{bmatrix} 8 \\ 4 \end{bmatrix}$, then scaled by 2 to $\begin{bmatrix} 16 \\ 8 \end{bmatrix}$.
- Add them: $\begin{bmatrix} 2 \\ 2 \end{bmatrix} + \begin{bmatrix} 16 \\ 8 \end{bmatrix} = \begin{bmatrix} 18 \\ 10 \end{bmatrix}$.
The scaling is a convention for controlling update strength; rank changes do not guarantee a proportional change in output magnitude.
Merge and compare
$$\mathbf{W}' = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix} + 2\begin{bmatrix} 6 & 2 \\ 3 & 1 \end{bmatrix} = \begin{bmatrix} 14 & 4 \\ 6 & 4 \end{bmatrix} \qquad \mathbf{W}'\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 18 \\ 10 \end{bmatrix}$$In words: you can run the detour live, or fold it into the grid once and drop the two factors. The merged grid gives identical outputs, $[18, 10]$ both ways.
Merging removes the extra inference route in exact arithmetic. Floating-point rounding or quantization can introduce small differences.
A common initialization sets B to zero and A to small random values, so BA initially vanishes. Setting both to zero would prevent either factor from receiving a useful initial gradient. Our α=2,r=1 scale equals α=16,r=8: both give two. The large toy update keeps the arithmetic visible; practical update sizes vary.
Why so little can be enough
Many adaptations can be expressed through a limited set of useful directions. Early LoRA experiments found small ranks competitive with full fine-tuning on several tasks. That is empirical evidence, not a guarantee that every style change or new topic is low rank.
Rank, target matrices, data quality, and optimization all affect results. A harder adaptation may need higher rank or full fine-tuning. For fresh facts that need citations, retrieval is often easier to inspect than changing weights.
Counting the savings
Count one adapted matrix
$$\underbrace{1{,}000 \times 1{,}000}_{\text{full } \Delta\mathbf{W}} = 1{,}000{,}000 \qquad \text{vs} \qquad \underbrace{(1{,}000 \times 8)}_{\mathbf{B}} + \underbrace{(8 \times 1{,}000)}_{\mathbf{A}} = 16{,}000$$In words: sixteen thousand trainable numbers stand in for a million, or 1.6 percent.
The saving is in trainable state: 16,000/1,000,000=0.016=1.6%. For a rectangular matrix, count r×input-width plus output-width×r.
An adapter stores A and B for each targeted matrix. Adapting only query and value projections can train far below 1% of a model's parameters. Other configurations adapt more matrices, including MLPs.
| Mode | Benefit | Tradeoff |
|---|---|---|
| Live adapter | Keep one base and switch adapters. | Extra matrix work and adapter memory. |
| Merged weights | Use one ordinary matrix path. | A separate merged variant; switching requires managing another weight set. |
LoRA removes gradients and optimizer state for frozen weights, but still needs the base weights and activation computation. Whether a run fits one consumer GPU depends on all those quantities, not rank alone.
See it move
Watch the rank-one outputs stay on a line, then count the two factor matrices.
Why the LLM cares
LoRA lets us reuse a large learned model while training and distributing much smaller task-specific updates. It changes the behavior of selected transformations without updating their base entries.
The same method also adapts image models. It is one member of parameter-efficient fine-tuning, a broader family of approaches.
What you now know
- LoRA learns an additive update to frozen weights.
- The product BA has rank at most the chosen bottleneck width.
- The scale α/r controls the update's contribution.
- Our live and merged routes both output [18,10].
- A rank-eight update to a 1,000-square matrix trains 16,000 numbers.
- Small trainable state does not eliminate base-model or activation memory.
Where we're headed
Our model is adapted. Next we'll run it efficiently, storing reusable attention work and reducing the bytes needed for its numbers.