AI Tools Field GuideFine-tuning, LoRA and distillation
The ladder starts with a better prompt and almost never reaches training. Climb it only as far as a given problem requires, and know what each rung costs in examples, hardware and upkeep before you climb it.
- A better prompt
- A skill file
- Retrieval
- A LoRA adapter
- Distil a small model
- Train from scratch
Each rung costs ten times the one before. Climb only when the rung below has demonstrably failed on a test set.
What fine-tuning changes
Fine-tuning continues a model's training on your examples. The weights move. Afterwards the model behaves differently without being told to: it writes in your format, uses your structure, applies your narrow judgment. That is the important word, behaves. Fine-tuning is good at teaching a way of doing something and poor at teaching facts, because facts go stale the day after training and the model has no way to tell the new ones from the old. Facts belong in retrieval. Behavior that a prompt cannot hold belongs in fine-tuning.
"Cannot hold" is the test. If a skill file with two examples gets the format right, fine-tuning buys nothing. If the model needs thirty examples to get it right and the window cannot spare them on every call, or if it gets it right and then drifts over a long session, that is a behavior a prompt cannot hold, and a few thousand examples in the weights will.
Full fine-tune, LoRA, distillation
| Method | What moves | What it needs | What you get |
|---|---|---|---|
| Full fine-tune | Every weight in the model | The whole model in GPU memory with room to update it; a large clean set; days | A new copy of the model. Expensive to make, expensive to keep. |
| LoRA (low-rank adaptation) | Small adapter matrices added beside the frozen weights, a fraction of a percent of the total | One or two workstation GPUs for a mid-size open-weight model; thousands of examples; hours | A small adapter file you load on top of the base. Swap it, stack it, version it. |
| Distillation | A small "student" model trained to imitate a large "teacher" on your inputs | A teacher you can call at volume; tens of thousands of inputs; a held-out set; a small base model | A fast, cheap model for one narrow lane that runs where the big one cannot |
Vendors that offer hosted fine-tuning document how your data is and is not used in the process (OpenAI fine-tuning docs); read that page before you upload a training set. LoRA is why fine-tuning is now within reach of a technical shop rather than a vendor. The base model stays as shipped; you train the adapter, and the adapter is small enough to keep in version control. On a vendor's closed model, fine-tuning exists only as a hosted service if the vendor offers one, under the vendor's rules, with your data on the vendor's side; check the vendor's terms and your organization's rules before assuming either.
Distillation is a different bet. You are not improving the big model; you are replacing it, for one job, with something much smaller. The shape of a sound distillation project:
- Pick the lane by volume times failure. A task that runs hundreds of times a day, where the cheap model you have is wrong a third of the time, is the candidate. A task that runs weekly is not.
- Short prompts. Distillation pays when the inputs are a few thousand tokens. Long agentic prompts stay on the frontier model.
- Label with the teacher, twice. Run the teacher on every training input two times and keep only the examples where it agreed with itself. Noisy labels are what a small student amplifies.
- Hold out a set the student never sees, and write the acceptance rule before training: beat the current model's score with a confidence interval that clears it, on that set.
- Budget shape. Labeling tens of thousands of inputs with a frontier model costs hundreds of dollars; the training run is hours on rented GPUs. The expensive parts are cleaning the data and building the evaluation, which is where the project time goes.
When each is worth it
| You need… | Reach for | Not before |
|---|---|---|
| A format or tone change | A skill file with examples | Nothing; start here |
| Current facts, citations | Retrieval | Nothing |
| A narrow behavior the prompt cannot hold, on an open-weight model you can host | LoRA | The prompt has failed on a test set, and you have a few thousand clean examples |
| Volume: the same narrow judgment thousands of times a day, fast and cheap | Distillation | A teacher exists, a held-out set exists, and the lane's failure is measured |
| A model with different knowledge from the ground up | Training from scratch | Almost never for a command. Read Training, weights and checkpoints to see why. |
What it costs after it works
A fine-tuned model is a maintained thing. When the base model updates, the adapter has to be retrained and re-evaluated. When the form changes, the examples are wrong. Training on your examples can push out abilities the base model had (the effect is called forgetting, and it is real), so the held-out set must include tasks the model used to do well. And the training set is now the most sensitive artifact you own: it is your documents, distilled. Whatever your organization bans from a chat box (personal data, controlled or classified material) applies to training data at least as strictly; and a fine-tuned model can reproduce fragments of what it was trained on.
What to do with this
If someone in your chain says "we should fine-tune a model", ask three questions: on what test set did the prompt fail, how many approved examples do we have, and who re-trains it when the base model changes. Then read what training is underneath all of this: Training, weights and checkpoints.
Vendor facts checked ; re-verify before relying on them.