Fine-Tuning: When, and Why
Picture a specific morning. Your support inbox fills with replies from the AI assistant your team switched on last month, and a customer has written in because the assistant told them refunds take fourteen days, when your policy changed to seven. Nothing is broken. The model is doing, very competently, exactly what it learned to do before your product existed. Fine-tuning is the name for one of the ways you could respond: take a model that already works and continue its training on thousands of your own examples, so its default behavior shifts and it starts answering the way your product should answer without being asked every time. That is different from the two moves you already know. A prompt is a request, and a request has to be repeated on every single call; retrieval is looking things up, and looking things up solves missing facts. Training changes the default itself: the patterns land in the weights, the model's adjustable settings, the ones that were set when the model learned the first time. The honest headline of this whole topic is a ladder. Better prompting costs nearly nothing and is the right first move almost every time. Adapter tuning, the light kind, where you train a small add-on called a LoRA (a low-rank adapter, a small add-on trained while the base model stays frozen) costs a little and needs a few thousand cleaned, labeled examples, examples where someone wrote down the answer that was wanted. A full fine-tune, where every core weight moves, costs a lot, eats your week, and almost no product ever needs one. Tuning can only teach what your examples show, so with messy examples you pay to teach the model your mistakes, and with facts that change weekly no amount of training can keep up, because a trained model is a snapshot. That is the shape of the classic interview question, RAG or fine-tune or a better prompt: knowledge that changes points to retrieval, style and format the model keeps getting wrong points to tuning or to a careful prompt, and no data points to neither. Whatever you pick, you prove it the same way every time: run the eval set, a batch of typical questions with known-good answers, before the change and after, and compare. One more word worth carrying into interviews: distillation, which means training a small model to imitate a big one on one narrow job, to cut the cost per answer. The three challenges below grade the parts you can write down: the route rule as code, the arithmetic of what an adapter run actually costs, and the audit that keeps a spoiled example out of your training pool.
Key Concepts
- 1Fine-tuning changes the default, not the instructions: the patterns from your examples land in the weights (the model's adjustable settings), so the behavior carries into every future call without being repeated. A prompt has to be re-sent and re-followed on every call, which is the whole difference
- 2The cost ladder reads: a better prompt costs nearly nothing, LoRA adapter tuning costs a little (a small add-on trained while the base model stays frozen, plus a data pipeline and an eval harness), and a full fine-tune of every weight costs a lot and is almost never the product answer
- 3Training copies your examples, including their mistakes: blanks, unvalidated labels, and duplicates in the training pool all come back out as behaviors the model now performs more confidently
- 4A fine-tuned model is a snapshot: facts that stay true for days belong in retrieval, where the answer is built from information that is still true at question time
- 5Whatever route you pick, the proof is the same eval set measured before and after the change. A number you have no before-shot for is not evidence, and the interviewer will ask for exactly that
- 6Distillation is the one-sentence shortcut: train a small model to imitate a big one on one narrow job, and the cost per answer drops with it
Price the three paths before you pick
The fine-tuning decision walkthrough
Anyone who has watched a support assistant confidently quote a plan that stopped existing last Tuesday knows this one. The model is not broken; it is doing what it learned before your product existed, and three tools fix different things. A prompt is a request you repeat on every call. Retrieval looks the current facts up at question time. Fine-tuning continues the model's training on thousands of your own examples so its default behavior shifts, and the light version, a LoRA (low-rank adapter, a small add-on trained while the base model stays frozen), costs a little instead of a lot.
The decision is a ladder, not a vibe. Below, four product problems, each drawn from a real support-ticket season. For each one, the path chips are the tools: pick the one you believe fits first, then pick again to commit, and the verdict prices every route for that problem, with the failure numbers that justify what it says. Nothing here is scripted: every number comes out of a small made-up cost model, with fictional engineer hours, training runs, and per-1K token prices, so the reasoning is the part worth keeping.
Scenario 1 of 4
The support voice problem
A support assistant answers fast and correctly, but it sounds like a press release, and the team wants it to sound like a specific product voice. A style guide and five example replies already sit in the system prompt; the voice still misses on 12 of every 100 replies.
What to measure first: Count tone misses per 100 replies on a fixed eval set, and count the labeled example replies sitting around in the ticket archive: 4,000 of them.
Predict first: pick the path you believe fits. Your first pick is the call, and it is recorded once; pick again to commit.
Scenario 2 of 4
The facts that move every week
The assistant quotes plan prices, and the plans change weekly. Last week's number is what a frozen model confidently says, and 9 of every 100 answers quote a plan that no longer exists. The example pool could reach 2,600 pairs if someone spent a weekend cleaning.
What to measure first: Check where the answer gets its number: inside the model, inside the prompt, or in a source that can be fetched fresh at question time. Count the stale-quote misses per 100 on the same eval set each week.
Predict first: pick the path you believe fits. Your first pick is the call, and it is recorded once; pick again to commit.
Scenario 3 of 4
The JSON that breaks twice in a hundred
The assistant fills a structured order form, its output is JSON, a format where the text has to parse as data with exactly the agreed fields, and it breaks that shape 2 times in 100. The current prompt asks politely. Nobody has shown it the exact shape with a worked example.
What to measure first: Log the parse failures per 100, and read the prompt: does it state the exact field shape and show one worked example, or does it just mention JSON in passing?
Predict first: pick the path you believe fits. Your first pick is the call, and it is recorded once; pick again to commit.
Scenario 4 of 4
The small-example-pool urge
A domain full of shorthand jargon the model misreads, and the team has 60 usable labeled examples. The production prompt already carries a glossary of the fifty terms everyone sees most. The eval still shows 14 misunderstandings per 100, measured with the glossary in place.
What to measure first: Count the labeled examples honestly, then compare that count against what a tuning run needs. Below the bar, a run teaches the 60 sentences, not the domain.
Predict first: pick the path you believe fits. Your first pick is the call, and it is recorded once; pick again to commit.
The tool list, one line each
Better prompt - Rewrite the request: state the rule, show worked examples. Nearly free, repeats on every call.
Retrieval - Look the current facts up at question time. Fixes stale knowledge, never changes how the model writes.
Fine-tune (LoRA) - Train a small add-on on your examples while the base model stays frozen. Changes the default itself.
None, for now - Stop and measure. Right when the failure is cheap to live with or the data is too thin to teach anything.