Fine-tune, retrieve, or just ask - choosing the lever
A teammate asks the obvious question: our chatbot keeps getting the company's own policies wrong, so why not just train it on our documents? That question hides three different fixes, and sorting them out is what this page is for. Prompting changes what you ask the model for and how. Fine-tuning means an extra training run that changes what the model is like: its habits, its voice, how strictly it follows a format. Retrieval means the system looks up the relevant documents when a question arrives and shows them to the model before it answers, so the model reads the current version instead of trying to remember. The distinction I missed for months is that these levers change different things, so a fact problem and a behavior problem do not share a fix. Facts that move, or facts a company owns, almost always belong in retrieval, because baking them into the model means every document change costs a new training run. Fine-tuning is where you go when the model already knows the material but keeps behaving the wrong way. Most requests shaped like "make the model know our internal stuff" turn out to be retrieval problems once you look at what is actually changing.
Most people have this backwards
When a model gets your facts wrong, the right lever is almost never fine-tuning.
The catch
It holds whenever the facts move. A fine-tune teaches habits well and facts poorly: facts frozen into the model's memory (its weights) go stale the moment their source changes, and every change bills a retrain plus a round of grading to prove the new version. The claim breaks only when behavior is what is broken and the knowledge underneath barely moves, like a house style or a strict output format. That is real fine-tuning territory, and it is rarer than people expect.
Baseline first - what is your honest guess?
A team's chatbot gives outdated answers about their own product. Someone proposes fine-tuning on the product docs. What is the catch?
Make each call, then see the verdict
Predict first
A team fine-tunes on their product docs, and the docs change weekly. Over year one, does the cost of keeping the answers correct rise, fall, or stay flat?
Four jobs to build. Pick your lever for each one before you read how it plays out.
A support bot over your 10,000-article knowledge base. The articles update weekly.
Answers must come out in a strict house voice, short, no headers, and reviewers keep rejecting them.
Summarize and lightly edit emails the user pastes in. Nothing more.
The model keeps stating wrong numbers for your internal API rate limits.
How it works
The work sorts by what is actually missing. When the model knows the material and misread the instruction, the fix is a better prompt, which is the cheapest lever and takes minutes. When the answer lives in a document and the document keeps changing, retrieval is the right lever, because the model reads the current version at question time and an index refresh is cheap. When the knowledge is stable but the behavior is not, fine-tuning is the lever left standing, since changing what the model is like takes a training run. In an interview, the shape of a strong answer is to name all three levers, pick one for the job in front of you, and defend the pick by how often the facts change and who owns them.
The new hire
I think of it like a new hire in their first week. You would never re-train the hire to teach facts; you point them at the policy database, which is always current, and you do it every time a fact matters. What you do train once is how the company sounds and writes, which rarely changes. One half keeps moving and lives in a database, the other half is stable and lives in the person. A new hire stays current by consulting the database; a model does nothing unless the pipeline makes it, which is why retrieval is plumbing and not convenience.
Where it breaks
The analogy breaks where the new hire differs from a model. A hire knows things are written down somewhere and asks; a model answers from whatever it has, so the write-stuff-down habit has to be enforced by the pipeline (retrieval before generation). And the hire who hears the old policy once from a colleague will repeat it for a week - that is exactly what a fine-tune does with a fact.
Why interviewers lean on it
This is one of those questions that decides budgets, which is why it shows up early in interviews. A retrieval build costs a search index and a pipeline to keep it current. A fine-tune costs training runs, rounds of checking that each new version is actually better, and a maintenance schedule that starts the day it ships. When an engineer answers "fine-tune it" for a knowledge problem, the employer picks up a year-long maintenance bill without seeing it, and interviewers know how to spot the difference. Being the one who asks how often the facts change and who owns them is how you show you would not be that engineer.
The follow-up chain
Interviewers do not stop at the surface answer. Answer the way you would out loud, or skip - skipping reveals the strong answer and ends the chain.
The interviewer's next question
Retrieval seems to cover everything. When would you actually fine-tune?
Now answer one you have not seen
A teammate proposes fine-tuning one base model on the entire company wiki and pointing every internal tool at it, replacing the retrieval stack the tools currently use. What do you push back on, and what part of their plan would you keep?
Expires fast - as of 2026-10-04
As of 2026-10, long-context models weaken one claim in the depth-1 answer: "you cannot fit the docs in the prompt" now holds only above a corpus size. The retrieval argument that survives at any size is updates and permissions, which long context does not solve.
As of 2026-10, tool-use and structured-output tuning are the fine-tune use cases most teams actually run, ahead of style tuning.