LoRA

A full fine-tune changes every weight in the model, so the result is a second full copy. LoRA freezes the base model and trains two small matrices of a chosen rank that multiply into a correction, so the learned behavior lives in a file thousands of times smaller than the model. At inference, either the adapters are merged into the base weights or the base computes an extra cheap pass per layer.

For serving, the point is multiplexing: one big base model holds many behaviors, and a request names which adapter it wants. The base weights are read once and shared by every variant, so adding a behavior costs megabytes instead of the gigabytes a full fine-tuned copy would need.

The interviewer will want the operational side named. Adapters need routing, selection, and warm-up, since swapping one into a live request path has its own overhead and the cold-start penalty of an unloaded variant. When the behavior change is genuinely large, full fine-tuning is still the right tool; LoRA covers the long tail of variants cheaply and lets you share one serving stack across all of them.

One shared 70B base model in 8-bit weights (70 GB) plus rank-16 LoRA adapters that weigh an assumed 60 MB each, for 100 behavior variants of the same model.

1. Rank 16 on one 4,096 x 4,096 matrix: 4,096 x 16 + 16 x 4,096 = 131,072 parameters, 0.78 percent of the matrix itself.
2. Whole-model adapter at 60 MB: 100 variants together are 100 x 60 MB = 6 GB.
3. One base plus 100 adapters: 70 GB + 6 GB = 76 GB, one card with cache headroom, instead of 100 fine-tuned copies at 100 x 70 GB = 7,000 GB.

A 76 GB serving footprint hosts every behavior variant, where full fine-tunes would need 7,000 GB - the variants became nearly free in memory, and the cost moved to routing them.

Where this shows up

• Fine-Tuning: When, and WhyPicture a specific morning. Your support inbox fills with replies from the AI assistant your team switched on last month, and a customer has written in because the assistant told them refunds take fourteen days, when your policy changed to seven. Nothing is broken. The model is doing, very competently, exactly what it learned to do before your product existed. Fine-tuning is the name for one of the ways you could respond: take a model that already works and continue its training on thousands of your own examples, so its default behavior shifts and it starts answering the way your product should answer without being asked every time. That is different from the two moves you already know. A prompt is a request, and a request has to be repeated on every single call; retrieval is looking things up, and looking things up solves missing facts. Training changes the default itself: the patterns land in the weights, the model's adjustable settings, the ones that were set when the model learned the first time. The honest headline of this whole topic is a ladder. Better prompting costs nearly nothing and is the right first move almost every time. Adapter tuning, the light kind, where you train a small add-on called a LoRA (a low-rank adapter, a small add-on trained while the base model stays frozen) costs a little and needs a few thousand cleaned, labeled examples, examples where someone wrote down the answer that was wanted. A full fine-tune, where every core weight moves, costs a lot, eats your week, and almost no product ever needs one. Tuning can only teach what your examples show, so with messy examples you pay to teach the model your mistakes, and with facts that change weekly no amount of training can keep up, because a trained model is a snapshot. That is the shape of the classic interview question, RAG or fine-tune or a better prompt: knowledge that changes points to retrieval, style and format the model keeps getting wrong points to tuning or to a careful prompt, and no data points to neither. Whatever you pick, you prove it the same way every time: run the eval set, a batch of typical questions with known-good answers, before the change and after, and compare. One more word worth carrying into interviews: distillation, which means training a small model to imitate a big one on one narrow job, to cut the cost per answer. The challenges below grade the parts you can write down: the route rule as code, the arithmetic of what an adapter run actually costs, the audit that keeps a spoiled example out of your training pool, and one written decision, graded against a published rubric, on the fine-tune versus the prompt.
• Inference Economics & LatencyEvery time someone sends a message to an AI feature, the model has to run on that request, and the run has a price in dollars and a wait in seconds. The industry word for the run itself is inference: the model processing your prompt and producing an answer, and it is the line item that grows with usage. Where does the money go? Mostly into tokens, the word-sized pieces of text that both prompts and answers are made of. Input tokens are re-sent with every call; output tokens are generated one at a time at a premium rate; and a large discount called prompt caching applies only while the opening of your prompt matches what the provider already has stored, which one careless changing detail destroys. The wait decomposes into stages that scale differently: reading the prompt grows with prompt size, writing the answer grows with answer length, and a fixed network round trip is attached either way - and the stages disagree about which lever is worth pulling. Interviewers pursue this topic because cost and latency are the two numbers that decide whether an AI feature survives its first real month, and both are governed by arithmetic you can do on a whiteboard. The three challenges below work that arithmetic: the cache hit rate and the savings a rotating preamble quietly destroys, the share of a latency budget that writing the answer consumes, and the work a hard output cap clips off real requests.

Related terms