AI engineer interview guide
RAG Interview Questions
The short answer. RAG interview questions test whether you can trace an answer back to the pipeline stage that produced it. The interviewer does not want the letters spelled out; they want retrieval, ranking, context packing, grounding, and an evaluation loop you can defend. This guide starts with the one-paragraph answer and then walks the failure cases.
What is RAG in one answer?
Retrieval-augmented generation fetches evidence before it answers. The pipeline takes a query, retrieves candidate documents, ranks them, packs the best ones under a token budget, prompts the model with that context, and asks it to cite what it used. The answer can only be as good as the evidence the retrieval step supplied, which is why retrieval quality is the first thing to measure.
RAG is not always the right lever. If the facts fit in a prompt you control, or a structured lookup returns the answer, a retrieval index adds work and a new failure surface. Use RAG when the corpus is large, private, or changes independently of your deploy. Use fine-tuning for format or behavior.
How do you choose a retrieval method?
Keyword search with TF-IDF or BM25 is strong on exact identifiers, error codes, and names, and weak when the user paraphrases. Embedding search compares meaning and misses rare tokens. Hybrid retrieval runs both and merges the ranked lists, often with reciprocal rank fusion, because ranks are more stable than raw scores across different scorers.
Recall@k tells you how often the right document appears in the top k, but not whether the answer used it. A relevant passage ranked low can still be cut by the context budget. Compare systems by rank and grounded accuracy, not by cosine scores from different collections, because those scores shift when the corpus changes.
How do you build the context window?
Chunk with the answer in mind. A chunk should be large enough to contain the fact and the sentence that qualifies it, and small enough that several candidates fit. Keep metadata on every chunk so you can cite it and boost recency. Deduplicate near-identical passages before you spend tokens on them.
Then pack under a budget. A rough planning rule for English is about four characters per token, so a 6,000-character context is roughly 1,500 tokens before the prompt wrapper. Leave room for the answer, put the strongest evidence where the model will read it, and do not pack to the last token. An overflowing prompt loses its tail silently, which looks like a retrieval failure.
Why do RAG systems answer wrong?
There are four stages to check, in order. Retrieval: the evidence was never fetched, because of a vocabulary gap, a bad filter, or a corpus that was not indexed. Ranking and packing: the evidence was fetched but ranked below the cut or truncated. Grounding: the evidence was present but the prompt or the model ignored it. Freshness: the top document was correct once and is stale now.
A useful diagnosis names the stage and the test that separates it from the neighbors. Inspect the retrieved set, read the exact cited span, replay the prompt with only the gold passage, and compare the document date with the product policy. Each test removes one suspect instead of guessing at the model.
How do you evaluate and ship RAG?
Build a small evaluation set before you tune anything. For each case record the expected evidence and a rubric: supported by a cited passage, covers the required points, refuses when evidence is missing. Measure retrieval with recall@k and the answer with the rubric. If a judge scores the answers, calibrate it against human labels and report agreement; keep fresh cases so the set does not become a memory test.
In production, watch the same stages. Log the query, retrieved ids and scores, prompt version, model, answer, citations, latency, and cost. Cache repeated prefixes and documents, keep a latency budget that includes retrieval and guardrails, and run a canary before a prompt change. Treat fetched text as untrusted, because a document can carry instructions as easily as facts.
What questions come up in a RAG interview?
1. Walk me through RAG from query to answer.
Rewrite the query if the conversation needs it, retrieve candidates with lexical and vector search, fuse and rerank them, deduplicate, and pack the best evidence under a token budget. Prompt the model to answer only from that evidence and to cite the passages it used. Return the answer with citations, and log retrieval and answer metrics so a failure can be traced to a stage.
Whether you know the stages and can name where quality is decided.
2. What is recall@k, and what does it miss?
Recall@k is the share of relevant documents in the top k results. It misses ranking quality inside the k, whether the model actually read the document, and whether the answer stayed grounded. A pipeline can have high recall and still bury the one passage that matters under four distractors, so pair it with an end-to-end check.
That you treat retrieval as necessary but not sufficient.
3. When would you use lexical search instead of embeddings?
When exact tokens carry the meaning: part numbers, error codes, API names, and rare identifiers that an embedding may blur. Lexical search is also cheaper and easier to explain. Use embeddings when users paraphrase, and hybrid retrieval when both cases appear in the same query stream.
That you choose the retrieval method from failure data, not from a default stack.
4. How do you pick a chunk size?
Start from the answer unit, not from a round number. A chunk should contain the claim and its qualifying sentence, and should still leave room for several candidates in the budget. Test a couple of sizes on the evaluation set, watch recall and grounded accuracy together, and change overlap only after you see a boundary failure. Chunking is a measured choice.
That you can connect a chunking decision to a measurable failure.
5. A user says the answer is wrong, with a citation. How do you debug it?
First check whether the cited passage supports the claim. Then ask whether that passage was in the retrieved set, where it ranked, and whether packing kept the whole chunk. If the evidence was present and the answer still failed, replay the prompt with only that passage. That order separates retrieval, packing, and prompting in three small tests.
That you debug by stage instead of rewriting the prompt immediately.
6. How do you stop a retrieved document from injecting instructions?
Treat all fetched text as data. Delimit it in the prompt, tell the model the passages cannot change its rules, and do not expose tools that a document could trigger without confirmation. On the server side, allowlist tools and validate arguments. Test with documents that contain hostile instructions, because a prompt-only guard is not a boundary.
That you know prompt injection is a boundary problem.
7. What do you log for a RAG request?
The query, retrieved ids and scores, the packed context or a hash of it, the prompt version, the model, the answer, the cited spans, latency per stage, and token cost. Keep private content out of logs where policy requires it, and retain enough to reproduce a bad answer. Without the trace you cannot tell a data problem from a prompt problem.
Whether you can operate the system after launch.
8. How do you choose a model for RAG?
Run a few candidates on the same evaluation set and compare grounded accuracy, refusal behavior, latency, and cost. A larger context window helps only if packing uses it well, and a cheaper model can win once retrieval is strong. Decide with the task metrics, not the leaderboard, and keep the prompt and evidence fixed across candidates.
That you benchmark the whole pipeline instead of swapping logos.
9. How do you handle stale documents?
Carry a source timestamp and a freshness policy on every chunk, boost recent documents where recency matters, and reindex on a schedule that matches the source. When two documents conflict, prefer the newer one and say so in the answer. Also surface the date in the citation, because a correct answer from an outdated policy is still wrong.
That you treat freshness as a retrieval property, not a model promise.
What mistakes show up in RAG interviews?
Treating RAG as a prompt trick
Retrieval quality, packing, and evaluation are most of the system. A clever prompt cannot recover a document that was never fetched, and a demo that only works on three hand-picked questions is not a pipeline.
Adding a vector database before measuring
A database is an operating choice; retrieval quality is the product problem. Start with a small index you can inspect, measure recall and grounded answers, then choose storage when the size or query load asks for it.
Comparing raw scores across collections
Cosine scores and BM25 scores move when the corpus changes, so a 0.9 from last month is not comparable with today. Compare ranks, or compare end-to-end outcomes, unless you control the collection and the scorer.
Packing to the last token
The answer needs room too, and an overflowing prompt loses its tail without an error. Leave headroom, put the strongest evidence early, and test the packed context as its own artifact rather than assuming it fits.
Evaluating only the final answer
An end-to-end score cannot tell you whether to fix retrieval or the prompt. Measure retrieval, packing, and grounding separately, then use the end-to-end rubric to confirm the fix actually reached the user.
What else do people ask about RAG?
Does RAG stop hallucinations?
No. It gives the model better evidence and a source to cite, which reduces unsupported answers when retrieval and grounding work. The model can still misread a passage, cite something that does not support the claim, or answer from memory. Treat grounding as something to measure, not a property you install.
Do I need a vector database?
Not to start. You need an index and a way to measure retrieval quality. A vector database helps when the corpus and query volume justify the operational cost; a small in-repo index with lexical search is enough to learn the pipeline and find the first failure.
What is the difference between RAG and fine-tuning?
RAG changes what the model can see at answer time. Fine-tuning changes how the model behaves by default. Use retrieval for facts, freshness, and citations, and fine-tuning for format, tone, and a narrow repeated task, and measure both against the same evaluation set before choosing.
How do I evaluate RAG without a labelled dataset?
Hand-build a small set of realistic questions and record the evidence you expect for each one. Score whether the evidence was retrieved, whether the answer was supported by it, and whether the required points were covered. Twenty careful cases reveal more than a large unlabelled sample, and you can grow the set from real failures.
What is reciprocal rank fusion?
It merges ranked lists by summing a score based on each document position, usually one divided by a constant plus the rank. It needs no score normalization, which is why it suits hybrid retrieval where lexical and vector scores live on different scales. It is a merging step, not a substitute for evaluating either retriever.
Written by the LearningTo team. The token, chunking, and evaluation guidance here mirrors the worked examples in the AI track; the planning ranges are labelled and the arithmetic is shown so you can check it.