RAG failure modes: find the broken stage
A chatbot confidently described a vacation policy that did not exist, cited the policy page it supposedly came from, and nothing flagged it. That invented answer is the moment this page lives in. The systems behind it are simple to describe: before answering, the system searches the company documents and pastes what it found into the prompt - the text the model reads - so the model answers from real text instead of from memory. A search step in the middle creates exactly two ways an answer can go wrong. Either the evidence made it into the prompt and the model wrote the wrong thing anyway, and the fix lives in grounding - the practice of keeping the answer tied to the exact text you showed it, often by making every fact come with a citation you can check. Or the evidence never arrived, and no improved instruction fixes that, because the model was asked to answer from a shelf that did not hold the book. Two tools carry the whole diagnosis. A trace is the saved record of one run: what the user asked, what the search found, what got sent to the model, what the model said back. Opening it first and asking one question - was the answer sitting in the passages the search retrieved? - splits every failure into one of two debuggable halves, before anything gets changed. Working teams sort the halves further into short labeled piles: invented content, a search miss, a bad plan, a misused tool. Each pile is pushed down by its own measurement, which is why knowing which pile you are standing in decides the fix.
Most people have this backwards
Most reported hallucinations in a RAG product are retrieval failures the reporter never inspected, not model failures.
The catch
It holds once the corpus is bigger than a demo set and retrieval was never measured, because the model can only write from its window and a never-retrieved passage is indistinguishable from absence. The claim breaks exactly where the trace contradicts it: if the passage is in the prompt above the question and the model still writes against it, that is a true generation failure - and in every trace review I have read or run, that pile is the smaller one.
Baseline first - what would you actually check?
A RAG bot answers a question about a product feature that shipped last week, and describes a feature that never existed, in fluent prose. Nothing in the product flags it. Where does your investigation start?
Make each call, then see the verdict
Predict first
Your support corpus grows 10x today. The retriever and the top 10 window stay exactly as they are. Does the share of answers that carry the needed evidence in the prompt rise, fall, or stay flat?
Four broken answers from one support bot. Name the bucket for each before the verdict reads itself.
The trace shows the vacation policy page in the prompt, and the answer states a rollover limit the page contradicts.
Replaying the query with debug flags on, the perfect passage sits at rank 400 of the retrieval. The window holds the top 10. The answer hedged that it had nothing.
The bot answered the new refund window correctly on Monday. On Thursday it answers with last month's refund window as if it were current.
The trace shows the pricing agent calling its price lookup with a field name that does not exist. The tool fell back to a default. The answer states the default as fact.
How it works
The diagnostic runs in one direction, against the saved trace. Open the prompt that was actually sent - not the template, the assembled one - and read what the model received a moment before it answered. That single read splits every failure into two halves: the evidence arrived and the model wrote around or against it, or the evidence did not arrive and the model did what it could with empty shelves. From there the piles name themselves. An answer that invents a number even though the passages it needed were right there in the prompt is an invented-content failure, and it lives in the model's writing step. A passage that exists in your documents but ranked so low it never reached the window - search only shows the best few - is a retrieval miss. An answer that was correct last Monday and reads last month's policy today lives in the index pipeline: the index is the store the search looks in, a background job refreshes it when documents change, and that job missed. In systems where the model also calls tools, the trace adds two more piles: a planning failure picks the wrong next step, and tool misuse picks the right tool with bad arguments, or swallows a made-up default the tool returned. Teams keep the piles separate because each one is pushed down by its own measurement, not by one general fix. The order of operations is the whole discipline: nothing gets fixed before the trace read.
The intake desk
I picture debugging a broken RAG answer the way intake works at an emergency room. A patient arrives with one symptom: a confidently delivered answer a user checked and did not believe. The intake desk does not start treatment at first sight of the symptom, because the same fever stands behind a dozen different conditions, and the wrong treatment can look like progress for a week. The chart that matters is the one behind the patient: what did the body actually experience? The trace is the chart. Reading it takes minutes, and it usually names the department - generation, retrieval, or the index pipeline - before anyone has written a line of code. What I like about the picture is the discipline it buys: treatment before diagnosis is the prompt tweak, and I shipped several of those before anyone taught me the chart.
Where it breaks
The analogy breaks at the chart. A patient cannot lie to a blood test, but a trace can read healthy while the system is broken: the prompt shows ten passages, search looks full, and the bug is that passage three sat in an old copy of the index that nobody re-updated after the documents changed. The chart says the evidence arrived and the diagnosis is still missing. That is the gap the picture hides: reading the trace is necessary and not sufficient, which is why version labels travel with the text and with the copy of the index it came from.
Why interviewers lean on it
Interviewers lean on this concept because blaming the prompt before inspecting the search results is the number one tell that someone has read about these systems but never run one. What follows the diagnosis is quantitative on purpose: how often did search actually deliver the passages a labeled set of test questions needed? How many saved runs did you read before touching anything? Which failure pile shrank after your fix, and how do you know the others did not grow? A candidate who has run the loop answers with numbers and names the measurement per pile; a candidate who has not answers with component names and hopes. The debugging round is also a prioritization interview - four piles, one afternoon - so arguing which pile to fix first, from the counts in your saved runs, is the strongest signal in the room.
The follow-up chain
Interviewers do not stop at the surface answer. Answer the way you would out loud, or skip - skipping reveals the strong answer and ends the chain.
The interviewer's next question
A user reports that the bot made something up. Where do you even start?
Now answer one you have not seen
A teammate proposes fixing the support bot's hallucination reports by adding "you must cite your sources" to the system prompt. You review 30 reported traces first. What do you bring to the meeting, and what do you commit to measuring?
Expires fast - as of 2026-10-05
The four-bucket error taxonomy (hallucination, retrieval miss, planning failure, tool misuse) is quoted from a Snowflake FDE Applied AI posting fetched in October 2026 for the research behind this track.
As of 2026-10, trace tooling such as LangSmith, Arize, and OpenTelemetry-based setups is the common way teams replay a query with its full prompt visible; the diagnostic in this concept needs only a stack that saves the assembled prompt, which is the one requirement.