Retrieval-augmented generation (RAG)
The first time a chat product answered something about last week, I assumed the model had somehow learned it. It had not. What happened instead is a simple arrangement the industry calls RAG, short for retrieval-augmented generation. The name is heavy but the idea is not: before the model sees anything, the system runs a search, grabs a handful of passages - short stretches of text from documents - that probably mention the answer, and pastes them into the prompt. The prompt is just the text the model reads, built by the system, and the model then writes its answer from what it just read. That arrangement is how products answer from fresh news and private company documents without ever retraining anything, because updating a list of documents is far cheaper than updating what a model remembers. The part I had backwards was the emphasis. Reading is the easy half; the model cannot complain about what it was given, so if the search grabs the wrong passages, the answer is still confident and now confidently wrong. Most questions about RAG quality are really questions about what got pulled into the prompt.
Most people have this backwards
Shrinking chunk size usually lowers the cost and latency of each query, not raises them.
The catch
This only holds while retrieval keeps fetching a fixed number of passages. Cut the documents into smaller chunks and each of the ten passages carries less text, so the prompt shortens and each query costs less. The index itself did grow, but the index is built offline and the query never pays for it. The intuition that smaller chunks must mean more work comes from thinking about the index instead of the query. What the cheaper prompt actually trades away is a quality risk you can feel in the demo below: the finer the cut, the more often a chunk slices straight through the passage an answer needed, and the model is left reading two fragments that used to be one.
Baseline first - what would you actually check?
A RAG product gets asked about something that happened last week. It answers instantly, describing a meeting that never happened, and nothing flags it. Where do you start digging?
Run the pipeline
Predict first
Retrieval keeps fetching the top 10 chunks. Shrink the chunk size. What happens to the prompt that generation reads?
A runnable model of the whole loop, on a pretend support KB. Pick a question, set the chunk size, and run the pipeline. Nothing here is randomized, so the same settings always produce the same answer, which is what makes it worth exploring: guess what happens as you drag the chunk size down, then drag it and see how close you were.
Toy RAG pipeline
Everything below runs on a scripted pretend pipeline. The relevance scores and the answers are written ahead of time. The part worth poking at is one variable: how much of each document a single chunk can carry.
Commit a guess above to start playing.
The knowledge base
Seven short fictional documents about Prizm, a code review bot. Cards below show abbreviated excerpts, so the character counts refer to the full pages, not to the excerpts.
Pick a question
Pick a question, set a chunk size, and press Run.
How it works
It helps to take the loop apart, because every failure I have hit lives in exactly one stage. Indexing runs beforehand: documents are split into chunks, each chunk becomes a list of numbers that roughly describes its meaning, and the numbers go into a searchable index. Retrieval happens when someone asks: the question becomes the same kind of list of numbers, and the index hands back whichever chunks sit numerically closest, usually the top ten. Generation goes last: the question and the ten chunks are formatted into one prompt, and the model writes an answer using only that window. What surprised me most is that the middle stage is a search engine problem, one of the oldest and least solved problems in software. The generation half is the least magical part of the whole stack, since it has no memory beyond its prompt and no way to ask for a second opinion.
The open-book exam
I picture the whole thing as an open-book exam. The model is the student, and the student is genuinely good at test-taking: a fast reader with clean synthesis and decent prose. But the exam is open-book, so the accuracy of the answers depends far more on how quickly the book opens to the right page than on how well the student crammed. Retrieval is the page-flipping and the index is the book. When the flip gets faster and more reliable, every answer gets better even though the student stayed exactly the same.
Where it breaks
The analogy breaks at the index. A human flips to the right page because an editor ordered the book that way, but a vector index opens to whatever page is numerically nearest to the question. Two passages can share most of their vocabulary and mean something different, and the index has no way to notice, so it lays the wrong page on the desk with total confidence. That is the gap the picture hides: the desk believes it is right. It is the reason RAG teams keep evaluation running after launch instead of trusting their search relevance.
Why interviewers lean on it
Interviewers ask about RAG because the tradeoffs are hard to fake. Saying the acronym is easy; the decisions underneath are not: how finely to slice the documents, whether search runs on meaning, on keywords, or on both at once, whether a second model gets hired to re-order the search results before the model reads them, and how the system keeps each user from being shown documents they have no right to see. Every one of those choices moves cost, wait time, and answer quality in a direction you are expected to name out loud. The difference between someone who has read about these systems and someone who has run one is usually that the second person can say which lever they turned and what it broke.
The follow-up chain
Interviewers do not stop at the surface answer. Answer the way you would out loud, or skip - skipping reveals the strong answer and ends the chain.
The interviewer's next question
Fuse how? The scores aren't comparable.
The interviewer's next question
Your vector search returned a document the querying user has no access to. Where did the permission check go wrong?
Now answer one you have not seen
A teammate proposes storing every chunk twice - the original text and a one-paragraph summary - and retrieving over both stores. Where does that design win, where does it break, and what would you measure before shipping it?
Expires fast - as of 2026-10-04
As of 2026-10, long-context prompts make "put the whole corpus in the prompt" viable for small, static document sets. The retrieval-first recommendation in this concept is for anything that must stay cheap per query, stay current as sources change, or respect document-level permissions - the three jobs retrieval does that a giant prompt does not.
As of 2026-10, hybrid retrieval (vector plus keyword) with rank fusion remains the common production baseline, with a reranking pass on the fused candidates as the usual next upgrade.