RAG System Design in Production
Building a support chatbot over a company wiki is a weekend project; keeping it correct for thousands of users is the interview. RAG, short for retrieval-augmented generation, is the standard architecture behind that chatbot: the product first searches the wiki for relevant passages (retrieval), then hands those passages to the model with the question, so the answer is written from the passages rather than from whatever the model happened to memorize during training. Once basic retrieval works, production raises four hard questions. Results often come from two search methods at once, keyword matching and meaning matching, and their scores are in different units, so what merges the two ranked lists into one ordering? Out of thousands of candidate passages, how do you re-check the top few with a slower, sharper scorer without paying to re-check all of them? How does a user fail to see documents their account is not allowed to read? And when a policy page is rewritten or deleted, when does the old copy stop being served? Interviewers ask these in sequence because they separate people who have watched a RAG system misbehave in production from people who built one in a notebook. The three challenges below drill the ranking formula that merges the two lists, the recall metric interviewers expect every candidate to know, and the half-finished migration of the embedder, the model that converts text into those number lists, which quietly breaks search for as long as it lasts.
Key Concepts
- 1Reciprocal rank fusion merges two ranked lists by position, not score: first place is first place on any list, so positions compare even when raw scores are in different units. A document adds 1 / (60 + its rank) from every list it appears in
- 2Rerank wide and cheap first, then narrow and expensive: the merged list is a candidate pool, and a cross-encoder (a model that reads the question and the passage together) re-checks only the finalists, where its slowness is affordable
- 3Permission filters must run inside retrieval, before ranking. Filtering afterward means the top spots were already spent on documents the user should never have reached
- 4Deleted and rewritten documents keep getting served until the pipeline removes them from the search index; freshness is a pipeline job the search step cannot do alone
- 5Recall@k is the retrieval metric interviewers treat as homework: a number saying what fraction of the documents a human marked relevant made it into the top k results. Evidence that never reached the prompt cannot be used, no matter how good the model is
Run the pipeline yourself
The full RAG walkthrough
You have felt this failure even if you never named it: a chatbot answers fluently and confidently, and the answer is simply wrong, because the model was writing from memory instead of reading your documents. Retrieval-augmented generation (RAG) is the fix: before the model answers, a search step finds the passages in your documents most likely to hold the answer, and the model is told to build its reply from those passages rather than from memory.
This whole board is that pipeline, small enough to follow a single query through. The trace below is the written record of one query moving through the six stages, and the knobs are the settings a search engineer tunes: your documents are pre-cut into chunks (short passages, the unit the search retrieves), one stage looks them up (the retrieval mode knob speaks in shorthand: BM25, the classic keyword search; dense, which matches passages by meaning; or the two combined), and a rerank is a second pass that re-orders the retrieved passages by usefulness. Two counters are worth translating before you touch anything: recall is how many of the passages that actually contain the answer the search brought back, and groundedness is how much of the final answer is backed by those retrieved passages (p95 is the response time of the slowest query in twenty). Nothing here is scripted: every number comes out of the same pipeline a real search would run, at toy scale.
Pick a question
The knobs
Chunking
Chunking mode: fixed · moves answer groundedness
Chunk size (tokens): 512 · moves p95 latency
Chunk overlap (tokens): 64 · moves cost per 1k queries
Heading trails: false · moves p95 latency
Retrieval
Retrieval mode: dense · moves retrieval recall
Retrieval k: 10 · moves retrieval recall
RRF constant (hybrid): 60 · moves p95 latency
Rerank
Reranker: false · moves p95 latency
Rerank pool size: 20 · moves p95 latency
Assembly
Chunks sent to the model: 5 · moves answer groundedness
Window order: natural · moves answer groundedness
Permissions
Permission filtering: none · moves retrieval recall
Index
Embedding version: single · moves retrieval recall
Set the knobs, pick a question, then press Run. Every number below is computed from the same pipeline; nothing is scripted.
cost per 1k queries
$1.000
+0.00
p95 latency
0 ms
+0.00
retrieval recall
-
answer groundedness
-