Design a RAG System: The Complete System Design Interview Walkthrough

"Design a RAG system" has become the most common AI system design interview question of 2026. It shows up at Amazon and OpenAI as a full system design round, at enterprise companies as the opening question in AI engineer loops, and increasingly inside general FAANG (Facebook/Meta, Apple, Amazon, Netflix, Google) system design interviews as roughly half of all loops now include an ML-adjacent question.

There is a reason interviewers love it: a retrieval-augmented generation system touches classic distributed systems (ingestion pipelines, indexes, caching, latency budgets), modern AI infrastructure (embeddings, vector search, LLM serving), and product judgment (grounding, hallucination control, cost per query) in a single 45-minute conversation. It cleanly separates candidates who have built with LLMs from candidates who have only read about them.

This guide walks through the full design the way it should unfold in a real interview — clarify first, then ingestion, retrieval, generation, and evaluation — with the follow-up questions interviewers actually use to probe depth.

Step 1: Clarify Before You Design

Jumping straight to "we'll embed the documents and use a vector database" is the fastest way to fail this question. The strongest candidates spend the first minutes pinning down the constraints that change the design. Ask:

• Corpus size and type. 50,000 internal wiki pages is a different system from 5 million PDFs, tickets, and web pages. Mixed formats mean a real parsing layer.
• Freshness. How quickly must a new or edited document become searchable? "Within 5 minutes" forces an incremental indexing pipeline; "nightly" allows batch.
• Query volume and latency. 300 QPS with a p95 of 1.2 seconds end-to-end is a realistic enterprise target — and it immediately constrains reranking depth and model choice.
• Multi-tenancy and access control. Enterprise RAG almost always needs per-document permissions. Retrieval must respect ACLs, not just relevance — leaking one confidential document is a company-ending bug for the product.
• Cost budget. A budget like $0.002 per query rules out sending 50 chunks to a frontier model on every request and pushes you toward caching, smaller models for reranking, and aggressive context pruning.

These are not stalling questions. Each one changes an architectural decision, and interviewers score you on making that connection explicitly. This clarify-first discipline is exactly what the interactive guides on TierOnePrep train — every system design question starts with a clarifying phase that locks the solution until you have asked the critical questions.

Step 2: The Ingestion Pipeline

The ingestion side prepares knowledge for retrieval. It has four stages:

Parsing and normalization. Convert every source format (PDF, HTML, tickets, docs) into clean text with metadata: source, author, timestamp, permissions. Real-world note worth saying in the interview: parsing quality caps the whole system — garbage extraction from PDFs cannot be fixed by better retrieval later.

Chunking. Documents are split into chunks small enough to retrieve precisely and large enough to preserve meaning. This is the tradeoff interviewers probe hardest:

Overlap between adjacent chunks (typically 10–20%) prevents answers from being lost across a boundary. If you want to build intuition here by writing the actual code, TierOnePrep has a dedicated coding question on exactly this: Text Chunking with Overlap for RAG.

Embedding. Each chunk is passed through an embedding model to produce a dense vector. Design points to mention: embedding model version must be stored with the vector (a model upgrade requires re-embedding the corpus, which is a batch backfill job), and embedding throughput — not vector insert speed — is usually the ingestion bottleneck.

Indexing. Vectors go into an approximate nearest neighbor (ANN) index — HNSW is the default choice for high recall at low latency — alongside a keyword index (BM25) over the same chunks. Storing both is deliberate; the next section explains why. For the full deep-dive on ANN indexes, sharding, and filtered search, work through Design a Vector Database.

Step 3: Retrieval — Hybrid Search and Reranking

Pure vector search fails on a predictable class of queries: exact identifiers, product codes, names, and rare acronyms that embedding models blur together. Pure keyword search fails on paraphrases and conceptual queries. Production RAG systems therefore use hybrid retrieval:

1. Run the query against both the BM25 index and the vector index in parallel.
2. Merge the two ranked lists (reciprocal rank fusion is the standard, parameter-free way).
3. Apply access-control filters — ideally pushed down into the index query itself, not applied after retrieval, so the system never ranks documents the user cannot see.
4. Rerank the top candidates (say, top 50) with a cross-encoder model that scores query-document pairs jointly, and keep the best 5–10 for the LLM context.

Reranking is the highest-leverage quality improvement in most RAG systems, and it is also where the latency budget goes: a cross-encoder over 50 candidates can cost 100–300ms. Say that tradeoff out loud — budget-aware reasoning is what separates senior answers.

Two standard optimizations worth naming: query rewriting (use a small model to expand or decompose the user's question before retrieval) and semantic caching (serve repeated or near-duplicate questions from a cache keyed on query embedding similarity instead of re-running the pipeline). The latter is its own interview-worthy component — practice it as code in Implement a Semantic Cache for LLM Responses.

Step 4: Generation and Grounding

The generation stage assembles a prompt — system instructions, the retrieved chunks with source metadata, and the user question — and calls the LLM. The design conversation here is about trust and cost:

• Citations. Instruct the model to cite chunk identifiers, and render those as links to source documents. Citations turn "the model said so" into "the model said so, based on this document you can check" — essential for enterprise adoption.
• Hallucination control. Grounding instructions ("answer only from the provided context; say you don't know otherwise"), a relevance threshold below which the system declines to answer rather than improvising, and answer-time checks that flag claims with no supporting chunk.
• Context budget. More chunks is not better. Irrelevant chunks actively degrade answers and every token costs money and latency. This is why reranking precision matters more than retrieval recall at the final stage.
• Serving. At high QPS the LLM serving layer is its own system — batching, KV caching, streaming. In an interview, name it as a component and offer to go deeper; if the interviewer takes you up on it, that is the LLM inference platform design, which we cover in a full separate guide.

Step 5: Evaluation — Where Most Candidates Fall Down

Ask engineers how they would know their RAG system is good, and most have no answer. Evaluation is consistently reported as the weakest area in AI engineering interviews, which makes it the cheapest place to stand out. Cover both halves:

Retrieval metrics (does the right chunk get retrieved?): recall@k and MRR against a golden set of query→document pairs. This golden set is the single most valuable eval asset a team builds, and it doubles as the regression suite for every chunking or embedding change.

Answer metrics (is the final answer good?): faithfulness (is every claim supported by the retrieved context?), answer relevance, and citation accuracy. Because outputs are non-deterministic free text, teams use LLM-as-judge scoring against rubrics, sampled human review, and online signals (thumbs ratings, deflection rates, follow-up-question rates).

The punchline to deliver in the interview: evaluate the stages separately. If faithfulness is high but answers are wrong, retrieval is failing; if retrieval recall is high but answers are wrong, the prompt or model is failing. A single end-to-end score cannot tell you which knob to turn. The metrics themselves are fair game as a coding exercise too — see Precision, Recall, and F1 Score.

Follow-Up Questions Interviewers Actually Ask

• "A document was updated 2 minutes ago and a user asks about it. Walk me through what happens." — tests the incremental indexing path and freshness SLA.
• "Your p95 latency doubled overnight. How do you debug it?" — tests per-stage observability: retrieval, rerank, and generation latencies must be tracked separately.
• "How do you handle a query about a confidential document the user can't access?" — ACL filtering at retrieval time; the correct answer must never depend on post-generation censorship.
• "Embedding model v2 is 10% better. How do you migrate?" — dual-index or versioned-collection strategy, shadow evaluation on the golden set, then cutover; never mix vectors from different models in one similarity space.
• "Cost per query is 4x over budget. What do you cut?" — semantic caching, smaller generation model with reranked tighter context, cascading (cheap model first, escalate hard queries).

How to Practice This

Reading a walkthrough is not the same as performing one. TierOnePrep's interactive system design questions put you in the interview seat: the prompt starts vague, the critical clarifying questions must be uncovered before the architecture unlocks, and difficulty-labeled follow-ups pressure-test the design — the same shape as the walkthrough above. Start with the interactive version of this exact question — Design a RAG System — which drills the clarifying phase, the hybrid-retrieval and reranking tradeoffs, and follow-ups like ACL enforcement and embedding-model migration under interview conditions. Then widen out: Design a Vector Database → Design an LLM Inference Platform → the RAG-adjacent coding questions (chunking, semantic cache) → the LLM & AI Engineering quiz to close knowledge gaps. If you are targeting a specific company, the OpenAI and Anthropic prep pages map which topics those loops emphasize.

FAQ

Is "design a RAG system" a machine learning question or a distributed systems question? Both, deliberately. Interviewers use it to test classic system design (pipelines, indexes, caching, latency budgets, multi-tenancy) and AI-specific judgment (chunking, hybrid retrieval, grounding, evaluation) in one question. Candidates who treat it as only one of the two leave half the rubric unscored.

Do I need to name specific vector database products in the interview? No — and at companies like Google, interviewers generally prefer reasoning about fundamentals (ANN indexes, HNSW behavior, filtering, sharding) over naming products. Understanding why an index choice matters beats reciting vendor names.

How is RAG different from fine-tuning, and which should I say? RAG injects knowledge at query time and handles fresh, permissioned, auditable data; fine-tuning bakes behavior and style into weights and cannot cite sources or respect per-user access. The strong interview answer is that they solve different problems and are often combined — fine-tune for format and tone, RAG for facts.

How much time should I spend on each stage in a 45-minute interview? A workable split: 8–10 minutes clarifying and scoping, 20–25 minutes on the core design (ingestion, retrieval, generation), and the remainder on evaluation, scaling, and follow-ups. Skipping either clarification or evaluation is the most common structural mistake.