All AI interview guides

AI engineer interview guide

AI System Design Interview

The short answer. An AI system design interview is a product design interview with a bill and a latency budget attached. You will be asked to design a support assistant, a document search feature, or an agent that takes actions, and the interviewer will push on quality, latency, cost, and failure. Start with the user and the constraint, not the model.

Updated 2026-11-05About 10 minutes to readReviewed against the LearningTo AI track

What is different about AI system design?

The output is not deterministic, so the specification has to be an evaluation. Two runs can look plausible while one is unsupported. The model is also an external dependency you do not control, with its own latency tail, price, rate limits, and outages. That means the design needs an eval loop, a fallback, and a cost model from the start.

The familiar system design questions still apply: data flow, storage, queues, timeouts, retries, and observability. What changes is that quality is a first-class budget next to latency and cost, and a change can improve one while quietly damaging another. A senior answer makes that triangle explicit and says who owns each number.

What should you clarify in the first five minutes?

Ask who the user is, what task they are doing, and what a wrong answer costs. Then ask about freshness, volume, peak traffic, latency expectations, privacy, and whether the system can take actions. For a support assistant, a wrong refund is different from a wrong article link. For an internal search tool, a boring answer is often acceptable.

Write the assumptions down and pick one primary success metric plus one guardrail metric. State the smallest baseline that could pass: a prompt with curated context, a lookup, or a small retrieval set. Say what you are deferring and why. The interviewer learns more from a clear constraint than from a large architecture.

How do you budget latency and cost?

Split the latency budget before you pick components. Time to first token, decode time, retrieval, prompt assembly, guardrails, and network all spend from the same p95 target. A streaming answer can feel fast even when total time is long, but the tail still matters when a request blocks a queue. Cache prefixes and repeated documents.

For cost, estimate tokens per request, requests per day, and cache hit rate, then price the model and the retrieval infrastructure together. Compare cost per successful task, not cost per token, because a cheap model that needs two retries can lose. An illustrative example: 1,000 requests at 2,000 input tokens is 2 million input tokens before caching; a 50 percent prefix-cache hit removes half of that line, not half of the answer quality work.

Where do failures come from and what do you do?

Group failures by stage: retrieval, prompt and context, model, tool, and serving. Each needs a different signal and a different fallback. A retrieval miss might serve a clarifying question; a provider timeout might route to a smaller model; a tool failure might return a safe message instead of retrying a destructive action. A fallback that has never been tested is just a second bug.

Then add the boring guarantees: timeouts, bounded retries with idempotency keys, a circuit breaker, and a queue with backpressure. Keep a rollback path for prompt and model changes, because a prompt is code. Decide what pages a human, what only logs, and what the product shows the user when the model is unavailable.

How do you evaluate and launch it?

Build an offline evaluation set from realistic tasks before the design review. Score the things the product cares about: task success, groundedness, refusal when evidence is missing, and the guardrail metric you named. If a judge scores the answers, calibrate it against human labels and report agreement. Add fresh cases each release so the set measures generalization rather than memory.

Launch behind a canary with online metrics and a human review queue for a sample. Compare task success, latency, cost per task, and escalation rate against the baseline. Version the prompt, the model, and the retrieval index together, so a regression can be traced. Define the rollback trigger before the launch, not after the first incident.

What questions come up in an AI system design interview?

1. Design a documentation assistant for 50,000 help articles.

Clarify the support goal, languages, freshness, and whether answers can be wrong or must escalate. Propose retrieval over the articles with citations, a prompt that refuses when evidence is missing, and a path to a human. Split the latency budget across retrieval, prefill, decode, and guardrails, and estimate cost per answer. Then name the eval set, the fallback, and what you would watch after launch.

Whether you cover retrieval, quality, latency, cost, and escalation before the model choice.

2. How do you set an SLO for a chat feature?

Pick a user-perceived milestone, such as time to first token for a streamed answer, and a total completion target, then set the percentile that matches the product risk. Keep separate targets for retrieval and guardrail stages so an internal regression is visible. Add a quality guardrail, because a fast answer that is wrong still fails the SLO that matters.

That you treat quality as part of the service objective.

3. Walk me through the latency budget for a 2-second p95 target.

Reserve a slice for the client and network, a slice for retrieval and cache lookup, a slice for prompt assembly and guardrails, and the rest for prefill and decode. Say which pieces stream so the user sees motion early. If the arithmetic does not fit, cut context, add a cache, or choose a faster path, and state the tradeoff.

That you can turn an SLO into component budgets.

4. How would you cut the bill in half without losing quality?

Measure cost per successful task, then attack the largest line. Prefix and document caching, context trimming, a smaller model on easy routes, batching, and quantization are the usual levers. Hold an evaluation set fixed and verify quality and p95 before shipping. A cheaper model that increases escalations or retries is not cheaper at the task level.

That you optimize unit economics without hiding the quality trade.

5. When would you fine-tune instead of retrieve?

Fine-tune when the gap is behavior: format, tone, a narrow repeated classification, or a task where prompt examples are expensive. Retrieve when the gap is knowledge, freshness, or citations. Run the same evaluation on both, and count the retraining and deployment cost of a tune, because it is a new artifact to own.

That you match the lever to the failure, not to a preference.

6. How do you handle a model provider outage?

Have a tested degraded path before the outage. That can be a second provider with a smaller model, a cached answer for repeated questions, a retrieval-only response that cites sources, or a clear message that the feature is unavailable. Timeouts and a circuit breaker keep the failure from consuming the queue. Rehearse the switch and monitor the fallback quality separately.

That you design for dependency failure, not just happy-path throughput.

7. How do you protect a tool-using agent from prompt injection?

Assume every fetched document and tool result is hostile. Keep instructions and data separate, let documents carry no authority, allowlist tools, validate arguments, and require confirmation for destructive actions. Add a loop and cost budget, and log the full chain. The prompt can reinforce the rule, but the enforcement belongs at the tool boundary.

That you know a model cannot be the security boundary.

8. What do you log and alert on?

Log the request, the retrieval trace, prompt and model versions, tool calls and results, latency per stage, token cost, and the guardrail outcome. Alert on error rate, p95, queue depth, provider failures, and a quality signal such as escalation rate or judge score. Keep private content out where policy requires it, and tie every alert to an owner and a runbook.

That you can operate the system, not only draw it.

9. How do you roll out a prompt change safely?

Treat the prompt as code: version it, review it, and run it against the fixed evaluation set before it reaches real users. Ship behind a canary, compare quality, latency, and cost against the baseline, and keep the previous version ready to restore. Record the prompt version on every request so a regression can be traced to a deployment.

That you apply release discipline to non-code artifacts.

What mistakes show up in AI system design interviews?

Designing the prompt before the evaluation

Without a success definition, the prompt cannot be judged. Name the task, the rubric, and a few example inputs and outputs first, then choose the model and the retrieval stack that meet them.

Sketching without latency or cost

An architecture that has no p95 and no cost per task is incomplete for an AI role. Put a number on the box, even a rough one, and say what it is made of. The interviewer will push on the largest line.

Assuming a stronger model fixes retrieval

A better model cannot read evidence that was never fetched. Diagnose the stage first; if retrieval recall is the problem, a model swap spends money and leaves the failure in place.

A fallback nobody tested

A second provider or a cached response only helps if it satisfies the same contract. Test the degraded path, monitor its quality, and say what the user sees when both paths fail.

No owner for quality regression

Latency and errors have obvious alerts; quality drift needs one too. Name the evaluation, the sampling rate, the human review, and the person who decides to roll back before launch.

What else do people ask about AI system design?

Do I need to know GPU internals?

You need enough to reason about latency and cost: how batching, KV cache, quantization, and context length move the numbers. You do not need to design hardware, but you should be able to explain why a longer context or a larger batch changes the bill and the tail.

Can I use rough numbers in the design?

Yes, if you label them as assumptions and keep the arithmetic visible. Say what the number depends on, then show how a change moves it. An explicit estimate that can be checked is stronger than a precise-looking number with no source.

How much API detail should I include?

Cover the contract between components: timeouts, retries, idempotency, streaming, and error shapes. For an MCP-based design, also cover the transport and whether the server is stdio or HTTP. The interviewer is testing whether the system can fail cleanly, not whether you can write an OpenAPI schema from memory.

What should I say about scale?

Name the bottleneck and the smallest measurement that would confirm it. Requests per second, context length, cache hit rate, and provider limits usually decide the shape. Scale honestly: a plan that works at one thousand requests a day can be different from one that works at a million, and saying so is part of the answer.

How do I close the design interview strongly?

Restate the tradeoffs you chose, the metrics that would tell you it is working, and the rollback you would use. Name one thing you would change with more time and one risk you would test first. A clean close is evidence that you can own the system after the diagram.

Written by the LearningTo team. The budgets here are illustrative and labelled as such; the serving and economics topics carry the full worked arithmetic.