AI engineer interview guide
AI Engineer Interview Questions
The short answer. An AI engineer interview tests whether you can build a product on top of a model without inventing answers, blowing the latency budget, or shipping a surprise bill. Expect questions about retrieval, prompts, evaluation, agents, serving, and cost, then one design exercise that ties them together. The answers below are the ones we would give out loud, short enough for a 60-second response and specific enough to survive the follow-up.
What does an AI engineer interview actually test?
Most loops test four layers in order. First the model-facing layer: can you write a prompt, choose context, and explain why the model saw what it saw? Second the data layer: can you retrieve and rank evidence, and do you understand that retrieval sets the ceiling on answer quality? Third the evaluation layer: can you prove the system improved instead of saying it feels better? Fourth production: latency, cost, failure modes, and what you would watch after launch.
A strong candidate moves between those layers without prompting. Asked about a wrong answer, they ask where in the pipeline it went wrong. Asked to cut cost, they ask which quality metric must hold. That habit is what the scenario questions are looking for, more than a memorized definition of RAG.
Which AI topics come up most often?
Our AI track is organized around 16 topics, and six of them carry most interview weight: semantic search and document QA, prompting and context windows, evaluation and judge calibration, agents and stopping, inference economics, and guardrails. Serving internals and quantization appear when the role owns infrastructure, and they reward the same arithmetic.
For each topic, prepare one concrete story: a bug you found, a metric you changed, a tradeoff you chose. Interviewers use the vocabulary to open the story, then listen for the decision inside it. A candidate who can recite the RAG diagram but cannot say what they would log has only half an answer.
How do you answer a scenario question you have never built?
Start with the user, the task, and the cost of being wrong. Then state the constraint you are designing against: a quality bar, a p95 latency target, a monthly budget, or a safety rule. Without one of those, any architecture sounds reasonable and none can be judged.
Next propose the smallest baseline that could work: a prompt with curated context, a lookup against structured data, or a small retrieval set. Say how you would measure it, then name the two most likely failures and the cheapest test for each. Finish by saying what you would defer until the baseline is measured.
What math should you be able to do on a whiteboard?
You should be able to price a request from tokens, split a latency budget across retrieval, prefill, decode, network, and guardrails, and estimate a cache hit rate. You should also be able to explain recall@k, judge agreement, and pass@k versus pass^k without looking them up.
Use the interviewer numbers when they give them. When they do not, state an assumption and keep the arithmetic visible. Our worked examples always show the assumption in the scenario, because a cost number without its price and volume is not an answer.
How do you talk about failure without sounding vague?
Name the stage before the fix. A retrieval miss, a ranking error, context truncation, a stale source, a prompt that ignores evidence, a tool call with bad arguments, or a serving timeout each produce a different symptom and a different test. 'The model hallucinated' is not yet a diagnosis.
Then say how you would confirm it: inspect the retrieved set, read the cited span, replay the prompt, or check the trace for a timeout. A short answer that names one stage, one test, and one fix is stronger than a long tour of every possibility.
What questions come up in an AI engineer interview?
1. Walk me through how you would build a support assistant that answers from documentation.
Clarify the corpus, the users, and what a wrong answer costs. Chunk and index the docs, retrieve a small candidate set, rank it, and pack the best evidence under a token budget with citations. Prompt the model to answer only from the supplied passages and to say when evidence is missing. Then build a small evaluation set, measure retrieval and answer quality separately, and add guardrails before launch.
Whether you separate retrieval, grounding, and evaluation instead of treating the answer as one black box.
2. When do you not need RAG?
When the facts fit in a prompt you control, when the data is already structured and a lookup returns the answer, or when the task is style and behavior rather than knowledge. Retrieval adds an index, a ranking step, and a new failure surface, so it earns its place when the corpus is large, private, or changing faster than you can redeploy.
That you choose the cheapest lever instead of applying retrieval to every problem.
3. How do you choose between retrieval and fine-tuning?
Retrieval changes what the model can see; fine-tuning changes how it behaves. Use retrieval for facts, freshness, and citations, and fine-tuning for format, tone, and a narrow repeated task. If a prompt plus retrieval passes the eval, stop there. Measure both against the same test set, and remember that a tune adds a retraining path you now own.
Whether you know the lever maps to the failure mode, not to fashion.
4. What does recall@k measure, and how can it mislead you?
It is the share of relevant documents that appear in the top k results. High recall does not guarantee the answer will use them: a relevant passage can be ranked low, truncated by the context budget, or contradicted by a stale document. Measure recall and the final grounded answer together, and inspect failures by stage.
That you know a retrieval metric is necessary but not sufficient.
5. How do you evaluate an answer when there is no single correct string?
Write a rubric with a few binary checks: is the answer supported by a cited passage, does it cover the required points, does it refuse when evidence is missing. Score a sample by hand, then calibrate any judge against those labels and report agreement. Rotate in fresh cases so the set does not become a memory test.
Whether your evaluation would survive a skeptical review.
6. What is an agent loop, and when should it stop?
An agent is a loop that chooses a tool, reads the result, and decides whether to continue. It should stop on a verified success, a clear failure, a step or cost budget, or a repeated action that signals a loop. Name the stop condition before the demo, because an agent without one is just an open-ended bill.
Whether you design the stopping rule and the budget, not only the tool calls.
7. A RAG answer cites the wrong paragraph. Where do you look first?
Separate retrieval from grounding. Check whether the cited passage was in the retrieved set: if not, the problem is recall or ranking. If it was there, check whether the prompt buried it, the chunk split it, or the model ignored it. Then check whether the passage actually supports the claim, because a confident citation is not evidence.
That you debug by stage instead of rewriting the prompt and hoping.
8. How would you cut inference cost without losing quality?
Measure the baseline first, then attack the biggest line. Prompt caching and context trimming often beat model swaps, and batching or quantization can cut serving cost if the quality eval holds. Set a quality floor and a latency ceiling before the change, and compare cost per successful task rather than cost per token.
Whether you optimize the right metric and keep quality in view.
9. What guardrails belong around a tool-using assistant?
Treat all fetched text as untrusted data, not instructions. Allowlist the tools, validate arguments before execution, require confirmation for destructive actions, and cap the loop. Log tool calls and results, and test with adversarial documents. The guard belongs at the trust boundary, not in a single prompt sentence.
That you put the control where the boundary is instead of trusting the model.
What mistakes show up in AI engineer interviews?
Treating the model as the only variable
When an answer is wrong, candidates reach for a better model or a longer prompt. Most failures start in retrieval or context packing, and a stronger model cannot read a passage that was never fetched.
Quoting a metric without defining it
Accuracy, quality, and hallucination rate mean different things in different answers. Define the check and the sample before you quote a number, or the interviewer will ask and the answer will collapse.
Designing the big system first
An architecture diagram with six services is not a baseline. Start with the smallest version that can be measured, then add components when a test shows what is missing.
Optimizing before evaluating
Swapping models, adding a reranker, or tuning chunk size without a fixed evaluation set produces motion, not progress. Build the test set before the change so you can tell whether the change helped.
Skipping failure and rollback
A design that only describes the happy path reads as incomplete in a senior loop. Name the timeout, the fallback, the metric that pages someone, and what you would roll back first.
What else do people ask about AI engineer interviews?
Do I need an ML degree to pass an AI engineer interview?
No. The interviews reward product and backend judgment applied to model behavior: retrieval, evaluation, latency, cost, and safety. You still need to reason about the model as a component with failure modes, and you should be able to explain every metric you use.
How long should an answer be?
Aim for 60 to 90 seconds, then pause and offer the follow-up. State the decision, the constraint, and one failure you would test. Interviewers learn more from a short answer you can defend than a long one you cannot.
What portfolio project is worth building?
A small retrieval system with citations, a real evaluation set, and a table of latency and cost per request. The size matters less than the evidence: show the failure you found, the change you made, and the metric that moved.
Is prompt engineering enough on its own?
No. Prompting is one layer, and it cannot fix a retrieval miss or prove quality. Pair it with evaluation, context budgeting, and a serving plan, then describe the prompt as a versioned artifact rather than a trick.
What follow-up should I expect?
How would you measure it, what would you do when it fails, and what would you cut for budget. Prepare one concrete answer for each, and you will be ready for most of the second half of the loop.
Written by the LearningTo team. We build the AI interview track and the question banks this guide links to; when an answer here and a lesson disagree, the lesson carries the worked example.