All question banks

AI Serving Infrastructure Interview Questions

These are the questions that come up when a company is actually running models for users: how the request moves through the server, what limits concurrency, and where the throughput comes from. Every answer below is plain language, and the ones with arithmetic show the full number walk so you can check the math yourself. The terms link to the glossary where they are defined.

Warm-up

1. Walk me through what happens to a request between arrival and the first token.

The server admits the request when there is room, runs one prefill pass over the whole prompt - all the prompt tokens ride through the model together, writing a KV cache entry for each - and then emits the first token. Prefill is one big parallel computation, so the wait for the first token is mostly the prefill time. Everything after that is decode: one token per step, each step reading the weights and the cache built so far.

Worked answer

  1. Compute per token for a 70B model: 70B parameters x 2 FLOPs = 140 GFLOPs.
  2. Prefill of a 2,000-token prompt: 2,000 x 140 GFLOPs = 280,000 GFLOPs = 2.8e14 FLOPs.
  3. One H100 at about 989 TFLOPs/s of dense 16-bit: 2.8e14 / 9.89e14 = about 0.28 s of compute.
  4. Then decode runs at what memory bandwidth allows: 3.35 TB/s / 140 GB = about 24 tokens per second.

The prompt is one large compute-bound pass costing about 0.28 s, and the reply then streams at the memory-bound rate of about two dozen tokens per second.

Core

2. What actually limits how many users one GPU can serve?

Two limits stack. Model weights are read once no matter how many users there are, but the KV cache is per user, so memory for the cache is the concurrency ceiling. And memory bandwidth caps how fast the cache and weights can be read, which caps tokens per second. Which limit binds first depends on context length: longer contexts make the cache the binding one.

Worked answer

  1. Per token: 2 (a key and a value) x 40 layers x 8 KV heads x 128 dims x 2 bytes = 160 KiB.
  2. One user at 4,096 tokens of context: 160 KiB x 4,096 = 640 MiB of cache.
  3. The card has about 9 GiB free for the cache after weights, so 9 / 0.625 = about 14 users.

Fourteen concurrent users at 4k context - the weights are not the problem, per-user cache memory is.

Core

3. Why does batching raise aggregate throughput so much more than per-user speed?

Because decoding is memory-bound: each step pays to read the whole model, and the read cost is the same whether the batch has one user or sixteen. Batching multiplies the useful output of the same read without multiplying the read. Users still see roughly the same speed until the cache reads and scheduler overhead start to add up, so the aggregate gain is nearly free at moderate batch sizes.

Worked answer

  1. One user, 70B model in 8-bit on an H100: 3.35 TB/s / 70 GB = 47.9 tokens per second.
  2. At batch 16 the weights are read once per step for all sixteen: 47.9 x 16 = about 766 tokens per second in aggregate.
  3. Per-user speed stays close to 47.9 until KV cache reads and scheduling overhead consume the slack.

Sixteen users cost about the same bandwidth as one - the weight read is shared, so batching converts idle bandwidth into aggregate tokens.

Core

4. What does continuous batching change, and what goes wrong if the scheduler admits too eagerly?

It re-marks the unit of scheduling: instead of starting a fixed batch and waiting for all of it to finish, every decode slot is refilled the moment it empties. GPU capacity stops idling behind the longest request in the batch. Admit too eagerly and the opposite failure appears: the KV cache fills, running requests get preempted and re-queued, and everyone pays the prefill twice.

Worked answer

  1. Capacity in a 25 s window with 8 slots at 40 tokens per second: 8 x 25 x 40 = 8,000 generated tokens.
  2. Static batching runs one group of 8 to the end, so only 8 requests complete, and early finishers idle their slots.
  3. Continuous batching refills slots immediately: 8,000 tokens of output over the same window.
  4. At 500 tokens per response: 8,000 / 500 = 16 completed responses instead of 8.

Same GPU, same window, about twice the completed questions answered - because no slot sits idle waiting for a long request.

Core

5. How does paged attention raise memory utilization without changing the model?

Serving stacks used to reserve each request a worst-case slab of KV cache, and real conversations never fill it, so most of the reserved memory did nothing while requests lived. Paged attention cuts the cache into small blocks and grows each sequence block by block, so the waste drops from a big fraction of a worst case to one partially used block per request.

Worked answer

  1. Worst-case reservation: 4,096 tokens x 160 KiB = 640 MiB per request, 6.25 GiB for ten.
  2. Actual conversations average 1,200 tokens: 1,200 x 160 KiB = 187.5 MiB in real use.
  3. Utilization: 187.5 / 640 = about 29 percent; the other 71 percent of the reservation idles.
  4. With 16-token blocks, tail waste is at most 15 unused slots x 160 KiB = about 2.3 MiB per request.

The same memory that hosted 10 reserved slabs can host several times more users once reservations stop guessing 4,096 tokens for every conversation.

Warm-up

6. When does a prefix cache pay, and when is it useless?

It pays when requests share a long, identical leading text - a common system prompt is the classic case - because every hit reuses stored work instead of prefilling it again. It is useless when each request leads with unique text: a per-user rolling history has no shared prefix, so the hit rate is near zero and the cache only adds capacity and routing overhead.

Worked answer

  1. Prompt tokens per day: 20,000 requests x 1,500 shared prompt tokens = 30 million.
  2. No cache: 30 million x $3.00 = $90 per day.
  3. At an 80 percent hit rate: 24 million x $0.30 = $7.20; misses 6 million x $3.00 = $18.00.
  4. Total with the cache: $7.20 + $18.00 = $25.20 per day, a saving of $90 - $25.20 = $64.80.

Sixty-five dollars a day saved on text that never changed between requests - the hit rate, not the token count, decides the bill.

Hard

7. How does speculative decoding give a speedup without changing model quality?

A small draft model proposes several tokens ahead, and the real model verifies all of them in one pass that would otherwise have produced one token. Accepted proposals ride along free; the first rejected one is corrected in the same pass. Because every shipped token was scored by the real model, the output distribution is the same as plain decoding - the speedup comes from spending idle bandwidth on parallel verification.

Worked answer

  1. Expected tokens per verified pass with a 70 percent acceptance rate and 3 drafted tokens: 1 + 0.7 + 0.49 + 0.343 = 2.53 with the bonus token counted.
  2. Ideal ceiling on a 24 tokens/s baseline: 24 x 2.53 = about 61 tokens per second.
  3. A realistic 1.5x to 2x after the draft pass and wider verify batches lands near 40 tokens per second.

Roughly 40 instead of 24 tokens per second, with text identical to what the big model writes alone.

Hard

8. Your GPU has free compute but is out of KV memory. What are your options?

Cut the bytes per cached token or add cheaper bytes. Quantize the KV cache to 8-bit, which halves cache size and roughly doubles concurrency; shrink the admitted context window; let paged attention recover the reserved-but-unused memory; or add memory, a second card, or route long-context users elsewhere. Offloading the cache to CPU memory works but stalls decode, so it is the last resort.

Worked answer

  1. Cache memory available: about 9 GiB. In 16-bit at 160 KiB per token: 9 GiB holds a 4k context for 9 / 0.625 = 14.4, so about 14 users.
  2. 8-bit cache halves the cost to 80 KiB per token: 9 GiB now holds 28.8 users worth of 4k context.
  3. Concurrency roughly doubles, for the price of one extra precision to validate.

From 14 to about 29 users worth of capacity on the same card, by halving the cache bytes rather than buying a second GPU.

Core

9. Which metrics tell you a serving stack is healthy, and which one do you watch first?

Time to first token at p95 (prefill health and prefix-cache hits), inter-token latency at p95 (decode health under load), tokens per second per GPU (capacity), prefix hit rate (economy), queue depth and preemption count (scheduler pressure). Watch p95 time to first token first: users feel it directly, it moves when prefill contention or a cold cache appears, and it degrades before anything else.

Hard

10. Why is decode memory-bound while prefill is compute-bound, and how do serving stacks treat the two phases differently?

Decode produces one token per weight read, so arithmetic intensity is one FLOP per byte for a 16-bit model - far below the memory-compute crossover, which means bandwidth is the wall. Prefill runs thousands of tokens against the same single weight read, pushing intensity past the crossover into the compute-bound range. Stacks exploit that: chunks of prefill are run when the card has idle compute between decode steps, so long prompts do not have to block short ones.

Worked answer

  1. Ridge point of an H100: 989 TFLOPs/s / 3.35 TB/s = about 295 FLOPs per byte.
  2. Decode intensity, 16-bit 70B model: 140 GFLOPs per 140 GB read = 1 FLOP per byte.
  3. Prefill intensity, 2,000 tokens in one weight pass: about 2,000 FLOPs per byte, past the ridge.
  4. Decode ceiling: 3.35 TB/s / 140 GB = 23.9 tokens per second; prefill instead approaches the 989 TFLOPs/s compute peak.

Decode is pinned about 300x below the ridge and rides bandwidth; prefill rides compute. The two phases optimize for different budgets, so servers schedule them separately.

Need the vocabulary first? The glossary defines each of these terms with a worked example, and the other banks cover the other half of the interview.