All question banks

Latency Engineering Interview Questions

These questions are about the wait: how long before the first token, how steady the typing is after it, and what the server was doing during both. Every answer states its assumptions and shows the arithmetic, so you can take a latency question apart the way an interviewer expects it taken apart. Terms link to the glossary where they are defined.

Warm-up

1. What is the difference between time to first token and time per token?

Time to first token is the wait before the reply starts: queue plus prefill over the whole prompt. Time per token is the pace while it streams: one decode step each, bounded by bandwidth. They move with different levers (a cache accelerates the first, weight quantization accelerates the second), and users experience most complaints as the first one.

Worked answer

  1. A 300-token reply: the first token arrives at 0.5 s, the whole reply at 0.5 + 300 / 30 = about 10.5 s, decoding at the measured 30 tokens per second.
  2. Cutting decode to 15 tokens per second moves the finish to 0.5 + 300 / 15 = about 20.5 s, while the first token is unchanged.
  3. Cutting the first-token wait to 0.1 s with nothing else changed moves the start earlier by 0.4 s and leaves the pace alone.

The wait and the pace are separate budgets: 15 tokens per second changed only the finish, 0.1 s of TTFT changed only the start.

Core

2. What lowers time to first token?

Ordered by effect: a prefix cache on the shared, repeated part of the prompt; shorter prompts; scheduling prefill so it shares the card with decode instead of blocking it (chunked prefill); and faster or quantized compute. The first two are free and the last one is capital - knowing their order is the interview answer.

Worked answer

  1. A 2,000-token prompt at 140 GFLOPs per token: 2,000 x 140 GFLOPs = 2.8e14 FLOPs, about 0.28 s at 989 TFLOPs/s.
  2. Add the stated 0.08 s of queue wait: about 0.36 s to the first token.
  3. With a prefix cache hitting all but 200 tokens: 200 x 140 GFLOPs = 2.8e13 FLOPs, about 0.03 s of prefill, so about 0.11 s total.
  4. The cache removed nearly twice what double-bandwidth hardware would have: 0.36 to 0.11 from the cache, versus prefill halving 0.28 to 0.14.

From 0.36 s to about 0.11 s on cache alone - the repeated prefix is most of the prefill, so caching it beats any hardware change.

Core

3. Why does a user see the model typing slower than the benchmark says?

Because the benchmark measured the model and the user measures the whole path. Under load, per-user speed drops as concurrency rises; caches miss and recompute; network and client rendering add their own delay; and a proxy that buffers the stream delivers chunks instead of tokens. The model may be meeting its number while the product is slow.

Worked answer

  1. The card claims about 48 tokens per second per user for this model at low load.
  2. Batching splits the same weight read: the measured per-user stream is the stated 40 tokens per second at batch 16, and the stated 20 tokens per second at batch 128.
  3. A buffering proxy that flushes 20 tokens at a time adds 20 / 20 = 1 s of perceived delay before every chunk the user sees.

The model number, the loaded number, and the perceived number are three different things - measure at the client edge or the argument is about nothing.

Hard

4. Your p50 time per token is fine but p95 doubles at peak. Where do you look?

At contention, not at the model. Candidates in order: queueing (requests waiting for admission), batch crowding (more users sharing the same bandwidth), KV memory pressure (preemptions and recomputation), and admission bursts that make cache routing miss. Plot p95 against queue depth and against concurrency; whichever correlates is the mechanism, and the fix follows from it.

Worked answer

  1. Baseline, a quiet machine: 24 tokens per second per user at batch 8, and 8 x 24 = 192 tokens per second in aggregate.
  2. At peak concurrency the batch doubles to 16, and the p95 user sees a stated 20 tokens per second.
  3. At batch 32 with preemption the stream falls toward the stated 10 tokens per second, and 32 x 10 = 320 tokens per second in aggregate.

The tail moved because the same bandwidth was split among more users, and then some users paid re-queues - contention, not the model.

Warm-up

5. What does streaming change for users, and for your measurements?

For users, it moves the wait: instead of one long silence ended by a wall of text, they wait once briefly and read from there. For measurements, it splits latency into time to first token and time per token, and it moves the measuring point: buffering anywhere between the model and the user shows the stream as chunky, so the honest reading is at the client edge.

Worked answer

  1. A 500-token reply at 24 tokens per second with 0.5 s of first-token wait finishes at 0.5 + 500 / 24 = about 21.3 s.
  2. Buffered, the user reads nothing until 21.3 s.
  3. Streamed and read at 5 words per second, the user finishes reading at about the same time the model finishes writing - about 21 s.
  4. The first visible content arrives at 0.5 s in the streamed version, a 21 s reduction in wait before anything appears.

Roughly the same total time, over 20 s less silence before the first word - the perceived wait, not the total cost, is what streamed.

Core

6. How much latency does batching cost the individual user?

Almost none at moderate batch sizes, rising thereafter. The weight read is shared, so a user in a batch of 16 runs at nearly the single-user rate while the card emits sixteen times the aggregate. Push the batch much higher and cache reads plus scheduling start charging back - the observed point where the user stream slows is the real capacity of the setup, not the paper aggregate.

Worked answer

  1. Bandwidth ceiling per user: 3.35 TB/s / 70 GB = about 48 tokens per second.
  2. Batch 16 measured per user: a stated 40 tokens per second, 40 / 48 = about 83 percent of the solo ceiling.
  3. Batch 64 measured per user: a stated 22 tokens per second, 22 / 48 = about 46 percent.
  4. Aggregate still rises in both (16 x 40 = 640; 64 x 22 = 1,408) while the user pays the second one.

Batching is nearly free to the user near batch 16 and increasingly priced above it - measure where per-user speed bends, not just the aggregate.

Core

7. Where does a latency budget come from, and who owns it?

It comes from the product commitment, decomposed: what the user is promised end to end, split into the first-token allowance and the streaming pace, at a stated percentile. Ownership follows the split: the waiting-to-admit and prefill pieces belong to the serving team, the streaming pace to the model and scheduler, the rendering to the client. A budget nobody owns is a wish.

Worked answer

  1. A 30 s end-to-end promise for a 600-token reply, prefill allowance 0.4 s at p95.
  2. Decode budget: 30 - 0.4 = 29.6 s for 600 tokens.
  3. Required pace: 600 / 29.6 = about 20 tokens per second per user at p95.
  4. A proposed change delivering 15 tokens per second at p95 misses the budget: 600 / 15 = 40 s of decode, 40.4 s end to end.

The budget is a floor of about 20 tokens per second at p95, and the proposal at 15 misses it by a third - arithmetic, so ownership has something to own.

Hard

8. How do you cut latency without buying hardware?

Spend cache and precision before dollars: a prefix cache on the repeated prompt, an 8-bit cache and 8-bit weights to double the bandwidth ceiling, shorter contexts per request, and speculative decoding where the workload has bandwidth headroom. Speculative decoding is the one that needs a spare draft pass; the first three are configuration. Order them by expected effect per risk, then measure at the user.

Worked answer

  1. Baseline: a 70B model in 16-bit weights decodes at 3.35 / 140 x 1,000 = about 24 tokens per second.
  2. An 8-bit weight change halves the bytes per read: 3.35 / 70 x 1,000 = about 48 tokens per second.
  3. Speculative decoding stacks the stated 2.53 accepted tokens per draft pass on the 8-bit baseline: 48 x 2.53 = about 121 tokens per second as the new ceiling.
  4. A prefix cache hitting 2,700 of a 3,000-token prompt cuts prefill from 0.42 s to about 0.04 s, which is where the first-token wait goes.

About 24 tokens per second became 48 in 8-bit weights with a speculative ceiling near 121, and the first-token wait fell from about 0.42 s to about 0.04 s - all software, hardware bill unchanged.

Core

9. What network overhead shows up in the client-observed latency?

The path from the server to the user adds every delay the model never sees: connection setup, TLS handshakes, proxy buffering, transport idle time between chunks, and distance. On a streaming reply each of those taxes the first token and then every subsequent chunk, which is why the same serving stack measures differently across regions, and why the client edge is where latency numbers must be read.

Worked answer

  1. A TLS setup plus connection setup of a stated 0.15 s comes before the server even receives the request.
  2. Server-side first-token wait: 0.11 s (cache hit case).
  3. Chunks flushed per 10 tokens at 24 tokens per second: 10 / 24 = about 0.42 s between chunk starts, so the last of a 12-chunk reply lands 11 x 0.42 = about 4.6 s after the first chunk.
  4. Client-observed first content: 0.15 + 0.11 = about 0.26 s after taps begin, of which the server contributed 0.11 s.

Well over half of a fast reply's perceived starting delay was network, not model - measure from the client or the server gets blamed for someone else's seconds.

Hard

10. How would you serve a draft experience with a 2-second first token and a 10-second total for 1,000 tokens?

Derive the budgets first: prefill plus queue must fit inside 2 s, and streaming must sustain 1,000 tokens in roughly 8 s, which is 125 tokens per second per user - well past a single card. So the answer is a combination: quantized memory, tensor parallelism or speculation for the decode rate, and a prefix cache or trimmed prompt for the first-token target, then confirm each with the arithmetic above.

Worked answer

  1. Decode budget: 10 - 2 = 8 s for 1,000 tokens = 125 tokens per second per user.
  2. One H100 with 8-bit weights tops at about 48 tokens per second per user, about 2.6x short of 125.
  3. Tensor-parallel across 4 cards: each reads 17.5 GB per step, 3.35 / 17.5 x 1,000 = about 191 steps per second, minus a stated 15 percent collective overhead = about 163 tokens per second.
  4. 163 covers 125 with about 30 percent headroom, and a 2,000-token prompt prefills in about 0.28 s, inside the 2 s first-token budget.

Four-way tensor parallel at 8-bit reaches about 163 tokens per second per user against the 125 the budget demands, and prefill is not the binding constraint.

Need the vocabulary first? The glossary defines each of these terms with a worked example, and the other banks cover the other half of the interview.