Serving
Time to first token
How long a caller waits before the reply's first word: queue wait plus the prefill pass over the prompt. The latency users judge a service by.
Time to first token (TTFT) is the wait from the request arriving to the first token of the reply existing. It has two parts: queue wait while the server finds room, and prefill - one big pass over the whole prompt that writes the KV cache entries decode will read from. Everything the user perceives as "it is thinking" is these two parts.
Prefill dominates on long prompts because it is compute-bound: the whole prompt rides through the model in one pass, and its cost grows with prompt length. The single biggest lever is a prefix cache, which reuses the KV states of text that repeats across requests - a fixed system prompt, a large document - so only the new tail prefills. After that: shorter prompts, chunked prefill scheduling that hides prefill inside decode traffic, and faster cards.
TTFT is also the metric users judge first, because a reply that starts fast and streams steadily at a modest per-token rate feels better than one that waits three seconds and types at triple the speed. Watch it at p95: the worst wait, not the average, is where caching misses, queueing, and cold starts appear.
A worked example
A 3,000-token prompt on an H100 (about 989 TFLOPs/s dense 16-bit), a 70B model at 140 GFLOPs per token, with 0.08 s of measured queue wait.
- Full prefill: 3,000 x 140 GFLOPs = 4.2e14 FLOPs; at 989 TFLOPs/s that is about 0.42 s.
- Time to first token: 0.42 s + 0.08 s queue = about 0.5 s.
- A prefix cache hits on 2,700 of the 3,000 tokens: only 300 new tokens prefill, 300 x 140 GFLOPs = 4.2e13 FLOPs, about 0.04 s.
- The first token now arrives after about 0.12 s - the cache, not a faster GPU, removed the wait.
Prefill is most of the wait on long prompts, and a cache on the repeated part removes almost all of it - for a fraction of the cost of new hardware.
Where this shows up
- LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
- Cache-Aware Routing & AffinityThe next hidden cost in a serving fleet is where each request lands. Every arrival carries a shared opening - a system line, a tool spec, a project brief - about 1200 tokens of it, and the fleet pays to prefill that opening on whatever card the request lands on unless the opening is already resident there. This is what makes the cached-input discounts on frontier model pricing real: the preamble is resident on the one card that prefilled it, so the discount is not a memory feature, it is a routing guarantee. Cache-blind round-robin destroys the guarantee politely: an arrival meets the card holding its family one time in n, so a 16-card fleet re-prefills the same openings dozens of times a second no matter how large the scratch space is. The fix is one line of arithmetic - hash the prefix family, land on that family card - and its costs show up elsewhere: a hot prompt template now concentrates every arrival of its family on one card and strands the rest, which is why the busiest-card metric (decode headroom, capped near 60%) is a first-class check, and why sharding the hot family across two cards is the follow-up interviewers wait for: it halves the hot card load and costs one duplicate preamble copy. Budget closes the loop, because preambles are big - 1200 tokens of 70B-class KV at 320 KB per token is about 384 MB a copy, so an 8 GB prefix cache holds 20 families - and cards that run short evict silently, turning hashed arrivals back into prefills. The three challenges below size the mechanics: the hit share round-robin throws away, the copies a scratch of a given size holds, and the busiest-card share the hash leaves behind.
- Serving AI ReliablyThe AI feature launched ten days ago and is the quietest thing you operate: the dashboard is a strip of green graphs, no complaints, nothing shipped this week. Then at 3am something changes, and running this feature is different because the dependency is a model you neither wrote nor can restart: a provider answers your users, and the provider can change its capacity, its pricing, or the model itself without telling you. So the board grows meters that watch the call rather than your code. p95 latency, the wait that 95 of every 100 requests stay under, catches a capacity squeeze that the median wait, p50, smooths right over. The error share catches a rate limit, a provider ceiling on how many calls fit in a minute that a launch surge can trip with zero code changes. Daily spend catches a prompt edit that quietly broke the prefix cache, the discount a provider grants while your requests resend the same opening text. And an eval set, a small fixed list of prompts with expected answers that your code reruns on a schedule, catches the scariest one: a silent quality drop after the provider swapped the model, the failure users notice and no latency meter ever blinks. What you do about it is the fallback ladder, an ordered list of answers to serve while the model misbehaves: retries with backoff first, then a smaller or older model, then a cached reply, ordered so the user meets slow long before they ever meet gone. Pinning, writing the exact model version into your config instead of floating on the provider default, is what turns a swap back into a rollback. Interviewers ask what you watch and what you do at 3am: a specific ladder is the answer of someone who ran a live AI feature, and we would monitor things is the answer of someone who has not. The three challenges below grade exactly that: the order you triage a crowded alert board in, the p95 and eval math that tells you to roll a model version back, and the share counters that say whether the ladder is holding.