Serving
Prefix cache
A saved set of KV states for text that repeats across requests, usually a shared system prompt. A cached prefix is read, not recomputed, so it costs less.
Every request in most applications carries the same leading text: a system prompt with instructions, a persona, a tools list. Without caching, that identical text is prefilled again and again. A prefix cache keeps the KV states of the shared prefix and reuses them, so the model only prefills the part that actually changed.
The savings appear on both sides of the ledger. Time to first token drops, because the expensive prefill of maybe thousands of tokens disappears. And the bill drops, because providers bill cached input tokens at a large discount - reusing stored work is cheaper than recomputing it.
The design questions that follow it are the interview material. Hit rate is everything: a prefix only caches if requests actually share one, so long per-user histories may gain nothing while a fixed system prompt gains on every call. Cache eviction and capacity bound how long the states survive. And routing matters - the request has to reach the machine holding the cache, which is why provider docs sometimes ask you to pin traffic.
Watch the interaction with privacy and correctness: a cache holds computed state from one request and serves it under another, so tenants that truly must not share anything need distinct routing. The mechanics are otherwise exactly the KV cache reuse you would expect.
A worked example
An app sends 50,000 requests a day, each carrying the same 2,000-token system prompt. Assumed provider prices: $3.00 per million uncached prompt tokens and $0.30 when the prefix cache hits, with a 90 percent hit rate.
- Prompt tokens per day: 50,000 x 2,000 = 100 million.
- No cache: 100 million x $3.00 = $300 per day.
- Cache hits cover 90 percent: 90 million x $0.30 = $27; misses 10 million x $3.00 = $30.
- Total with the cache: $27 + $30 = $57 per day.
The prefix cache saves $243 per day on prompt tokens that all read the same - the hit rate, not the token count, decides the bill.
Where this shows up
- Cache-Aware Routing & AffinityThe next hidden cost in a serving fleet is where each request lands. Every arrival carries a shared opening - a system line, a tool spec, a project brief - about 1200 tokens of it, and the fleet pays to prefill that opening on whatever card the request lands on unless the opening is already resident there. This is what makes the cached-input discounts on frontier model pricing real: the preamble is resident on the one card that prefilled it, so the discount is not a memory feature, it is a routing guarantee. Cache-blind round-robin destroys the guarantee politely: an arrival meets the card holding its family one time in n, so a 16-card fleet re-prefills the same openings dozens of times a second no matter how large the scratch space is. The fix is one line of arithmetic - hash the prefix family, land on that family card - and its costs show up elsewhere: a hot prompt template now concentrates every arrival of its family on one card and strands the rest, which is why the busiest-card metric (decode headroom, capped near 60%) is a first-class check, and why sharding the hot family across two cards is the follow-up interviewers wait for: it halves the hot card load and costs one duplicate preamble copy. Budget closes the loop, because preambles are big - 1200 tokens of 70B-class KV at 320 KB per token is about 384 MB a copy, so an 8 GB prefix cache holds 20 families - and cards that run short evict silently, turning hashed arrivals back into prefills. The three challenges below size the mechanics: the hit share round-robin throws away, the copies a scratch of a given size holds, and the busiest-card share the hash leaves behind.
- Inference Economics & LatencyEvery time someone sends a message to an AI feature, the model has to run on that request, and the run has a price in dollars and a wait in seconds. The industry word for the run itself is inference: the model processing your prompt and producing an answer, and it is the line item that grows with usage. Where does the money go? Mostly into tokens, the word-sized pieces of text that both prompts and answers are made of. Input tokens are re-sent with every call; output tokens are generated one at a time at a premium rate; and a large discount called prompt caching applies only while the opening of your prompt matches what the provider already has stored, which one careless changing detail destroys. The wait decomposes into stages that scale differently: reading the prompt grows with prompt size, writing the answer grows with answer length, and a fixed network round trip is attached either way - and the stages disagree about which lever is worth pulling. Interviewers pursue this topic because cost and latency are the two numbers that decide whether an AI feature survives its first real month, and both are governed by arithmetic you can do on a whiteboard. The three challenges below work that arithmetic: the cache hit rate and the savings a rotating preamble quietly destroys, the share of a latency budget that writing the answer consumes, and the work a hard output cap clips off real requests.
- LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
Test yourself
- AI Serving Infrastructure Interview Questions2 questions on this term
- Quantization and Inference Cost Questions1 question on this term
- Scaling and Load Balancing Interview Questions2 questions on this term
- Latency Engineering Interview Questions2 questions on this term
- Cache Routing Interview Questions8 questions on this term
- Cost Metering Interview Questions3 questions on this term