Serving
KV cache
The model's running notes on text it has already read, kept so each new token reuses them. It grows with context length and caps how many users one GPU holds.
A language model reads a conversation token by token. Before it predicts the next token, every layer computes a key and a value vector for each word so far, which is what lets attention look back at earlier text. Those keys and values are the KV cache. If the model kept recomputing them for every new token, a 5,000-token conversation would redo the work of all 5,000 tokens on every single step, and nobody would get an answer.
So servers keep the cache in GPU memory and only compute the key and value for each new token as it arrives. That is the good news and the bad news. The good news: the tenth token of a long reply costs about the same compute as the first. The bad news: the cache grows without bound as the conversation grows, and it has to stay resident for as long as the user might send another message.
The size per token is fixed by the model shape: two vectors (a key and a value) for every layer and every key-value head, times the bytes per element. Multiply that by the context length and you get the memory a single conversation occupies. A 40,000-token context on a typical large model can hold more memory than the model weights themselves.
That is why KV memory, not model size, usually decides how many users one GPU can host. Interviewers like this point because it separates people who think of the model as the only resident data from people who have actually sized a serving stack. Related mechanics: paged-attention for how the cache is allocated, and the roofline-model for why its reads also cost time.
A worked example
Ten users on one 80 GB card, where the model and other allocations leave about 9 GiB free for the KV cache. The model has 40 layers, 8 key-value heads of 128 dims each, and a 16-bit KV cache.
- Per token: 2 (a key and a value) x 40 layers x 8 heads x 128 dims x 2 bytes = 163,840 bytes, which is 160 KiB.
- One user at 4,096 tokens of context: 160 KiB x 4,096 = 640 MiB.
- Ten concurrent users: 640 MiB x 10 = 6,250 MiB, which is 6.25 GiB of cache.
- With about 9 GiB free for the cache, the card hosts 9 / 0.625 = about 14 such users, and the fifteenth is the one that fails.
The KV cache is the concurrency ceiling. Weights fit once; cache fits per user, and it scales with the context you allow.
Where this shows up
- LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
- Design ChatGPT: Tiers & Rate LimitsDesign ChatGPT is the most-cited systems design question at AI companies, and it is really one question: when everyone arrives at once, who gets served, who waits, and who is told no, all without the paying tier ever noticing. The answer interviewers grade against has three parts. First, three pools: a small provisioned pool whose cards exist for one tier only, a larger shared pool where paid requests carry priority over free ones, and the queue itself, which lives in card memory, not in a message broker. Second, a token-bucket rate limiter at the gateway, held in Redis for atomicity under concurrent writers, that admits free arrivals at a chosen share of demand and reconciles the token count after each grant, so a burst of arrivals is answered instantly without consulting a GPU. Third, shedding: a request refused at the gateway with a 429 costs nothing; the same request admitted into a saturated pool holds an expensive seat and lengthens every wait behind it, which is why the tail wait punishes utilization non-linearly, and why an incident that removes 30% of capacity can turn a 97 millisecond paid wait into an 850 millisecond one. Interviewers test this because every knob in the design is a money or trust decision: renting burst capacity for a launch day, tightening the bucket over a jammed pool, or letting the free tier shed so the subscribers do not notice an outage. The three challenges below work the mechanics: the admit count a token bucket grants, the queue wait the utilization formula quotes, and the shed arithmetic that keeps the shared pool from jamming.
- Quantization & the Startup StackThe second serving job is not the fleet; it is the one card, when the budget is what a startup can put on the balance sheet instead of a Vercel bill. Open weights changed the question: the model can be downloaded, and the job becomes making it fit and making it fast on hardware a 24 GB card describes. Three compression decisions run in a fixed order. Weight precision comes first: a model of P parameters takes 2 x P bytes at bf16, 1 x P at INT8, and 0.5 x P at 4-bit, and because decode is bandwidth bound - the card re-reads every weight byte for each token - cutting the bytes in half doubles tokens/sec per stream exactly. An 8B brain goes from 16 GB to 4 GB, and a card re-reading 4 GB through a 1008 GB/s bus decodes 252 tokens a second per stream. KV precision comes second and is the cheaper knob: the per-seat rent of 128 KB a token at bf16 for an 8B-class model collapses quarter-price at 4-bit, which is what llama.cpp KV quantization and vLLM FP8 KV do, while costing about a quality point. Model size, the third knob, is the expensive one - but distillation is how startups move it: train a smaller student on the outputs of a bigger teacher and keep most of the quality at a fraction of the arithmetic. The follow-up interviewers run at startups is the trap answer: quantization is never one decision. Weights, KV, and context length are three separate dial groups, and the 4-bit-weights-on-4-bit-KV stack is the combination that lives just under the quality floor, which is why the quality regression has to be measured, not assumed - bad quantization scales have produced long-context regressions that dropped retrieval recall from 91 to 13 percent before a recalibration put it back. The three challenges below work the stack: the bytes of weights a precision choice buys, the seats a card grants after the per-seat KV rent, and the tokens/sec the memory bus lets through.