Efficiency
KV cache quantization
Storing the per-token cache in 8 bits instead of 16 halves its memory, roughly doubling the concurrent users one card holds, at one measurable risk.
The KV cache usually ships in the same 16-bit precision as the weights, and it is the memory that caps concurrency. Halving the cache bytes per token - by storing keys and values in 8-bit - doubles how many conversations one card holds before it runs out, without touching the weights or the model at all.
The risk is numerical, and it lands in attention. Attention scores come from inner products of keys and queries, and half the representable range per element can shift them, especially with outliers. Real deployments therefore calibrate per-dimension scales over real activations and keep a close eye on their eval set, the same discipline quantizing weights needs.
It is worth knowing what the alternatives do, since a good interview answer lists them: fewer key-value heads (grouped-query attention) shrink the cache in the architecture; shorter context caps and sliding windows spend less cache per user; paged attention recovers reserved-but-unused memory. Cache quantization is the one that leaves both the model and the product untouched, which is why it is usually the first try.
A worked example
A model with a 160 KiB per-token 16-bit cache, 9 GiB free for the cache after weights, and 4,096-token conversations, before and after moving the cache to 8-bit.
- 16-bit cache: 4,096 x 160 KiB = 640 MiB per user, so 9 GiB holds 9 / 0.625 = about 14 conversations.
- 8-bit cache: 80 KiB per token, 4,096 x 80 KiB = 320 MiB per user, so 9 GiB holds 9 / 0.3125 = about 29 conversations.
- The concurrency ceiling roughly doubles by accepting one measurable source of precision error.
From about 14 to about 29 users on one card without touching the weights - the cache, not the model, was the concurrency budget.
Where this shows up
- Quantization & the Startup StackThe second serving job is not the fleet; it is the one card, when the budget is what a startup can put on the balance sheet instead of a Vercel bill. Open weights changed the question: the model can be downloaded, and the job becomes making it fit and making it fast on hardware a 24 GB card describes. Three compression decisions run in a fixed order. Weight precision comes first: a model of P parameters takes 2 x P bytes at bf16, 1 x P at INT8, and 0.5 x P at 4-bit, and because decode is bandwidth bound - the card re-reads every weight byte for each token - cutting the bytes in half doubles tokens/sec per stream exactly. An 8B brain goes from 16 GB to 4 GB, and a card re-reading 4 GB through a 1008 GB/s bus decodes 252 tokens a second per stream. KV precision comes second and is the cheaper knob: the per-seat rent of 128 KB a token at bf16 for an 8B-class model collapses quarter-price at 4-bit, which is what llama.cpp KV quantization and vLLM FP8 KV do, while costing about a quality point. Model size, the third knob, is the expensive one - but distillation is how startups move it: train a smaller student on the outputs of a bigger teacher and keep most of the quality at a fraction of the arithmetic. The follow-up interviewers run at startups is the trap answer: quantization is never one decision. Weights, KV, and context length are three separate dial groups, and the 4-bit-weights-on-4-bit-KV stack is the combination that lives just under the quality floor, which is why the quality regression has to be measured, not assumed - bad quantization scales have produced long-context regressions that dropped retrieval recall from 91 to 13 percent before a recalibration put it back. The three challenges below work the stack: the bytes of weights a precision choice buys, the seats a card grants after the per-seat KV rent, and the tokens/sec the memory bus lets through.
- LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.