Grouped-query attention

In plain multi-head attention, every query head carries its own key and value states, so the per-token cache is one key-value entry per head per layer. Grouped-query attention assigns several query heads to one shared key-value head, so the cache entry count per layer - the number of KV heads - drops. Multi-query attention is the limit case of one shared head. The quality cost is usually small because queries vary more than keys and values do, and the concurrency gain is direct.

The cache bytes per token fall in exact proportion to the KV head count: bytes per token = 2 (keys plus values) x layers x KV heads x head dimension x bytes per value. Halve the KV heads and the cache budget halves - no quantization of the cache itself, no precision change, purely an architecture choice made before training. That is why serving teams ask which attention variant a candidate model uses before sizing a fleet: it is a multiplier on every downstream memory number.

The trade is breadth against depth: with fewer KV heads the card holds more users per card, but attention has less state to distinguish contexts, which can show up as quality loss on tasks that genuinely need many distinguishable positions. The measured comparison is a concurrency-per-card figure against an eval pass rate, and the answer one model at a time - which is why two models of comparable quality at different KV head counts can price serving very differently.

Two variants of the same 70B-class model at 40 layers, 128 head dimension, fp16: one with 32 KV heads, one trained with grouped-query attention at 8 KV heads. A card has 9 GiB free for the cache after its weights.

1. Multi-head, 32 KV heads: 2 x 40 x 32 x 128 x 2 = 655,360 bytes, about 640 KiB per token, and one 4k-token user holds 4,096 x 640 KiB = 2,560 MiB.
2. Grouped-query, 8 KV heads: one quarter of the state, 160 KiB per token and 640 MiB per 4k user.
3. Concurrency: 9 GiB holds 9 / 2.5 = about 3 users under multi-head and 9 / 0.625 = about 14 under grouped-query - the same card, the same quality class, 4x the seats.

Dropping KV heads from 32 to 8 quarters the per-token cache, 2,560 MiB down to 640 MiB per 4k user - about 3 users per card becomes about 14 without touching precision.

Where this shows up

• LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
• Quantization & the Startup StackThe second serving job is not the fleet; it is the one card, when the budget is what a startup can put on the balance sheet instead of a Vercel bill. Open weights changed the question: the model can be downloaded, and the job becomes making it fit and making it fast on hardware a 24 GB card describes. Three compression decisions run in a fixed order. Weight precision comes first: a model of P parameters takes 2 x P bytes at bf16, 1 x P at INT8, and 0.5 x P at 4-bit, and because decode is bandwidth bound - the card re-reads every weight byte for each token - cutting the bytes in half doubles tokens/sec per stream exactly. An 8B brain goes from 16 GB to 4 GB, and a card re-reading 4 GB through a 1008 GB/s bus decodes 252 tokens a second per stream. KV precision comes second and is the cheaper knob: the per-seat rent of 128 KB a token at bf16 for an 8B-class model collapses quarter-price at 4-bit, which is what llama.cpp KV quantization and vLLM FP8 KV do, while costing about a quality point. Model size, the third knob, is the expensive one - but distillation is how startups move it: train a smaller student on the outputs of a bigger teacher and keep most of the quality at a fraction of the arithmetic. The follow-up interviewers run at startups is the trap answer: quantization is never one decision. Weights, KV, and context length are three separate dial groups, and the 4-bit-weights-on-4-bit-KV stack is the combination that lives just under the quality floor, which is why the quality regression has to be measured, not assumed - bad quantization scales have produced long-context regressions that dropped retrieval recall from 91 to 13 percent before a recalibration put it back. The three challenges below work the stack: the bytes of weights a precision choice buys, the seats a card grants after the per-seat KV rent, and the tokens/sec the memory bus lets through.

Related terms