Quantization error

Every quantization step trades memory for representable range. Storing weights in fewer bits means fewer distinct values each weight can take, so the model answers slightly differently than it did in full precision. Whether that difference matters is a question an eval set answers - a fixed list of prompts with known expected answers, re-run before and after the precision change - and the honest deployment treats the eval delta as part of the migration, not an afterthought.

The error is not spread evenly. A few sensitive layers or channels carry most of it, which is why calibrated methods that measure real activations and set per-channel scales often land close to full precision at 8 bits, while a naive per-tensor scale at 4 bits can move eval pass rates visibly. Attention internals are a known risk area for cache quantization for the same reason: attention scores come from small inner products, and narrowing the range of the elements feeding them shifts the scores.

The arithmetic is cheap and it frames the negotiation. Each halving of precision halves the weight bytes, which halves the memory bill and raises the memory-bound decode ceiling for the same card. The negotiation is in eval points: a step that buys a doubling of seats and concurrency at an unmeasured quality cost is a bet, and a step that buys it with a flat eval set is a decision. Production quantization is mostly the paperwork around that distinction.

One 70B model on a 160 GB two-card node moves down the precision ladder, with the decode ceiling read from the 3.35 TB/s card at each step.

1. fp16: 70B parameters x 2 bytes = 140 GB of weights, 160 - 140 = 20 GB left for the cache, and a decode ceiling of 3.35 TB/s / 140 GB = about 23.9 tok/s.
2. fp8 with calibrated scales: 70 GB of weights, 90 GB left, and the ceiling doubles to 3.35 TB/s / 70 GB = about 47.9 tok/s - the memory bill halved.
3. 4-bit: 35 GB of weights, 125 GB left, and the eval-set re-run is the step that decides whether this migration ships, because the memory savings are already certain.

Each precision step halves weights - 140, 70, 35 GB - and doubling decode ceilings along the way; the eval delta is the only part of the trade that is not arithmetic.

Where this shows up

• Quantization & the Startup StackThe second serving job is not the fleet; it is the one card, when the budget is what a startup can put on the balance sheet instead of a Vercel bill. Open weights changed the question: the model can be downloaded, and the job becomes making it fit and making it fast on hardware a 24 GB card describes. Three compression decisions run in a fixed order. Weight precision comes first: a model of P parameters takes 2 x P bytes at bf16, 1 x P at INT8, and 0.5 x P at 4-bit, and because decode is bandwidth bound - the card re-reads every weight byte for each token - cutting the bytes in half doubles tokens/sec per stream exactly. An 8B brain goes from 16 GB to 4 GB, and a card re-reading 4 GB through a 1008 GB/s bus decodes 252 tokens a second per stream. KV precision comes second and is the cheaper knob: the per-seat rent of 128 KB a token at bf16 for an 8B-class model collapses quarter-price at 4-bit, which is what llama.cpp KV quantization and vLLM FP8 KV do, while costing about a quality point. Model size, the third knob, is the expensive one - but distillation is how startups move it: train a smaller student on the outputs of a bigger teacher and keep most of the quality at a fraction of the arithmetic. The follow-up interviewers run at startups is the trap answer: quantization is never one decision. Weights, KV, and context length are three separate dial groups, and the 4-bit-weights-on-4-bit-KV stack is the combination that lives just under the quality floor, which is why the quality regression has to be measured, not assumed - bad quantization scales have produced long-context regressions that dropped retrieval recall from 91 to 13 percent before a recalibration put it back. The three challenges below work the stack: the bytes of weights a precision choice buys, the seats a card grants after the per-seat KV rent, and the tokens/sec the memory bus lets through.
• LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.

Related terms