Efficiency
FP8 quantization
Storing model weights as 8-bit numbers instead of 16-bit halves their size. A model that needed two cards shrinks onto one, and decode reads less data.
Deep learning weights ship in 16-bit floats as a default. Half the values fit in 8-bit with almost no visible quality change, and the memory halves with them: a 70-billion-parameter model goes from about 140 GB to about 70 GB. Quantization is the umbrella term; FP8 is the serving-era variant people actually deploy for both weights and, carefully, activations.
The reason it helps serving so much is that decode reads the entire model every step. Halve the bytes and you double the tokens-per-second ceiling from bandwidth alone, before touching any software. Halve the weights and you also double how many users fit next to the model before the KV cache runs out of room.
It is not free, and the honest answer names the costs. Extreme values (outliers) in activations can wreck accuracy, so FP8 requires calibrated scales computed from real activations, and some sensitive layers are held in higher precision. The KV cache can be kept in 8-bit too, which doubles concurrency again, at the cost of one more error source to measure.
The usual decision frame for an interview: quantize the weights when bandwidth or card count is the binding constraint; keep 16-bit when accuracy measurement, unusual numerical behavior, or vendor tooling makes the risk not worth it; and validate with your own eval set either way, not with a leaderboard.
A worked example
A 70B model on 80 GB H100 cards, each with about 3.35 TB/s of memory bandwidth, deciding between 16-bit and 8-bit weights.
- Weights in 16-bit: 70B x 2 bytes = 140 GB, which is 1.75 cards, so it needs two.
- Weights in 8-bit: 70B x 1 byte = 70 GB, which fits on one card with about 10 GB left for the KV cache.
- Decode is bandwidth-bound: 3.35 TB/s over 140 GB is 23.9 tokens per second; 3.35 TB/s over 70 GB is 47.9.
Same weights, half the bytes per element: 8-bit doubles the single-token decode ceiling and cuts the card count from two to one.
Where this shows up
- Quantization & the Startup StackThe second serving job is not the fleet; it is the one card, when the budget is what a startup can put on the balance sheet instead of a Vercel bill. Open weights changed the question: the model can be downloaded, and the job becomes making it fit and making it fast on hardware a 24 GB card describes. Three compression decisions run in a fixed order. Weight precision comes first: a model of P parameters takes 2 x P bytes at bf16, 1 x P at INT8, and 0.5 x P at 4-bit, and because decode is bandwidth bound - the card re-reads every weight byte for each token - cutting the bytes in half doubles tokens/sec per stream exactly. An 8B brain goes from 16 GB to 4 GB, and a card re-reading 4 GB through a 1008 GB/s bus decodes 252 tokens a second per stream. KV precision comes second and is the cheaper knob: the per-seat rent of 128 KB a token at bf16 for an 8B-class model collapses quarter-price at 4-bit, which is what llama.cpp KV quantization and vLLM FP8 KV do, while costing about a quality point. Model size, the third knob, is the expensive one - but distillation is how startups move it: train a smaller student on the outputs of a bigger teacher and keep most of the quality at a fraction of the arithmetic. The follow-up interviewers run at startups is the trap answer: quantization is never one decision. Weights, KV, and context length are three separate dial groups, and the 4-bit-weights-on-4-bit-KV stack is the combination that lives just under the quality floor, which is why the quality regression has to be measured, not assumed - bad quantization scales have produced long-context regressions that dropped retrieval recall from 91 to 13 percent before a recalibration put it back. The three challenges below work the stack: the bytes of weights a precision choice buys, the seats a card grants after the per-seat KV rent, and the tokens/sec the memory bus lets through.
- Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.
- LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.