Quantization & the Startup Stack

The second serving job is not the fleet; it is the one card, when the budget is what a startup can put on the balance sheet instead of a Vercel bill. Open weights changed the question: the model can be downloaded, and the job becomes making it fit and making it fast on hardware a 24 GB card describes. Three compression decisions run in a fixed order. Weight precision comes first: a model of P parameters takes 2 x P bytes at bf16, 1 x P at INT8, and 0.5 x P at 4-bit, and because decode is bandwidth bound - the card re-reads every weight byte for each token - cutting the bytes in half doubles tokens/sec per stream exactly. An 8B brain goes from 16 GB to 4 GB, and a card re-reading 4 GB through a 1008 GB/s bus decodes 252 tokens a second per stream. KV precision comes second and is the cheaper knob: the per-seat rent of 128 KB a token at bf16 for an 8B-class model collapses quarter-price at 4-bit, which is what llama.cpp KV quantization and vLLM FP8 KV do, while costing about a quality point. Model size, the third knob, is the expensive one - but distillation is how startups move it: train a smaller student on the outputs of a bigger teacher and keep most of the quality at a fraction of the arithmetic. The follow-up interviewers run at startups is the trap answer: quantization is never one decision. Weights, KV, and context length are three separate dial groups, and the 4-bit-weights-on-4-bit-KV stack is the combination that lives just under the quality floor, which is why the quality regression has to be measured, not assumed - bad quantization scales have produced long-context regressions that dropped retrieval recall from 91 to 13 percent before a recalibration put it back. The three challenges below work the stack: the bytes of weights a precision choice buys, the seats a card grants after the per-seat KV rent, and the tokens/sec the memory bus lets through.

4-bit weights cut the memory bill 4x and double-pair decode speed, but the quality bill is real and stack-specific: INT8 costs about a point, 4-bit about three, and stacking 4-bit KV on 4-bit weights is the combination that crosses the floor. The professional answer is to pick a target metric, measure the stack against it, and keep a fallback config on disk - not to assume "quantized models are fine now".

A 96 GB unified-memory machine and a 24 GB consumer card both hold seats, but they decode differently: the unified-memory machine trades bandwidth (400 GB/s) for room, the consumer card trades room for bandwidth (1008 GB/s). Long-context pods need the VRAM, interactive pods need the bus, and a fleet plan that mixes both cards by workload is stronger than a fleet plan that buys one size.

Serving the 70B keeps full quality and pays in cards, concurrency, and speed; distilling to a 14B pays a one-time training bill and rents that back every request as faster decode and more seats. The breakeven is usage-weighted, and startups that measure it choose the student; the interview answer names the student's real failure mode - quality on inputs the teacher never generated.

Put Quantization & the Startup Stack into practice

The best way to lock in what you've learned is to write code. Solve real interview-style problems right now.