Bytes of Weights a Precision Buys
easyThe first scaling decision on a consumer card is weight precision, because the model's weights are the memory bill that pays for everything else before a single seat exists. The arithmetic is one line: a model of P parameters at bf16 takes 2 x P bytes, at INT8 or FP8 1 x P, and at 4-bit (GPTQ or AWQ quantization) 0.5 x P. The numbers interviewers check: an 8B brain is 16 GB bf16 - too big to leave KV room on a 24 GB card - and 4 GB at 4-bit, leaving 18 GB for seats; a 70B brain is 140 GB bf16 and fits no card a startup can afford, while at 4-bit it is 35 GB and fits a 48 GB card with room for eight seats. Since decode re-reads every weight byte per token, each precision step also exactly doubles the decode rate, so this one knob is the whole budget story: memory, speed, and about one to three quality points. Implement weight_gb(params_b, bytes_per_param): return the GB of weights a model of params_b billions of parameters needs at that precision.
Run your code to see output