All question banks

Quantization and Inference Cost Questions

These questions are about the money and the precision: what a token costs to serve, when 8-bit weights are safe, and when renting beats owning. Each answer states its assumptions and shows the arithmetic, so you can quote the numbers in an interview and defend them. Terms link to the glossary where they are defined.

Warm-up

1. What changes when weights move from 16-bit to 8-bit, and what stays the same?

Weight memory halves and, because decode reads every weight every step, the bandwidth-limited decode ceiling doubles. Model capacity does not change: it is the same 70B parameters, stored in half the bytes. Quality usually holds with calibrated scales, though a few sensitive layers may stay in higher precision, and your own eval set is the judge.

Worked answer

  1. A 70B model in 16-bit: 70B x 2 bytes = 140 GB.
  2. A 70B model in 8-bit: 70B x 1 byte = 70 GB.
  3. Decode ceiling on a 3.35 TB/s H100: 3.35 / 140 x 1000 = 23.9 tokens per second in 16-bit; 3.35 / 70 x 1000 = 47.9 in 8-bit.

Half the memory, double the single-user decode ceiling, and the same model - the read shrinks, the work does not.

Core

2. Why does 8-bit need calibration when 16-bit never seemed to, and what breaks if you skip it?

A 16-bit float has enough range and precision that one shared scale per tensor is a non-issue. In 8-bit you are spending that headroom, and real activations contain outliers - a few values enormously larger than typical - that would crush the useful range of everything else if one scale covered them. Calibration runs real data through the model to fit per-tensor or per-channel scales; activations usually need more care than weights. Skip it and the model typically degrades sharply rather than slightly, which is why 8-bit goes live only after an eval set says the numbers still hold.

Core

3. Is 8-bit the stopping point? What about 4-bit?

It is the current workhorse for serving, not a law. 4-bit shrinks weights again by half and can fit far larger models per card, at a bigger accuracy risk and with more exotic kernels. The honest decision frame is per deployment: compute the card count and cache headroom each precision buys, measure quality on your own eval set, and pick the smallest precision that passes. For the KV cache, 8-bit is a common, low-risk win; 4-bit cache is more aggressive.

Worked answer

  1. The same 70B model in 4-bit: 70B x 0.5 bytes = 35 GB.
  2. On an 80 GB card, 4-bit weights leave 80 - 35 = 45 GB for the KV cache and activations, versus about 10 GB under 8-bit weights.
  3. That is about 4.5x the cache headroom, at a bigger quality risk to validate.

Precision is a dial between accuracy and bytes per card - 4-bit buys huge cache headroom but costs more eval work than FP8.

Core

4. What does it cost to serve a million tokens yourself?

Divide the hourly cost of the hardware by the tokens it emits per hour, and say your utilization assumption out loud. Self-hosting has no list price: the per-token cost is a ratio, and idle time, preemptions, and ops staff all sit in the numerator. The honest version of the number is always "at this achieved throughput".

Worked answer

  1. Assume one H100 rented at $2.50 per hour, running a workload that sustains 400 tokens per second of output.
  2. Tokens per hour: 400 x 3,600 = 1.44 million tokens.
  3. Cost per million output tokens: $2.50 / 1.44 = $1.74.

About $1.74 per million output tokens at 400 tokens per second sustained - halve the achieved throughput and the cost per million doubles.

Hard

5. When does renting an API beat running your own GPU?

Compare the API rate per million tokens against your own division of hourly cost by achieved tokens per second. Your machine wins when it stays busy enough; below the break-even throughput, the API is cheaper because your idle seconds are still billed. Add what renting includes for free: capacity for spikes, uptime, and no ops work - all of which move the break-even point.

Worked answer

  1. Own an H100 at $2.50 per hour; your API alternative costs $4.80 per million output tokens (assumed rate).
  2. Break-even throughput: $2.50 per hour must buy 2,500,000 / 4.80 = about 521,000 tokens per hour to tie the API price.
  3. Per second: 521,000 / 3,600 = about 145 tokens per second of sustained output.

Below about 145 tokens per second of sustained output, the $4.80-per-million API is cheaper than an idle $2.50-per-hour card you run yourself.

Hard

6. Two rental quotes, different card counts - how do you compare them with one number?

Tokens per dollar: aggregate tokens per second times 3,600, then hourly cost divided by that hourly token count. It collapses card count, hourly rate, and throughput into one comparable figure. Then sanity-check the assumptions behind each quote: sustained aggregate output is not the same as a burst on an empty machine, and prefill-heavy workloads measure differently from decode-heavy ones.

Worked answer

  1. Option A: 8 cards at $2.50 per hour each = $20 per hour, sustaining 3,000 tokens per second in aggregate.
  2. Tokens per hour: 3,000 x 3,600 = 10.8 million, so $20 / 10.8 = $1.85 per million output tokens.
  3. Option B: 2 larger cards at $5.00 each = $10 per hour, sustaining 1,200 tokens per second.
  4. Tokens per hour: 1,200 x 3,600 = 4.32 million, so $10 / 4.32 = $2.31 per million output tokens.

Option A is $1.85 per million against Option B at $2.31 - the fleet with more cheap busy hours wins even though it costs more per hour.

Core

7. Where does prompt caching show up on a monthly bill?

On the input line, at a steep discount per cached token. Multiply the shared prefix by the request volume to get the tokens at stake, then apply the hit rate: the discount only lands on what actually hits the cache. The reason it matters monthly is that prompt tokens are the part of a request that repeats, while completions never do.

Worked answer

  1. A million requests a month, each with a 1,200-token shared prompt: 1,000,000 x 1,200 = 1.2 billion prompt tokens a month.
  2. Uncached at an assumed $3.00 per million: 1,200 million x $3.00 = $3,600 per month.
  3. At an 85 percent hit rate: 1,020 million x $0.30 = $306; misses 180 million x $3.00 = $540.
  4. Total: $306 + $540 = $846 per month, a saving of $3,600 - $846 = $2,754.

The shared prefix went from a $3,600 line item to $846 a month, because 85 percent of it is never prefilled again.

Hard

8. Why do serving shops run mixture-of-experts models even though the whole model barely fits?

Because memory and compute separate. A mixture stores a very large parameter count but only activates a small slice per token, so you pay full memory rent for capacity and only partial compute per token. The trade makes sense when the extra expert capacity improves quality more than a dense model of the same active size would - and it stops making sense when memory per card is the binding constraint, since the resident parameters crowd out the KV cache.

Worked answer

  1. A synthetic mixture: 5B shared parameters plus 24 layers of 4 experts at 3B each, top-2 routing.
  2. Resident: 5B + 24 x 4 x 3B = 293B parameters.
  3. Active per token: 5B + 24 x 2 x 3B = 149B, about half the resident count.
  4. In 16-bit, 293B parameters take 586 GB - about 8 80 GB cards - while each token only computes like a dense 149B model.

Twice the resident parameters of its active compute: capacity is bought with memory, latency is paid in routing, and the router decides who works.

Hard

9. Long-context users are slow and expensive. What is happening, and what do you change?

Their KV cache dwarfs everyone else's, so a few long conversations crowd out many short ones, and every decode step reads a longer cache, which slows inter-token latency for that user. The usual set of changes: quantize the KV cache, cap the context you accept and route anything longer to bigger machines, let paged attention recover the memory, and price long contexts so they carry their own cost.

Worked answer

  1. Cache cost per token stays 160 KiB (40-layer, 16-bit cache); only the length changes.
  2. One user at 32,768 tokens: 160 KiB x 32,768 = 5.0 GiB of cache - eight times a 4k user.
  3. In about 9 GiB of free cache memory, that is 9 / 5 = 1.8, so roughly one long-context user at a time versus 14 at 4k.

The same memory that held 14 concurrent 4k sessions holds one or two 32k sessions - which is why long contexts get routed, capped, or priced separately.

Hard

10. Your machine gets 100 tokens per second and users say it is slow. Memory-bound, compute-bound, or misconfigured?

Measure before blaming hardware. Compute the bandwidth ceiling for your model and compare the measured rate against it: far below the ceiling with idle compute usually means too little batching or a scheduling gap, not a memory wall. If you are near the ceiling, you are memory-bound and the fixes are quantization, shorter contexts, or a bigger card. Either way, say the arithmetic, not just the label.

Worked answer

  1. Per-card ceiling, 70 GB of 8-bit weights on a 3.35 TB/s H100: 3,350 / 70 x 10 = about 47.9 tokens per second.
  2. An 8-card node tops out near 47.9 x 8 = 383 tokens per second before KV reads and prefill contention.
  3. Measured 100 tokens per second across 8 cards is 12.5 per card, only 12.5 / 47.9 = about 26 percent of the bandwidth ceiling.

Running at a quarter of the bandwidth ceiling is a scheduler and batching answer, not a physics answer - raise the batch before blaming the card.

Need the vocabulary first? The glossary defines each of these terms with a worked example, and the other banks cover the other half of the interview.