Efficiency
Roofline model
A rule for guessing a kernel's speed: the smaller of what its math could reach and what memory bandwidth allows. Where your workload sits says what to cut.
Every operation on a GPU has two limits: the compute ceiling (peak FLOPs) and the memory ceiling (bandwidth). Which one applies depends on arithmetic intensity, the number of FLOPs performed per byte moved. Plot intensity on the horizontal axis, achievable FLOPs on the vertical, and you get a roof: a sloped line where bandwidth runs out, then a flat line at compute peak. The corner of the two is the ridge point.
The practical use is diagnosis. Work under the ridge point on the sloped line is memory-bound: more FLOPs will not help, fewer bytes will. Work near the flat top is compute-bound: the bandwidth is not the problem, the math is. In serving, decode of a big model is the classic memory-bound case, since each step reads every weight to produce one token, while prefill over a long prompt is compute-bound.
That split shapes the optimizations you choose. Memory-bound work rewards 8-bit quantization (half the bytes), batching (amortize one read across many users), and keeping a KV cache small. Compute-bound work rewards shorter sequences, fused kernels, and mixed precision where it helps. Applying the wrong fix is the failure interviewers hunt for: batching helps a memory-bound decode but a compute-bound prefill may just queue.
The model is a first-order estimate, not a simulator: it ignores cache hierarchy, occupancy and overlap. Say that out loud in an interview and then use it anyway - a back-of-envelope roofline on the weights, the bandwidth and the batch size usually lands within a useful distance of the measured truth.
A worked example
Decode on one H100: about 989 TFLOPs/s of dense 16-bit compute and 3.35 TB/s of bandwidth. A 70B model held in 16-bit decodes one token per step.
- Ridge point: 989 TFLOPs/s / 3.35 TB/s = 295 FLOPs per byte.
- Decode intensity: one step reads 140 GB of weights and performs 2 x 70B = 140 GFLOPs, so 140 GFLOPs / 140 GB = 1 FLOP per byte.
- 1 is nowhere near 295, so decode rides the memory line: 3.35 TB/s / 140 GB = 23.9 tokens per second.
- Prefill of a 2,000-token prompt does 2,000 x 140 GFLOPs per 140 GB read: intensity about 2,000 FLOPs per byte, well past the ridge, so prefill runs near the compute peak.
Decode buys bandwidth, prefill buys FLOPs. The roofline tells you which budget each phase is spending, and so which fix can work.
Where this shows up
- Fleet Efficiency & EvictionWhen a serving pool grows past a handful of cards, someone has to judge the cards, and that job is a real interview subject: a senior inference-gateway role lists it as fleet-efficiency work, autoscaling and scoring a rented fleet, evicting underperformers, all aimed at peak tokens per dollar. The model in this topic is one month of one rented pool. Every card in the pool is the same class, but not the same card: each draws a quality lottery ticket at boot, a true serving rate for the rated number, and rides silent dips, rate blips that cost part of the window while they last. The autoscaler never sees the truth. It sees a measured score read through a measurement window, and the window is a lever: over one hour a card can look 25% better or worse than it is, over a day 5%, over a week 2%. The month's bill has three parts: the cards the run set keeps warm bill their rent in full, the cards the score evicts bill 15% of a node-month on the way out (a churned card holds reserved capacity while it drains), and the cards outside the run set never bill at all. Three checks judge the month, and the interview version of this topic is knowing which failure each check can see. The capacity check reads the delivered tokens against demand, because no eviction policy repairs a rate shortage; the convergence check reads the real bill against the ideal operator's bill, the perfect selection with no churn fees sized to the demand shape, because that is what an autoscaler is actually judged on; and the eviction check counts wrong evictions, the cards whose true rate sat at or above the fleet median when the score threw them out, because a wrong eviction is the one failure that refunds nothing. The three presets are the month's three verdicts, and all three pass at least one check, which is why each is dangerous as an incident report: the golden idler bills $69,940 keeping 26 flagship cards warm for a 60,000 tok/s average and burns $44,348 of it on capacity that moved no tokens, yet shows zero unmet demand and zero wrong evictions; the noise evictor scores through the hour window and evicts 12 of 32 cards on a 25% error, 4 of them healthy, 33% wrong; the bargain shortfall bills a tempting $28,840 and covers only 61,015 of 90,000 tok/s, a 32% shortage the bill never could buy back. The fixes carry verified numbers too: the idler on steady mode bills $25,740 for the same demand and converges at 90.9%; the evictor moving to the day window at a 10% eviction share converges at 97.9% with no wrong evictions, and the shortfall swap to flagships covers 90,000 tok/s for $35,360 at 95.6% convergence. The three challenges below work the arithmetic the autoscaler acts on: the tokens-per-dollar score of one card, the whole-card greedy that sizes a run set, and the eviction policy that decides who leaves.
- Quantization & the Startup StackThe second serving job is not the fleet; it is the one card, when the budget is what a startup can put on the balance sheet instead of a Vercel bill. Open weights changed the question: the model can be downloaded, and the job becomes making it fit and making it fast on hardware a 24 GB card describes. Three compression decisions run in a fixed order. Weight precision comes first: a model of P parameters takes 2 x P bytes at bf16, 1 x P at INT8, and 0.5 x P at 4-bit, and because decode is bandwidth bound - the card re-reads every weight byte for each token - cutting the bytes in half doubles tokens/sec per stream exactly. An 8B brain goes from 16 GB to 4 GB, and a card re-reading 4 GB through a 1008 GB/s bus decodes 252 tokens a second per stream. KV precision comes second and is the cheaper knob: the per-seat rent of 128 KB a token at bf16 for an 8B-class model collapses quarter-price at 4-bit, which is what llama.cpp KV quantization and vLLM FP8 KV do, while costing about a quality point. Model size, the third knob, is the expensive one - but distillation is how startups move it: train a smaller student on the outputs of a bigger teacher and keep most of the quality at a fraction of the arithmetic. The follow-up interviewers run at startups is the trap answer: quantization is never one decision. Weights, KV, and context length are three separate dial groups, and the 4-bit-weights-on-4-bit-KV stack is the combination that lives just under the quality floor, which is why the quality regression has to be measured, not assumed - bad quantization scales have produced long-context regressions that dropped retrieval recall from 91 to 13 percent before a recalibration put it back. The three challenges below work the stack: the bytes of weights a precision choice buys, the seats a card grants after the per-seat KV rent, and the tokens/sec the memory bus lets through.
- Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.