Efficiency
GPU utilization
The share of a card's billed time that actually does serving work, and the share of its bandwidth serving reaches. The bill only knows the first.
GPU utilization is quoted by two very different gauges. The dashboard gauge (SM utilization) says how often any compute kernel ran in the last sample window; decode keeps it near 100 percent while running at a fraction of its actual bandwidth, so it says nothing about how busy the card truly is. The utilization that matters for cost is the roofline one: measured tokens per second against the bandwidth ceiling the model allows.
Neither gauge is what the invoice reads. A card is billed by the hour whether it computed or waited, so idle hours are pure cost: a card at 20 percent of its achievable throughput delivers tokens at roughly five times the unit price of the same card kept busy. Raising utilization is the cheapest performance purchase available, because it changes the denominator of cost per million tokens without touching the rate card.
The interview framing: report utilization as a fraction of the honest ceiling, not as the dashboard gauge, and then connect it to money - the bill is fixed and the output is what varies.
A worked example
One card at $2.50 per hour and 70 GB of 8-bit weights, sustaining 400 aggregate tokens per second today, with a batching fix expected to reach 800.
- At 400 tokens per second: tokens per hour = 400 x 3,600 = 1.44 million, so cost per million is $2.50 / 1.44 = about $1.74.
- At 800 tokens per second: 800 x 3,600 = 2.88 million tokens an hour, so $2.50 / 2.88 = about $0.87 per million.
- The card is billed $2.50 both ways; doubling the achieved throughput halves the unit cost.
Utilization, not the hourly rate, is the lever on self-hosted cost per token: the bill is fixed and the output is what varies.
Where this shows up
- Fleet Efficiency & EvictionWhen a serving pool grows past a handful of cards, someone has to judge the cards, and that job is a real interview subject: a senior inference-gateway role lists it as fleet-efficiency work, autoscaling and scoring a rented fleet, evicting underperformers, all aimed at peak tokens per dollar. The model in this topic is one month of one rented pool. Every card in the pool is the same class, but not the same card: each draws a quality lottery ticket at boot, a true serving rate for the rated number, and rides silent dips, rate blips that cost part of the window while they last. The autoscaler never sees the truth. It sees a measured score read through a measurement window, and the window is a lever: over one hour a card can look 25% better or worse than it is, over a day 5%, over a week 2%. The month's bill has three parts: the cards the run set keeps warm bill their rent in full, the cards the score evicts bill 15% of a node-month on the way out (a churned card holds reserved capacity while it drains), and the cards outside the run set never bill at all. Three checks judge the month, and the interview version of this topic is knowing which failure each check can see. The capacity check reads the delivered tokens against demand, because no eviction policy repairs a rate shortage; the convergence check reads the real bill against the ideal operator's bill, the perfect selection with no churn fees sized to the demand shape, because that is what an autoscaler is actually judged on; and the eviction check counts wrong evictions, the cards whose true rate sat at or above the fleet median when the score threw them out, because a wrong eviction is the one failure that refunds nothing. The three presets are the month's three verdicts, and all three pass at least one check, which is why each is dangerous as an incident report: the golden idler bills $69,940 keeping 26 flagship cards warm for a 60,000 tok/s average and burns $44,348 of it on capacity that moved no tokens, yet shows zero unmet demand and zero wrong evictions; the noise evictor scores through the hour window and evicts 12 of 32 cards on a 25% error, 4 of them healthy, 33% wrong; the bargain shortfall bills a tempting $28,840 and covers only 61,015 of 90,000 tok/s, a 32% shortage the bill never could buy back. The fixes carry verified numbers too: the idler on steady mode bills $25,740 for the same demand and converges at 90.9%; the evictor moving to the day window at a 10% eviction share converges at 97.9% with no wrong evictions, and the shortfall swap to flagships covers 90,000 tok/s for $35,360 at 95.6% convergence. The three challenges below work the arithmetic the autoscaler acts on: the tokens-per-dollar score of one card, the whole-card greedy that sizes a run set, and the eviction policy that decides who leaves.
- Metering & Cost AttributionThe fourth serving job is the meter: the model serves traffic, the fleet is paid for, and the dollars land on an invoice that says per 1M, in and out, so the interview question becomes who spent what and on whose behalf. The model is one month of one product. Tokens per request are stated per side - a chat assistant sends 800 prompt tokens and writes 300 out, a coding copilot 2,400 and 600, and a research agent 12,000 and 4,000, because an agentic loop re-sends its context after every tool call and produces tool chatter of its own, so the same "request" moves 15x the tokens. Rates are per 1M tokens, priced separately for prompt and output: the fast tier is $0.80 in and $4.00 out, the flagship $3 and $15, the frontier $15 and $75 - and the out-rate is always the 5x side, because output is generated token by token against a weight read that a prompt prefix caches but an answer never does. Three policy lines decide whether the ledger can be trusted. The prefix cache sells the repeated prompt at a tenth of the fresh rate, so the cached share of prompt tokens is the cheapest dollar on the bill and the saved line meters the discount forgone. Retries re-bill the whole request at full price - both sides - so a 20% retry rate is a 17% tax that product analytics never sees, because from the code's point of view nothing changed. Attribution, tagging the bill by named spender, adds no dollars at all, but a bill with no owner is invisible to the chargeback, the margin check, and the anomaly alert, and the sim lets at most 10% of the bill go untagged before it stops being a ledger. The output numbers compound: the blended rate - bill over every token moved - is the one-line answer to "what does this product cost us per use," and the raw bill against the budget line is the one finance reads. Interviewers test this because the meter is where engineering meets accounting: the xAE-style system-design round asks how you would bill two teams sharing one serving pool, and cost-per-request arithmetic shows up under the scaling follow-ups of nearly every AI design question. The three challenges below work the meter: the dollars one request bills, the bill the cached prefix discounts, and the ghost dollars a half-tagged bill leaks.