Tokens per second per GPU

Tokens per second per GPU is the serving stack's productivity number, and it has two faces. Per-user tokens per second is what the caller feels as typing speed. Aggregate tokens per second is what the operator sells, and the two are not the same: at batch 16, each user might see 40 tokens per second while the card emits 600 in aggregate, because the weight reads are shared across the whole batch.

For single-token decode the ceiling is simple to reason about. Each step must read every weight byte once, so tokens per second cannot exceed memory bandwidth divided by the size of the model. Batch scaling then multiplies that ceiling, because the same weight read serves many users at once, until KV cache reads and scheduler overhead eat the difference.

Use it for honest comparisons, and say the conditions out loud: tokens per second per GPU across an 8B fp8 model says nothing about a 70B fp16 setup, and prefill-heavy workloads measure differently from decode-heavy ones. The variants you may be asked to compare are time to first token (prefill speed), inter-token latency (decode speed), and aggregate throughput (fleet fitness).

On the fleet level, this number per card times the number of cards, divided by the hourly cost, gives tokens per dollar, the unit that actually decides rental choices. A card with half the tokens per second per GPU is only interesting if it costs more than half as much.

A 70B model in 8-bit on one H100 (about 3.35 TB/s of memory bandwidth), counting weight traffic only; KV cache reads trim these numbers at long context.

1. One user: 3.35 TB/s / 70 GB = 47.9 steps per second, so about 48 tokens per second.
2. At batch 16 the weights are still read once per step: 47.9 x 16 = about 765 tokens per second in aggregate.
3. Per-user speed stays near 48 until KV reads and scheduling overhead consume the slack.

One card, one model: 48 tokens per second for a single user, about 765 across a batch of 16 - batching reuses reads instead of repeating them.

Where this shows up

• Fleet Efficiency & EvictionWhen a serving pool grows past a handful of cards, someone has to judge the cards, and that job is a real interview subject: a senior inference-gateway role lists it as fleet-efficiency work, autoscaling and scoring a rented fleet, evicting underperformers, all aimed at peak tokens per dollar. The model in this topic is one month of one rented pool. Every card in the pool is the same class, but not the same card: each draws a quality lottery ticket at boot, a true serving rate for the rated number, and rides silent dips, rate blips that cost part of the window while they last. The autoscaler never sees the truth. It sees a measured score read through a measurement window, and the window is a lever: over one hour a card can look 25% better or worse than it is, over a day 5%, over a week 2%. The month's bill has three parts: the cards the run set keeps warm bill their rent in full, the cards the score evicts bill 15% of a node-month on the way out (a churned card holds reserved capacity while it drains), and the cards outside the run set never bill at all. Three checks judge the month, and the interview version of this topic is knowing which failure each check can see. The capacity check reads the delivered tokens against demand, because no eviction policy repairs a rate shortage; the convergence check reads the real bill against the ideal operator's bill, the perfect selection with no churn fees sized to the demand shape, because that is what an autoscaler is actually judged on; and the eviction check counts wrong evictions, the cards whose true rate sat at or above the fleet median when the score threw them out, because a wrong eviction is the one failure that refunds nothing. The three presets are the month's three verdicts, and all three pass at least one check, which is why each is dangerous as an incident report: the golden idler bills $69,940 keeping 26 flagship cards warm for a 60,000 tok/s average and burns $44,348 of it on capacity that moved no tokens, yet shows zero unmet demand and zero wrong evictions; the noise evictor scores through the hour window and evicts 12 of 32 cards on a 25% error, 4 of them healthy, 33% wrong; the bargain shortfall bills a tempting $28,840 and covers only 61,015 of 90,000 tok/s, a 32% shortage the bill never could buy back. The fixes carry verified numbers too: the idler on steady mode bills $25,740 for the same demand and converges at 90.9%; the evictor moving to the day window at a 10% eviction share converges at 97.9% with no wrong evictions, and the shortfall swap to flagships covers 90,000 tok/s for $35,360 at 95.6% convergence. The three challenges below work the arithmetic the autoscaler acts on: the tokens-per-dollar score of one card, the whole-card greedy that sizes a run set, and the eviction policy that decides who leaves.
• LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
• Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.

Test yourself

• AI Serving Infrastructure Interview Questions2 questions on this term
• Quantization and Inference Cost Questions3 questions on this term
• Inference GPU Selection Questions4 questions on this term
• Scaling and Load Balancing Interview Questions1 question on this term
• Latency Engineering Interview Questions2 questions on this term

Related terms