Efficiency
Speculative decoding
A cheap draft model proposes several tokens and the main model verifies them in one pass, so a step can emit more than one token while the wording stays exact.
Normal decode produces one token per forward pass, which is slow because every pass pays the full cost of reading the model. Speculative decoding notices that the pass is mostly a memory read, not math: a batch of 8 positions through the same weights costs nearly the same time as 1 position. So let a small draft model guess the next few tokens, then verify all of them through the big model in one pass.
Verification is the part that keeps the output honest. The big model scores every drafted token in that one pass; a token that matches what the big model would have preferred is accepted, the first mismatch ends the block, and the corrected token plus one extra replaces the rejected tail. The published guarantee is that the output distribution matches plain decoding of the big model: it is the same text you would have gotten, just reached faster.
The gain has a ceiling: acceptance rate times draft length, minus the cost of running the draft and the extra memory traffic of verification. It pays most when there is bandwidth headroom, which is why it is a favorite interview question for serving roles: the right answer is about where the idle bandwidth is, not about the draft model being clever.
Practical details worth being able to name: the draft must be genuinely cheap relative to the target; long draft blocks waste more when acceptance is low; and tree-shaped proposals (several candidate continuations verified at once) raise the expected accepted tokens per pass at the cost of more complex verification.
A worked example
A large model decodes alone at 24 tokens per second. A small draft proposes 3 tokens per step, and 70 percent of draft tokens get accepted. The draft is cheap enough that its cost is folded into the final numbers below.
- Expected tokens accepted per verify step, bonus token included: 1 + 0.7 + 0.7^2 + 0.7^3 = 2.53.
- Theoretical ceiling: 24 x 2.53 = about 61 tokens per second.
- Real runs give back a share for the draft pass and the wider verify batch: a realistic gain with a small draft is 1.5x to 2x, so 24 becomes roughly 40.
Speculation converts idle memory bandwidth into throughput, and the text is unchanged - every proposal passes through the real model before it ships.
Where this shows up
- LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
- Metering & Cost AttributionThe fourth serving job is the meter: the model serves traffic, the fleet is paid for, and the dollars land on an invoice that says per 1M, in and out, so the interview question becomes who spent what and on whose behalf. The model is one month of one product. Tokens per request are stated per side - a chat assistant sends 800 prompt tokens and writes 300 out, a coding copilot 2,400 and 600, and a research agent 12,000 and 4,000, because an agentic loop re-sends its context after every tool call and produces tool chatter of its own, so the same "request" moves 15x the tokens. Rates are per 1M tokens, priced separately for prompt and output: the fast tier is $0.80 in and $4.00 out, the flagship $3 and $15, the frontier $15 and $75 - and the out-rate is always the 5x side, because output is generated token by token against a weight read that a prompt prefix caches but an answer never does. Three policy lines decide whether the ledger can be trusted. The prefix cache sells the repeated prompt at a tenth of the fresh rate, so the cached share of prompt tokens is the cheapest dollar on the bill and the saved line meters the discount forgone. Retries re-bill the whole request at full price - both sides - so a 20% retry rate is a 17% tax that product analytics never sees, because from the code's point of view nothing changed. Attribution, tagging the bill by named spender, adds no dollars at all, but a bill with no owner is invisible to the chargeback, the margin check, and the anomaly alert, and the sim lets at most 10% of the bill go untagged before it stops being a ledger. The output numbers compound: the blended rate - bill over every token moved - is the one-line answer to "what does this product cost us per use," and the raw bill against the budget line is the one finance reads. Interviewers test this because the meter is where engineering meets accounting: the xAE-style system-design round asks how you would bill two teams sharing one serving pool, and cost-per-request arithmetic shows up under the scaling follow-ups of nearly every AI design question. The three challenges below work the meter: the dollars one request bills, the bill the cached prefix discounts, and the ghost dollars a half-tagged bill leaks.
- Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.