Model architecture
Pipeline parallelism
The model is cut into stages that run on different GPUs, one after another. Adds capacity in whole-card chunks but does not speed one request.
Pipeline parallelism cuts the model by depth instead of width: the first layers live on one GPU, the next layers on another, and so on. A request walks the stages in order, passing activations down the line, so the model can be spread across cards that need no special interconnect - only a network link, because no two GPUs need to talk about the same layer.
The trouble with a plain pipeline is idleness: while one card works on stage two, the card holding stage four waits. Serving stacks fill the wait with microbatches - many requests in flight, each at a different stage - and the leftover idle time between fills is the bubble. More microbatches shrink the bubble; fewer make the cards idle more of the time.
The interview framing sets pipeline against tensor: pipeline adds capacity in whole-card chunks and tolerates slow links, tensor buys single-request speed and needs fast ones. Pipelines also set a floor on latency, because even an idle pipeline pays for the whole chain of stages on the first request in.
A worked example
A 70B model in 8-bit cut into 4 stages of about 17.5 GB each, run over cards linked by a slower network link, with the choice of 8 or 16 microbatches in flight.
- Stage memory: 70 GB / 4 = 17.5 GB per card, so four modest cards can each hold one stage.
- Bubble share with 4 stages and 8 microbatches: (4 - 1) / (4 + 8 - 1) = 3/11 = 27 percent of the cycle idles.
- With 16 microbatches: 3 / (4 + 16 - 1) = 3/19 = 16 percent idle.
- Aggregate throughput still scales toward 4 cards of capacity once the pipe fills; the very first request in still walks all 4 stages before any token appears.
Pipelines add capacity and tolerate slow links; the tax is the bubble, which microbatching buys down, and the latency floor comes from the stage count.
Where this shows up
- LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
- Fleet Efficiency & EvictionWhen a serving pool grows past a handful of cards, someone has to judge the cards, and that job is a real interview subject: a senior inference-gateway role lists it as fleet-efficiency work, autoscaling and scoring a rented fleet, evicting underperformers, all aimed at peak tokens per dollar. The model in this topic is one month of one rented pool. Every card in the pool is the same class, but not the same card: each draws a quality lottery ticket at boot, a true serving rate for the rated number, and rides silent dips, rate blips that cost part of the window while they last. The autoscaler never sees the truth. It sees a measured score read through a measurement window, and the window is a lever: over one hour a card can look 25% better or worse than it is, over a day 5%, over a week 2%. The month's bill has three parts: the cards the run set keeps warm bill their rent in full, the cards the score evicts bill 15% of a node-month on the way out (a churned card holds reserved capacity while it drains), and the cards outside the run set never bill at all. Three checks judge the month, and the interview version of this topic is knowing which failure each check can see. The capacity check reads the delivered tokens against demand, because no eviction policy repairs a rate shortage; the convergence check reads the real bill against the ideal operator's bill, the perfect selection with no churn fees sized to the demand shape, because that is what an autoscaler is actually judged on; and the eviction check counts wrong evictions, the cards whose true rate sat at or above the fleet median when the score threw them out, because a wrong eviction is the one failure that refunds nothing. The three presets are the month's three verdicts, and all three pass at least one check, which is why each is dangerous as an incident report: the golden idler bills $69,940 keeping 26 flagship cards warm for a 60,000 tok/s average and burns $44,348 of it on capacity that moved no tokens, yet shows zero unmet demand and zero wrong evictions; the noise evictor scores through the hour window and evicts 12 of 32 cards on a 25% error, 4 of them healthy, 33% wrong; the bargain shortfall bills a tempting $28,840 and covers only 61,015 of 90,000 tok/s, a 32% shortage the bill never could buy back. The fixes carry verified numbers too: the idler on steady mode bills $25,740 for the same demand and converges at 90.9%; the evictor moving to the day window at a 10% eviction share converges at 97.9% with no wrong evictions, and the shortfall swap to flagships covers 90,000 tok/s for $35,360 at 95.6% convergence. The three challenges below work the arithmetic the autoscaler acts on: the tokens-per-dollar score of one card, the whole-card greedy that sizes a run set, and the eviction policy that decides who leaves.