Model architecture
Tensor parallelism
The model's weight matrices are split across several GPUs that compute every layer together. Fits big models and speeds up single-stream decode.
Tensor parallelism cuts one layer of the model into pieces and gives each GPU in a group one piece. The GPUs compute that layer at the same time and combine their results with a collective operation (an all-reduce), so from the outside the group behaves like one very wide GPU. The model only has to fit in the group, not on any single card inside it.
The payoff is speed on a single request. One decode step has to read the whole set of weights, and the group reads those weights in parallel, so the step finishes sooner than one card could manage. That is a different trade from data parallelism, where each card owns the whole model and serves different users: tensor parallelism makes one request faster, it does not add serving capacity on its own.
The cost is communication. Every layer ends in a collective exchange that crosses the interconnect between the cards, so tensor parallelism wants very fast links such as NVLink between GPUs and it hurts badly over slow ones. The interview framing: tensor parallelism buys single-request speed with interconnect bandwidth, and replicas buy capacity with dollars - the two moves solve different problems.
A worked example
A 70B model in 8-bit weights (70 GB), on one H100 with 3.35 TB/s of bandwidth versus two of them joined tensor-parallel, where about 15 percent of each step is spent on the collectives.
- One card: 3.35 TB/s / 70 GB = about 48 tokens per second, the decode ceiling from bandwidth.
- Two cards tensor-parallel: each reads half the weights at the same time, 35 GB apiece, so the step takes 35 GB / 3.35 TB/s = about 96 steps per second.
- Subtract the collectives: 96 x 0.85 = about 81 tokens per second for the single user.
- The gain holds only while the interconnect keeps up; over slow links the all-reduce can erase it.
Two cards in lockstep roughly double single-user decode minus the communication tax. The same request gets faster; the fleet, and its capacity, do not change.
Where this shows up
- LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
- Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.