Serving
Model warm pool
Standby servers with the model already loaded, ready to admit traffic the moment a burst starts. Warm readiness trades idle rent for the cold-start wait.
A warm pool is a set of replicas that have finished booting - weights loaded, cache allocated, kernels warm - and sit ready for traffic. When a burst arrives, the requests meet ready machines instead of a boot sequence, and the pool refills from new launches while the first responders absorb the peak.
The design question is the pool size, and it is priced in idle hours. Every standby replica rents its card whether or not anyone calls it, and that rent is the cost of deleting the cold start. The honest middle path keeps a small pool (often one) to absorb the first seconds of a burst while autoscaling launches the rest - the burst experiences little queueing, and the rent stays modest.
How many standbys to keep follows the burst shape: how big arrivals get, how fast they rise, and what a boot costs on your storage. It is the same arithmetic at every provider, which is the interview takeaway: warm pools are not a product feature you switch on, they are a lease you buy, and the size is a risk decision, not a constant.
A worked example
Cold start of 32 s per replica, an H100 renting at $2.50 per hour, and bursts that need 4 replicas up at once.
- Always-warm 4 replicas all day: 4 x $2.50 x 24 = $240 per day, about $87,600 a year.
- Cold-spun on bursts instead: the burst pays 32 s per replica and users queue - $0 of idle rent.
- Middle ground: 1 warm replica (about $21,900 a year) absorbs the first arrivals while the other 3 launch.
Readiness has a rent of about $21,900 per replica per year at this rate card - buy as much of it as your burst shape needs, and only that much.
Where this shows up
- Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.
- Design ChatGPT: Tiers & Rate LimitsDesign ChatGPT is the most-cited systems design question at AI companies, and it is really one question: when everyone arrives at once, who gets served, who waits, and who is told no, all without the paying tier ever noticing. The answer interviewers grade against has three parts. First, three pools: a small provisioned pool whose cards exist for one tier only, a larger shared pool where paid requests carry priority over free ones, and the queue itself, which lives in card memory, not in a message broker. Second, a token-bucket rate limiter at the gateway, held in Redis for atomicity under concurrent writers, that admits free arrivals at a chosen share of demand and reconciles the token count after each grant, so a burst of arrivals is answered instantly without consulting a GPU. Third, shedding: a request refused at the gateway with a 429 costs nothing; the same request admitted into a saturated pool holds an expensive seat and lengthens every wait behind it, which is why the tail wait punishes utilization non-linearly, and why an incident that removes 30% of capacity can turn a 97 millisecond paid wait into an 850 millisecond one. Interviewers test this because every knob in the design is a money or trust decision: renting burst capacity for a launch day, tightening the bucket over a jammed pool, or letting the free tier shed so the subscribers do not notice an outage. The three challenges below work the mechanics: the admit count a token bucket grants, the queue wait the utilization formula quotes, and the shed arithmetic that keeps the shared pool from jamming.