Serving
Cold start
The stretch a newly started server spends loading weights and allocating cache before it can answer. Bursts pay it again on every replica that launches.
A model server is not ready the moment its process starts. It has to read the weights from storage, allocate the KV cache, warm the kernels and any compiled graphs, and only then can it admit a request. That boot stretch is the cold start, and it is dead time for everyone whose request arrives during it.
Cold starts mostly hurt at the moments you care about most: a burst that triggers autoscaling, a failed replica being replaced, spot capacity being evicted, or a serverless deployment that pooled the model away. A steady-state fleet on busy hardware never reboots, which is why the cost hides in benchmarks and shows up in launch weeks.
The mitigations are each worth being able to name: keep a warm pool so readiness is rented in advance; snapshot memory so the load step is skipped; quantize the weights so there are fewer bytes to read; and keep the model count small. The interview answer pairs the diagnosis (where the 30 seconds go) with the right mitigation for the workload, and it says out loud that warm readiness has its own rent.
A worked example
A replica must load 70 GB of 8-bit weights from NVMe storage at 6 GB/s and then run a 20 s initialization before its first token.
- Weight load: 70 GB / 6 GB per second = about 12 s to read the model.
- Initialization adds the stated 20 s, so first token is about 12 + 20 = 32 s after launch.
- A burst of 10 arrivals inside that window all wait in queue; none of them receives a token before second 32 of the boot.
- A warm pool holding the weights ready removes nearly the whole wait.
About 32 s of dead wait per freshly launched replica - the cost that autoscaling and serverless deployments pay on every burst.
Where this shows up
- Design ChatGPT: Tiers & Rate LimitsDesign ChatGPT is the most-cited systems design question at AI companies, and it is really one question: when everyone arrives at once, who gets served, who waits, and who is told no, all without the paying tier ever noticing. The answer interviewers grade against has three parts. First, three pools: a small provisioned pool whose cards exist for one tier only, a larger shared pool where paid requests carry priority over free ones, and the queue itself, which lives in card memory, not in a message broker. Second, a token-bucket rate limiter at the gateway, held in Redis for atomicity under concurrent writers, that admits free arrivals at a chosen share of demand and reconciles the token count after each grant, so a burst of arrivals is answered instantly without consulting a GPU. Third, shedding: a request refused at the gateway with a 429 costs nothing; the same request admitted into a saturated pool holds an expensive seat and lengthens every wait behind it, which is why the tail wait punishes utilization non-linearly, and why an incident that removes 30% of capacity can turn a 97 millisecond paid wait into an 850 millisecond one. Interviewers test this because every knob in the design is a money or trust decision: renting burst capacity for a launch day, tightening the bucket over a jammed pool, or letting the free tier shed so the subscribers do not notice an outage. The three challenges below work the mechanics: the admit count a token bucket grants, the queue wait the utilization formula quotes, and the shed arithmetic that keeps the shared pool from jamming.
- Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.