Buying the Fleet

The third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.

Owning wins the monthly per-token race (H100 8B chat at $0.08/1M owned vs $0.10 rented) but takes the idle risk and a 36-month commitment to hardware that depreciates toward a chip transition; renting runs ~1.35x the same bill with zero commitment and zero idle risk. The professional answer sizes the buffer of owned capacity from measured demand, not from a fundraise deck.

The burst buffer either sits on your balance sheet (bigger owned fleet, higher idle floor exposure) or on rented cards (available in hours, expensive per month) or on the API (instant, retail per token). Each rung up costs more per token and is easier to unwind, and startups usually die holding the wrong end of this: too much owned buffer at small demand, too much API at big demand.

A used A100 is $12k against the H100's $30k, but it carries 0.45x the tokens, so the per-token rate is what it prices - and on premium workloads its rented rate sits $0.81 under the API, leaving no margin for the buffer. The B200's 2.4x rate is what buys the margin room; the used A100's discount is what buys the capex story. An interviewer scores the answer that prices the bargain in tokens, not cards.

Put Buying the Fleet into practice

The best way to lock in what you've learned is to write code. Solve real interview-style problems right now.