Pricing
Spot instances
Spare GPU capacity rented at a deep discount, with the provider allowed to reclaim it on short notice. For workloads where a restart is cheap.
Spot (or preemptible) instances are idle machines the provider rents out cheaply on the condition that they can be taken back at any time. The discount is steep - commonly around half to two thirds off on-demand - because the buyer is accepting both eviction risk and the work of handling it.
The handling is the actual work, and it shapes which workloads belong on spot. Evicted replicas must be replaced by working autoscaling; in-flight requests must drain or requeue; anything stateful must rebuild its cache (and pay the cold start) after each eviction. Batch workloads, prefill stages, and fault-tolerant serving absorb all of that cheaply. A single-replica, latency-critical service on spot is a bet, not a design.
The arithmetic to run in an interview is per effective card-hour: price the discount, subtract the expected share of evicted-at-any-moment cards, and see whether spot still wins. The discount usually survives its own downtime, which is exactly why the fault-tolerant workloads live there.
A worked example
On-demand H100s at an assumed $2.50 per hour versus spot at $1.00, evicting about 10 percent of cards at any moment. The fleet runs 8 cards.
- On-demand: 8 x $2.50 = $20 per hour for 8 effective cards, $2.50 per effective card-hour.
- Spot: 8 x $1.00 = $8 per hour, but about 10 percent of cards are evicted at any moment: 8 x 0.9 = 7.2 effective cards.
- Per effective card-hour: $8 / 7.2 = about $1.11, still less than half the on-demand rate.
- The discount is the payment for absorbing restarts and cold starts.
Spot at about $1.11 per effective card-hour against $2.50 on-demand fits fault-tolerant work - prefill stages, batch jobs, replicas that can restart.
Where this shows up
- Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.
- Inference Economics & LatencyEvery time someone sends a message to an AI feature, the model has to run on that request, and the run has a price in dollars and a wait in seconds. The industry word for the run itself is inference: the model processing your prompt and producing an answer, and it is the line item that grows with usage. Where does the money go? Mostly into tokens, the word-sized pieces of text that both prompts and answers are made of. Input tokens are re-sent with every call; output tokens are generated one at a time at a premium rate; and a large discount called prompt caching applies only while the opening of your prompt matches what the provider already has stored, which one careless changing detail destroys. The wait decomposes into stages that scale differently: reading the prompt grows with prompt size, writing the answer grows with answer length, and a fixed network round trip is attached either way - and the stages disagree about which lever is worth pulling. Interviewers pursue this topic because cost and latency are the two numbers that decide whether an AI feature survives its first real month, and both are governed by arithmetic you can do on a whiteboard. The three challenges below work that arithmetic: the cache hit rate and the savings a rotating preamble quietly destroys, the share of a latency budget that writing the answer consumes, and the work a hard output cap clips off real requests.