Pricing
Serverless inference
Calling a provider's pooled GPUs per request instead of running your own. Nothing idles on your bill, in exchange for per-token prices and limits.
Serverless inference means sending requests to a provider whose GPUs are always on and shared among many customers, and paying per token (or per second) for what you use. No machines to rent, patch, or keep warm; when traffic stops, the bill stops with it, which is the whole appeal for spiky or low-volume workloads.
The exchange on offer is utilization for control. The provider pools many customers onto each card and keeps its own fleet busy, which is how their per-token price can be sensible. What you give up: per-token prices carry a margin over raw card cost at high utilization; throughput can be throttled or rate limited, and capacity is shared; long-context and very large models may be unavailable or priced steeply; and caching behavior (prefix reuse, routing) may leave your control.
The comparison to rent-your-own is a function of utilization, not brand preference: idle rent is what self-hosting adds, and per-token margin is what serverless adds. A workload that keeps a card busy most of the day moves toward renting its own; a workload that spikes a few hours a week moves toward serverless. State the utilization and the arithmetic makes the case on its own.
A worked example
A feature needs inference 4 hours a day, 30 days a month, at 500 output tokens per second. Assumed serverless price: $4.80 per million output tokens; a dedicated H100 rents at $2.50 per hour.
- Tokens per month: 4 x 3,600 x 500 x 30 = 216 million.
- Serverless: 216 million x $4.80 per million = $1,036.80 for the month.
- Dedicated rental for the month is billed whether it is used: $2.50 x 24 x 30 = $1,800.
- At all-day usage the comparison flips: 720 h x 3,600 x 500 tokens = 1,296 million tokens, and 1,296 million x $4.80 = $6,220.80.
At 4 hours a day serverless is $1,037 versus $1,800 rented - and it inverts when the workload runs all day, because per-token margins beat idle rent only on idle boxes.
Where this shows up
- Inference Economics & LatencyEvery time someone sends a message to an AI feature, the model has to run on that request, and the run has a price in dollars and a wait in seconds. The industry word for the run itself is inference: the model processing your prompt and producing an answer, and it is the line item that grows with usage. Where does the money go? Mostly into tokens, the word-sized pieces of text that both prompts and answers are made of. Input tokens are re-sent with every call; output tokens are generated one at a time at a premium rate; and a large discount called prompt caching applies only while the opening of your prompt matches what the provider already has stored, which one careless changing detail destroys. The wait decomposes into stages that scale differently: reading the prompt grows with prompt size, writing the answer grows with answer length, and a fixed network round trip is attached either way - and the stages disagree about which lever is worth pulling. Interviewers pursue this topic because cost and latency are the two numbers that decide whether an AI feature survives its first real month, and both are governed by arithmetic you can do on a whiteboard. The three challenges below work that arithmetic: the cache hit rate and the savings a rotating preamble quietly destroys, the share of a latency budget that writing the answer consumes, and the work a hard output cap clips off real requests.
- Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.