Serving
Consistent hashing
Routing each request family to a card by hashing its key, so most families keep their card and their resident prefix even when the fleet grows or shrinks.
Hash routing makes a cache hit a routing decision: hash the request family key, land it on one card, and that card keeps the family preamble resident so every subsequent request starts from a prefix cache hit instead of a full prefill. The design question is what the hash does when the fleet changes, because rented fleets change constantly.
Naive modulo - family index mod card count - redraws nearly every assignment on a resize, so a fleet that doubles re-prefills half its preamble copies at prices of hundreds of megabytes each. Consistent hashing solves this by placing both families and cards on one shared hash ring: each family belongs to the card clockwise of its position, so adding or removing a card moves only the arc that new card owns. Doubling a fleet with a ring still moves families, but modulo doubling moves far more, and the moved share is what the rebuild bill is multiplied by.
The remaining questions are load ones, and they answer themselves once the numbers are visible: one hot family lands entirely on one card, so the busiest card needs an overflow rule or a sharded hot copy; and evicted families - the ones the scratch could not keep - fall back to full prefill no matter how good the ring is, which is why the scratch size gets compared against the copy count the routing plan demands.
A worked example
A fleet grows from 64 cards to serve a rising arrival rate. Compare modulo routing against a consistent hash ring for 64 cached prompt families, where each displaced family must re-prefill a 384 MB preamble copy.
- Modulo doubling, 64 to 128 cards: a family keeps its card only when its hash mod 128 stays under 64, so exactly 64/128 = 50 percent of families remap.
- Rebuild cost: 64 displaced families x 384 MB = 24,576 MB, about 24.6 GB of re-prefill work across the fleet on the resize itself.
- With a hash ring, adding cards one at a time takes over only the new arcs, so roughly 1 of 65 families moves per added card - the ring turns half the fleet into a few percentages of it.
The resize is where routing meets arithmetic: modulo moves 50 percent of families and about 24.6 GB of preamble, and a hash ring moves a small fixed slice per added card.
Where this shows up
- Cache-Aware Routing & AffinityThe next hidden cost in a serving fleet is where each request lands. Every arrival carries a shared opening - a system line, a tool spec, a project brief - about 1200 tokens of it, and the fleet pays to prefill that opening on whatever card the request lands on unless the opening is already resident there. This is what makes the cached-input discounts on frontier model pricing real: the preamble is resident on the one card that prefilled it, so the discount is not a memory feature, it is a routing guarantee. Cache-blind round-robin destroys the guarantee politely: an arrival meets the card holding its family one time in n, so a 16-card fleet re-prefills the same openings dozens of times a second no matter how large the scratch space is. The fix is one line of arithmetic - hash the prefix family, land on that family card - and its costs show up elsewhere: a hot prompt template now concentrates every arrival of its family on one card and strands the rest, which is why the busiest-card metric (decode headroom, capped near 60%) is a first-class check, and why sharding the hot family across two cards is the follow-up interviewers wait for: it halves the hot card load and costs one duplicate preamble copy. Budget closes the loop, because preambles are big - 1200 tokens of 70B-class KV at 320 KB per token is about 384 MB a copy, so an 8 GB prefix cache holds 20 families - and cards that run short evict silently, turning hashed arrivals back into prefills. The three challenges below size the mechanics: the hit share round-robin throws away, the copies a scratch of a given size holds, and the busiest-card share the hash leaves behind.
- Buying the FleetThe third serving job is the balance sheet: the product found users, the token demand no longer fits on a demo pod, and the interview question becomes procurement. Do you buy the fleet, rent it month to month, or route the overflow through a frontier API's meter? The model is one month. Demand is stated in terabytes of tokens (10T a month is a serious product, 1000T is frontier traffic), a month is 730 hours, and each card contributes an aggregate tokens/sec rate a well-fed, batched steady state sustains: an H100 carries about 10,000 tokens/sec on an 8B chat and 700 on a 70B premium, a B200 is 2.4x either, and a used A100 is 0.45x an H100. Three dollar lines then price the month. Owned cards bill their card-month no matter what flows - about $2,100 to $2,900 for the modern chips once the capex is amortized over 36 months and power, colo, network, and staff are added - rented cards bill the cloud's roughly 1.35x premium on the same floor, and the API bills per token at retail: $0.20 per 1M for the 8B chat, $2.50 for the 70B premium. Notice what that ordering says: ownership beats renting on the per-token rate everywhere, because the cloud premium is the margin it takes for owning. So the question is never "own or rent on rate" - it is utilization. An owned GPU idling is a mortgage payment, and the idle check lets at most 15% of owned capacity sit unused; sizing the fleet to a story about growth instead of the demand curve is how a $441M bet runs 71% idle. The margin check says self-hosting must beat the API by a real quarter - blended cost at most 75% of the API's price - because owning takes financing risk a rental does not: a failed chip generation or a slower model year erases a thin saving. The overflow check caps the API's share at 10% of needed tokens, because the API is metered per token at retail while rented cards bill wholesale: at sustained volumes, 40% of a big month's shortfall bills more than the whole rented cloud would. The sim's three stories are the used-chip bargain that was cheap per card and expensive per token, the shiny chip bet that bought for a demand curve that had not arrived, and the API tax where the overflow meant to catch the spike swallowed the month. The three challenges below work the model: the tokens a fleet carries, the whole rented cards that fill the shortfall, and the dollars the API meter bills on its share.