Cache-Aware Routing & Affinity
The next hidden cost in a serving fleet is where each request lands. Every arrival carries a shared opening - a system line, a tool spec, a project brief - about 1200 tokens of it, and the fleet pays to prefill that opening on whatever card the request lands on unless the opening is already resident there. This is what makes the cached-input discounts on frontier model pricing real: the preamble is resident on the one card that prefilled it, so the discount is not a memory feature, it is a routing guarantee. Cache-blind round-robin destroys the guarantee politely: an arrival meets the card holding its family one time in n, so a 16-card fleet re-prefills the same openings dozens of times a second no matter how large the scratch space is. The fix is one line of arithmetic - hash the prefix family, land on that family card - and its costs show up elsewhere: a hot prompt template now concentrates every arrival of its family on one card and strands the rest, which is why the busiest-card metric (decode headroom, capped near 60%) is a first-class check, and why sharding the hot family across two cards is the follow-up interviewers wait for: it halves the hot card load and costs one duplicate preamble copy. Budget closes the loop, because preambles are big - 1200 tokens of 70B-class KV at 320 KB per token is about 384 MB a copy, so an 8 GB prefix cache holds 20 families - and cards that run short evict silently, turning hashed arrivals back into prefills. The three challenges below size the mechanics: the hit share round-robin throws away, the copies a scratch of a given size holds, and the busiest-card share the hash leaves behind.
Key Concepts
- 1The cached-input discount is a routing guarantee, not a memory feature: a prefix is resident on the one card that prefilled it, so the 90% discount survives only while the router sends each family back to that card. Cache-blind round-robin gives an arrival a one-in-n chance of meeting it
- 2The preamble arithmetic is short: 1200 tokens of 70B-class KV at 320 KB per token is about 384 MB per resident copy, so an 8 GB prefix cache holds 20 families, and a family-hashed fleet of 8 cards needs 8 resident families per card at most
- 3Concentration is the price of affinity: a hot prompt template owns a share of traffic and the hash sends every arrival of that family to one card, so the busiest-card metric, a decode-headroom guard near 60%, is the check that catches a routing plan that is quietly winning the hit rate and losing the balance column
- 4Sharding the hot family across two cards halves its card load and costs one duplicate preamble copy: the honest trade between hit rate and balance, and the follow-up interviewers wait for after the hash answer
- 5Budget closes the loop: cards evict resident prefixes when the scratch runs short, and eviction turns hashed arrivals back into tensor-time prefills, so fitting the cache is part of routing it. Under eviction the hit rate scales with the share of required copies the budget holds
- 6The same arithmetic prices self-hosted fleets too: whether the fleet bills for prefill GPU-seconds or skips them is decided by routing before it is decided by hardware, and a frontier API cached-input discount the fleet cannot reproduce is often a router it did not build
The trade-offs to name out loud
Hashing each prefix family to its card holds hit rates near 100% and concentrates load exactly where traffic already concentrates; round-robin spreads load perfectly and throws the cache away. Sharding the hot family across two cards is the compromise that buys back balance with one duplicate copy, and the right answer names all three numbers: hit rate, busiest card, and copies paid.
Prefix cache competes with weights and KV for the same 80 GB of card memory, and an undersized scratch undoes the router silently: 2 GB holds 5 preamble copies where an 8-card hash needs 8, so hits fall to 62.5% with the routing code untouched. The design answer is a minimum-scratch floor in the capacity plan, not a resize after the hit-rate graph breaks.
Round-robin has no stickiness to manage: nodes restart, drain, and join without state. A cache-aware hash owns a second lifecycle, because growing the fleet reassigns families and dumps warm prefixes across the whole fleet at once, so production deployments keep a consistent hash ring and rebalance gradually. Interviewers score the answer that admits the ops cost.
Route the arrivals yourself
The routing walkthrough
The tier walkthrough beside this one decides who is allowed to enter; this one decides which door each arrival walks through. Every request here carries the same 1200-token opening, and the opening is resident on one card only: the card that prefilled that prompt family. Point the router with the four knobs below and watch two numbers instead of one - the share of arrivals that draw a warm preamble, and the share that lands on the busiest card. The cached-input discount is the first number, the decode headroom is the second, and dialing either one is how a fleet pays for the discount.
The knobs
Routing strategy
Routing strategy: aware
A cache-blind router treats a prefix as any other token; a cache-aware hash keeps each family on the one card that pre-filled it.
Traffic shape
Hot family share (%): 20
The one dominant prefix family: one prompt template that owns this share of arrivals. The other 63 families split the rest evenly.
Fleet and cache
Fleet cards: 8
More cards spread load but cut the odds a cache-blind rotation lands back on the card that holds the prefix.
Prefix cache per card (GB): 8
Room a card keeps for resident preambles at 384 MB each: 1200 tokens of 70B-class KV. Under the requirement, eviction turns hashes into misses.
of arrivals draw a warm preamble from the card cache
100.0%
re-prefills a second of the 40 req/s arriving
0.0
of arrivals land on the busiest card, cap 60%
21.3%
card-seconds per second spent re-prefilling openings
0.00
resident preamble copies a card needs vs what fits
20 of 8
the fleet card bill, every month
$23,302
The two shares at this config
The routing ledger
The plan holds: 100.0% of arrivals skip the prefill and the busiest card carries 21.3%, inside the 60% decode-headroom cap. Affinity buys the hit rate and concentration bills it - every fix in this walkthrough trades one against the other.
Hashing each prefix family to its card holds the family's preamble resident: 100.0% of arrivals skip prefill. The cost arrives in the balance column, not the cache column.
The hot family owns 20% of arrivals and the hash concentrates exactly that: its card carries 21.3%. Cache-aware routing never spends concentration without buying a hit rate, or the other way around.
8 cards: the hash spreads the families to 8 resident families per card. The bill is $23,302 a month, and 8 is exactly what keeps the balance column honest.
8 GB of prefix cache holds 20 preamble copies of 384 MB each - 20 of 8 of the copies this plan demands.
Mattered vs noise (which knobs actually moved the routing plan)
Load a famous outage, then fix the routing plan yourself
See the whole lifecycle as traffic: The Request Highway lab turns every AI request into a vehicle that stops at the edge cache, the guardrails, and the rate limiter before the model pool - and drives it home live, with a surge dial and an outage switch.