Efficiency
Eviction policy
What a full prefix cache forgets when a new family needs the space. Hit rate is capped by the share of families the scratch can keep resident at once.
A prefix cache is a fixed amount of card memory holding copies of prompt preambles, and when it fills, something leaves. The policy that chooses the eviction decides whether the cache is serving the traffic or its own history: families that stop appearing should leave fast, families that are about to return should stay, and both rules only work if someone measured which families actually return.
The first number is capacity, not cleverness. The scratch holds whole copies and nothing between: a partially resident preamble is not a hit, so the copy count is the floor of scratch size over copy size. Routing determines how many distinct families a card must serve; the scratch determines how many it can keep. When demand for residency exceeds capacity, the hit rate collapses toward the ratio of kept copies to demanded copies, no matter how the router behaves.
The interview answer pairs the two levers with their failure modes. Grow the scratch or route more tightly until demanded copies fit within held copies; otherwise eviction dominates the hit rate and the fix is capacity or routing, not the eviction algorithm. Choosing least-recently-used is the sensible default, but it is the small lever - which is why the copy arithmetic comes first and the policy argument second.
A worked example
A card keeps an 8 GB prefix scratch. A resident preamble copy of a 1,200-token prompt family is 384 MB. Traffic touches 20 families per card today.
- Capacity: 8 GB x 1,000 = 8,000 MB, and 8,000 / 384 = 20.8, so the scratch holds 20 whole copies.
- At a 20-family working set, every arrival can hit a resident copy: 20 kept / 20 demanded = 100 percent.
- When traffic widens to 25 families, capacity is short 5 copies: the best a policy can do is 20 / 25 = 80 percent, and each miss re-prefills a 384 MB copy.
The scratch holds 20 whole copies out of 25 demanded, capping the hit rate at 80 percent - a policy choice between what stays, but an arithmetic ceiling on what is possible.
Where this shows up
- Cache-Aware Routing & AffinityThe next hidden cost in a serving fleet is where each request lands. Every arrival carries a shared opening - a system line, a tool spec, a project brief - about 1200 tokens of it, and the fleet pays to prefill that opening on whatever card the request lands on unless the opening is already resident there. This is what makes the cached-input discounts on frontier model pricing real: the preamble is resident on the one card that prefilled it, so the discount is not a memory feature, it is a routing guarantee. Cache-blind round-robin destroys the guarantee politely: an arrival meets the card holding its family one time in n, so a 16-card fleet re-prefills the same openings dozens of times a second no matter how large the scratch space is. The fix is one line of arithmetic - hash the prefix family, land on that family card - and its costs show up elsewhere: a hot prompt template now concentrates every arrival of its family on one card and strands the rest, which is why the busiest-card metric (decode headroom, capped near 60%) is a first-class check, and why sharding the hot family across two cards is the follow-up interviewers wait for: it halves the hot card load and costs one duplicate preamble copy. Budget closes the loop, because preambles are big - 1200 tokens of 70B-class KV at 320 KB per token is about 384 MB a copy, so an 8 GB prefix cache holds 20 families - and cards that run short evict silently, turning hashed arrivals back into prefills. The three challenges below size the mechanics: the hit share round-robin throws away, the copies a scratch of a given size holds, and the busiest-card share the hash leaves behind.
- Fleet Efficiency & EvictionWhen a serving pool grows past a handful of cards, someone has to judge the cards, and that job is a real interview subject: a senior inference-gateway role lists it as fleet-efficiency work, autoscaling and scoring a rented fleet, evicting underperformers, all aimed at peak tokens per dollar. The model in this topic is one month of one rented pool. Every card in the pool is the same class, but not the same card: each draws a quality lottery ticket at boot, a true serving rate for the rated number, and rides silent dips, rate blips that cost part of the window while they last. The autoscaler never sees the truth. It sees a measured score read through a measurement window, and the window is a lever: over one hour a card can look 25% better or worse than it is, over a day 5%, over a week 2%. The month's bill has three parts: the cards the run set keeps warm bill their rent in full, the cards the score evicts bill 15% of a node-month on the way out (a churned card holds reserved capacity while it drains), and the cards outside the run set never bill at all. Three checks judge the month, and the interview version of this topic is knowing which failure each check can see. The capacity check reads the delivered tokens against demand, because no eviction policy repairs a rate shortage; the convergence check reads the real bill against the ideal operator's bill, the perfect selection with no churn fees sized to the demand shape, because that is what an autoscaler is actually judged on; and the eviction check counts wrong evictions, the cards whose true rate sat at or above the fleet median when the score threw them out, because a wrong eviction is the one failure that refunds nothing. The three presets are the month's three verdicts, and all three pass at least one check, which is why each is dangerous as an incident report: the golden idler bills $69,940 keeping 26 flagship cards warm for a 60,000 tok/s average and burns $44,348 of it on capacity that moved no tokens, yet shows zero unmet demand and zero wrong evictions; the noise evictor scores through the hour window and evicts 12 of 32 cards on a 25% error, 4 of them healthy, 33% wrong; the bargain shortfall bills a tempting $28,840 and covers only 61,015 of 90,000 tok/s, a 32% shortage the bill never could buy back. The fixes carry verified numbers too: the idler on steady mode bills $25,740 for the same demand and converges at 90.9%; the evictor moving to the day window at a 10% eviction share converges at 97.9% with no wrong evictions, and the shortfall swap to flagships covers 90,000 tok/s for $35,360 at 95.6% convergence. The three challenges below work the arithmetic the autoscaler acts on: the tokens-per-dollar score of one card, the whole-card greedy that sizes a run set, and the eviction policy that decides who leaves.