Serving
Caching layers
The stack of caches above a model - reply, prompt prefix, and gateway - where a hit at a higher layer costs less of the bill than any hit at the GPU layer.
A model-serving stack caches at several heights at once, and each height prices a hit differently. A reply cache stores whole answers keyed by the question family and serves a hit at near-zero marginal cost, billing the provider nothing. A prefix cache keeps request preambles resident near the model and serves a hit by removing most of the prefill - big savings in both wait and the prompt side of the bill. The GPU-level KV cache keeps per-request state within one generating request. None of them overlap; each is the cheapest answer at a different granularity.
The layering is an ordering of trust. A replayed answer is cheapest and riskiest, because a question family can drift while its cached answers stay frozen - which is why reply caches carry freshness budgets and why semantic matching at this layer is a measured trade, not a default. Prefix hits are the workhorse: the preamble is usually stable by construction, so the risk is negligible and the savings show up on every request of the family.
Metering the layers separately is what turns this from folklore into a dashboard. Hit share per layer, and the bill each layer saved, are the two lines that answer the question when a bill surprises you: a cache that went cold on the prompt side, or a reply cache serving stale answers, looks identical to a traffic change unless each layer is watching its own ledger.
A worked example
A request with a 1,500-token prompt on a rate card of $3 per million fresh tokens and $0.30 per million for a cached prefix. The reply cache hits whole answers, the prefix cache hits preambles, and a miss falls through to the full card.
- Reply cache hit: $0 billed and answered in about 50 ms - the cheapest tier, bounded by freshness risk.
- Prefix cache hit: 1,500 x $0.30 / 1,000,000 = about $0.0005 of prompt billing, and most of the prefill wait is gone.
- Cold path: 1,500 x $3.00 / 1,000,000 = $0.0045 of prompt billing plus the full prefill - about 10x the bill of the prefix hit.
A hit one layer higher is a different price: $0, then about $0.0005, then $0.0045 for the same request - the stack saves the bill from the top down.
Where this shows up
- Cache-Aware Routing & AffinityThe next hidden cost in a serving fleet is where each request lands. Every arrival carries a shared opening - a system line, a tool spec, a project brief - about 1200 tokens of it, and the fleet pays to prefill that opening on whatever card the request lands on unless the opening is already resident there. This is what makes the cached-input discounts on frontier model pricing real: the preamble is resident on the one card that prefilled it, so the discount is not a memory feature, it is a routing guarantee. Cache-blind round-robin destroys the guarantee politely: an arrival meets the card holding its family one time in n, so a 16-card fleet re-prefills the same openings dozens of times a second no matter how large the scratch space is. The fix is one line of arithmetic - hash the prefix family, land on that family card - and its costs show up elsewhere: a hot prompt template now concentrates every arrival of its family on one card and strands the rest, which is why the busiest-card metric (decode headroom, capped near 60%) is a first-class check, and why sharding the hot family across two cards is the follow-up interviewers wait for: it halves the hot card load and costs one duplicate preamble copy. Budget closes the loop, because preambles are big - 1200 tokens of 70B-class KV at 320 KB per token is about 384 MB a copy, so an 8 GB prefix cache holds 20 families - and cards that run short evict silently, turning hashed arrivals back into prefills. The three challenges below size the mechanics: the hit share round-robin throws away, the copies a scratch of a given size holds, and the busiest-card share the hash leaves behind.
- Design ChatGPT: Tiers & Rate LimitsDesign ChatGPT is the most-cited systems design question at AI companies, and it is really one question: when everyone arrives at once, who gets served, who waits, and who is told no, all without the paying tier ever noticing. The answer interviewers grade against has three parts. First, three pools: a small provisioned pool whose cards exist for one tier only, a larger shared pool where paid requests carry priority over free ones, and the queue itself, which lives in card memory, not in a message broker. Second, a token-bucket rate limiter at the gateway, held in Redis for atomicity under concurrent writers, that admits free arrivals at a chosen share of demand and reconciles the token count after each grant, so a burst of arrivals is answered instantly without consulting a GPU. Third, shedding: a request refused at the gateway with a 429 costs nothing; the same request admitted into a saturated pool holds an expensive seat and lengthens every wait behind it, which is why the tail wait punishes utilization non-linearly, and why an incident that removes 30% of capacity can turn a 97 millisecond paid wait into an 850 millisecond one. Interviewers test this because every knob in the design is a money or trust decision: renting burst capacity for a launch day, tightening the bucket over a jammed pool, or letting the free tier shed so the subscribers do not notice an outage. The three challenges below work the mechanics: the admit count a token bucket grants, the queue wait the utilization formula quotes, and the shed arithmetic that keeps the shared pool from jamming.
- Prompting & Context WindowsYou have typed the same question into a chat window twice, gotten two different answers, and watched a third rewording produce a third answer. Nothing about the model changed between those calls. What changed was the wording - and the wording is close to the whole input the model ever sees. That input is the prompt: the entire text sent with each call, your question plus every instruction and worked example around it. It matters because a model reads a fixed amount of text per call, called the context window, counted in tokens, the word-sized pieces models are measured in at roughly 4 characters each; everything you send counts against the window, and so does everything the model writes, and when the count runs out the oldest text drops silently, exactly like a capped log buffer, with no error anywhere. Two skills inside prompting carry the interview weight. One is format control: a program can only route on replies whose shape it can read, and there are two ways to get the shape, describing the rule in prose, which models follow loosely, and few-shot examples, worked question and answer pairs shown instead of described, which anchor the format at the price of tokens carried on every call. The other is the skill interviewers probe hardest: when you change the wording, prove the change made it better before anyone ships it, the same fixed question set graded under the old wording and the new - show me you would not ship a prompt change on vibes. The three challenges below build exactly that: the swap audit that catches what a single pass rate hides, the packing rule that keeps the most recent turns inside the window, and the label contract that shows what format control has to survive.