Serving
Idempotency
Designing requests so a retry cannot bill twice or answer twice, because a provider retry re-bills the whole request at full prompt and output price.
A timeout is ambiguous: the request may have failed to leave, failed mid-flight, or completed while the answer was lost on the way back. Retrying is the only recovery, and a retry that the provider cannot tell apart from the original runs the work again and bills it again. Idempotency is the design that removes the ambiguity: a unique key sent with the request, and the provider deduplicates on it by returning the first result for a repeated key instead of running the work twice.
In model serving, the retry economics are unusually harsh because retries are whole-request events. A re-issued request re-sends the full prompt, re-bills prompt and output at full price, and reports itself as brand new traffic everywhere but the meter. Agent-style products, where requests are long and timeouts are likeliest, spend their retry budget on the most expensive requests they have.
The design moves are ordinary but they are deliberate. Send an idempotency key on every state-changing or billable call, and cache completed results against it so a retransmission replays instead of re-runs. Treat key handling as part of the client contract: the same retry may reach two instances, and the dedupe has to survive that. Meter the retried share separately, because the gap between requests the product counted and requests the bill counted is exactly the share the retry design was supposed to make visible.
A worked example
A research-agent request moves 12,000 input tokens and 4,000 output tokens on the flagship tier, at $3 and $15 per million tokens. A timeout hits 8 percent of runs, and each timeout re-issues the whole request once.
- One clean request: 12,000 x $3 / 1,000,000 = $0.036 of prompt, 4,000 x $15 / 1,000,000 = $0.06 of output, $0.096 in total.
- A retry re-bills the whole request, not the tail: with 8 percent of runs retried once, the expected bill per issued request is 0.096 x 1.08 = about $0.104.
- A successful deduplication turns that re-billed $0.096 into a $0 replay of a cached result - the gap between retrying and resuming.
One retried agent request in twelve moves the expected bill from $0.096 to about $0.104, all of it prompt and output re-billed - which is the cost idempotency keys exist to prevent.
Where this shows up
- Metering & Cost AttributionThe fourth serving job is the meter: the model serves traffic, the fleet is paid for, and the dollars land on an invoice that says per 1M, in and out, so the interview question becomes who spent what and on whose behalf. The model is one month of one product. Tokens per request are stated per side - a chat assistant sends 800 prompt tokens and writes 300 out, a coding copilot 2,400 and 600, and a research agent 12,000 and 4,000, because an agentic loop re-sends its context after every tool call and produces tool chatter of its own, so the same "request" moves 15x the tokens. Rates are per 1M tokens, priced separately for prompt and output: the fast tier is $0.80 in and $4.00 out, the flagship $3 and $15, the frontier $15 and $75 - and the out-rate is always the 5x side, because output is generated token by token against a weight read that a prompt prefix caches but an answer never does. Three policy lines decide whether the ledger can be trusted. The prefix cache sells the repeated prompt at a tenth of the fresh rate, so the cached share of prompt tokens is the cheapest dollar on the bill and the saved line meters the discount forgone. Retries re-bill the whole request at full price - both sides - so a 20% retry rate is a 17% tax that product analytics never sees, because from the code's point of view nothing changed. Attribution, tagging the bill by named spender, adds no dollars at all, but a bill with no owner is invisible to the chargeback, the margin check, and the anomaly alert, and the sim lets at most 10% of the bill go untagged before it stops being a ledger. The output numbers compound: the blended rate - bill over every token moved - is the one-line answer to "what does this product cost us per use," and the raw bill against the budget line is the one finance reads. Interviewers test this because the meter is where engineering meets accounting: the xAE-style system-design round asks how you would bill two teams sharing one serving pool, and cost-per-request arithmetic shows up under the scaling follow-ups of nearly every AI design question. The three challenges below work the meter: the dollars one request bills, the bill the cached prefix discounts, and the ghost dollars a half-tagged bill leaks.
- Design ChatGPT: Tiers & Rate LimitsDesign ChatGPT is the most-cited systems design question at AI companies, and it is really one question: when everyone arrives at once, who gets served, who waits, and who is told no, all without the paying tier ever noticing. The answer interviewers grade against has three parts. First, three pools: a small provisioned pool whose cards exist for one tier only, a larger shared pool where paid requests carry priority over free ones, and the queue itself, which lives in card memory, not in a message broker. Second, a token-bucket rate limiter at the gateway, held in Redis for atomicity under concurrent writers, that admits free arrivals at a chosen share of demand and reconciles the token count after each grant, so a burst of arrivals is answered instantly without consulting a GPU. Third, shedding: a request refused at the gateway with a 429 costs nothing; the same request admitted into a saturated pool holds an expensive seat and lengthens every wait behind it, which is why the tail wait punishes utilization non-linearly, and why an incident that removes 30% of capacity can turn a 97 millisecond paid wait into an 850 millisecond one. Interviewers test this because every knob in the design is a money or trust decision: renting burst capacity for a launch day, tightening the bucket over a jammed pool, or letting the free tier shed so the subscribers do not notice an outage. The three challenges below work the mechanics: the admit count a token bucket grants, the queue wait the utilization formula quotes, and the shed arithmetic that keeps the shared pool from jamming.