Serving
Fan-out
One user request that spawns many model calls at once, multiplying token volume and provider pressure. It is the first load to trip a rate limit or a budget.
Orchestration changes traffic shape as much as volume. A plain chat feature turns one user turn into one model call; an agent or a synthesis step turns one user turn into several parallel calls - one per tool turn, sub-question, or source - and the provider sees the multiplication, not the user. Concurrency limits, rate limits, and the billing meter all key on calls, so a feature that looks modest by user count can arrive at the provider with the load of many times its own traffic.
The bill sees it too, and the arithmetic stacks. Each spawned call carries its own prompt - often re-reading shared context - and its own output side, priced at the rate card, with the cache discount reaching only whatever prefix the calls genuinely share. This is why token-per-request profiling is an interview answer: the multiplicative feature is rarely the one that surprises the budget, the profile is.
The design controls are also multiplicative. Bound the fan-out width explicitly - a fixed cap, not whatever the sub-tasks happen to be - because the cap is the number the rate limiter and cost alert can be sized against. Share context through a common cached prefix so spawned calls pay the discounted prompt rate where possible. And meter at the orchestration layer, so one user turn has a name in the ledger even when it became nine provider calls.
A worked example
One user turn fans out to 4 parallel agent calls, each moving 12,000 input and 4,000 output tokens at $3 and $15 per million. The same turn as a plain chat request moves 800 input and 300 output tokens on the fast tier.
- Fan-out volume: 4 parallel calls x (12,000 + 4,000) = 64,000 tokens of provider traffic from one user turn.
- Chat volume: 800 + 300 = 1,100 tokens of provider traffic from the same turn.
- The ratio: 64,000 / 1,100 = about 58x - and each of the 4 calls is also a separate rate-limit event at the gateway.
Fan-out turns 1,100 tokens and a single call into 64,000 tokens and 4 calls - about 58x the traffic, and 4x the pressure on every per-call limit, from one user turn.
Where this shows up
- AI Agents: Loops, Tools, and StoppingAn ordinary AI chat answers; an AI agent acts. Give a model access to tools - a search function, a database lookup, a send-email function - and it becomes an agent: after each reply it can call a tool, read what the tool returned, and keep going until it believes the job is done. The loop behind that is four lines: ask the model which tool to call next, run the tool, append the result, decide whether to keep going. Three of those lines are plumbing. The fourth, the decide line, is where agents fail in ways that never throw an error, and it is where this topic lives. A loop that re-calls the same tool with the same arguments until the budget expires crashes nothing; it just bills you, minute after minute, for re-reading a page it already has. A run that ends because the model said it was done, when the work is not done, ships a confidently wrong result with no error attached. Both failures are invisible to the usual alerting, which is why interviewers press on them: the questions separate people who have run agents in production from people who have heard of them. The three challenges below drill the repeat detector that catches a stuck run, the error triage that decides which tool failures deserve a retry and which should stop the run, and the running cost curve that names the turn where the spend should have been cut.
- Metering & Cost AttributionThe fourth serving job is the meter: the model serves traffic, the fleet is paid for, and the dollars land on an invoice that says per 1M, in and out, so the interview question becomes who spent what and on whose behalf. The model is one month of one product. Tokens per request are stated per side - a chat assistant sends 800 prompt tokens and writes 300 out, a coding copilot 2,400 and 600, and a research agent 12,000 and 4,000, because an agentic loop re-sends its context after every tool call and produces tool chatter of its own, so the same "request" moves 15x the tokens. Rates are per 1M tokens, priced separately for prompt and output: the fast tier is $0.80 in and $4.00 out, the flagship $3 and $15, the frontier $15 and $75 - and the out-rate is always the 5x side, because output is generated token by token against a weight read that a prompt prefix caches but an answer never does. Three policy lines decide whether the ledger can be trusted. The prefix cache sells the repeated prompt at a tenth of the fresh rate, so the cached share of prompt tokens is the cheapest dollar on the bill and the saved line meters the discount forgone. Retries re-bill the whole request at full price - both sides - so a 20% retry rate is a 17% tax that product analytics never sees, because from the code's point of view nothing changed. Attribution, tagging the bill by named spender, adds no dollars at all, but a bill with no owner is invisible to the chargeback, the margin check, and the anomaly alert, and the sim lets at most 10% of the bill go untagged before it stops being a ledger. The output numbers compound: the blended rate - bill over every token moved - is the one-line answer to "what does this product cost us per use," and the raw bill against the budget line is the one finance reads. Interviewers test this because the meter is where engineering meets accounting: the xAE-style system-design round asks how you would bill two teams sharing one serving pool, and cost-per-request arithmetic shows up under the scaling follow-ups of nearly every AI design question. The three challenges below work the meter: the dollars one request bills, the bill the cached prefix discounts, and the ghost dollars a half-tagged bill leaks.