Inference Economics & Latency
Every time someone sends a message to an AI feature, the model has to run on that request, and the run has a price in dollars and a wait in seconds. The industry word for the run itself is inference: the model processing your prompt and producing an answer, and it is the line item that grows with usage. Where does the money go? Mostly into tokens, the word-sized pieces of text that both prompts and answers are made of. Input tokens are re-sent with every call; output tokens are generated one at a time at a premium rate; and a large discount called prompt caching applies only while the opening of your prompt matches what the provider already has stored, which one careless changing detail destroys. The wait decomposes into stages that scale differently: reading the prompt grows with prompt size, writing the answer grows with answer length, and a fixed network round trip is attached either way - and the stages disagree about which lever is worth pulling. Interviewers pursue this topic because cost and latency are the two numbers that decide whether an AI feature survives its first real month, and both are governed by arithmetic you can do on a whiteboard. The three challenges below work that arithmetic: the cache hit rate and the savings a rotating preamble quietly destroys, the share of a latency budget that writing the answer consumes, and the work a hard output cap clips off real requests.
Key Concepts
- 1A cached prefix bills at a fraction of the fresh-token rate: the provider keeps the opening part of your prompt, called the prefix, in a cache, and only requests whose opening text matches get the discount. One changing value near the top - a run id, a timestamp, a shuffled instruction - voids the whole shared head
- 2Output tokens are the expensive ones: each generated token bills above a cached input token and adds its own writing time, so answer length prices the request more than prompt length does
- 3Latency is a stage budget, not one number: reading the prompt grows with prompt size, writing the answer grows with answer length, and network overhead is fixed - cutting the wrong stage buys nothing
- 4A hard output cap is deterministic cost control with an accuracy price: it trims the longest answers first, and those tend to be the ones carrying real work
- 5Cheapest per request and cheapest per correct answer are different numbers, and a drop in answer quality moves the second one in the expensive direction
Work the budget dial
The inference budget walkthrough
Anyone who has shipped an AI feature and then opened the first full month's invoice knows this one: the feature worked great in testing at a hundred calls, and the bill arrived looking nothing like the estimate. Inference is simply the word for the paid act of a model generating an answer - every chat reply, summary, or filled-in field is one billed call, priced in tokens (the units of text models count and bill; both sides are charged, what you send and what the model writes back).
With that resource, the job is the same one every backend service has: keep it fast, keep it good, keep it affordable. One fictional service, two workloads (the shapes of traffic it faces: short chat answers and long batch documents), five knobs, a fixed latency SLO (service-level objective: the speed and quality the service promises its users, often measured at the slow end) and a fixed monthly ceiling. One label pair to translate before you touch anything: latency reads p50 (the median request) and p95 (how long the slowest request in twenty takes), because averages hide the slow tail users actually complain about. Nothing here is scripted: every number comes out of the same billing and serving arithmetic a real deployment would run, at toy scale with fictional prices.
Pick the workload
A consumer chat surface: hundreds of thousands of small requests a month, a system prompt plus a few-shot block up front, a short user turn after it. The SLO: p95 under 2000 ms. The ceiling: $2,000 a month.
The knobs
Caching
Stable prefix (tokens): 600
Prefix stability: stable
Serving
Batch size: 1
Generation
Output cap (tokens): 512
Model tier
Model tier: small
monthly bill
$496 / mo
vs defaults: $0
p50 latency
1767 ms
vs defaults: +0 ms
p95 latency
1778 ms
vs defaults: +0 ms
answer quality
92%
vs defaults: +0 pts
Same config, the other workload
On batch document runs this exact config reads latency: pass, quality: fail, cost: pass. One config never covers both workloads: the traffic volume, prompt shape, and answer length all move the same knobs.
The answer as served
Refund requests under sixty dollars are approved automatically. Anything larger routes to a human reviewer with a full purchase history summary attached to the ticket.
The stable prefix covers 600 of the workload's 900 prompt tokens at a tenth of the input price; the other 300 bill in full on every request.
The prefix stays identical across requests, so the cache pays out: 67% of input tokens ride at the discount.
Serving one request at a time skips the queue entirely, and pays full price for the machine time a request holds while it decodes.
The cap allows the 220 tokens this workload needs, so answers finish on their own and the cap costs nothing extra.
The small tier bills $0.001 per 1K input tokens and $0.004 per 1K output tokens, decodes about 125 tokens a second, and lands near 92% quality on this workload against the 85% floor.
Mattered vs noise on this workload (which knobs actually moved the bill)
Load a famous failure, then fix it yourself