Inference economics: the bill and the wait
Somewhere near the top of every AI product's paperwork are two numbers nobody on my first project could explain to a manager: what one message costs, and why some users waited two seconds while others waited nine. Both come from mechanics worth learning once. The cost mechanic is the token, the small piece of text - a few characters, usually a word-ish chunk - that models count everything in. Every request bills for the text sent in plus the answer generated, and the text sent in gets a big discount when the first part of it is identical to what recent requests sent, because the provider can reuse its work on those words; a request that gets the discount is called cached. The wait mechanic splits into stages: a fixed leg across the network, a stretch that grows with how much text is sent in, called prefill, and a stretch that grows with the answer being written word by word, called decode. What made both ledgers click for me is that most levers act on only one ledger. Making the incoming text more cacheable cuts money and prefill but leaves decode untouched. Capping the answer cuts both ledgers and buys an accuracy risk. A cheaper, smaller model cuts money and does nothing for the wait. Answering an interview question about AI cost or speed with the wrong lever for its ledger sounds like reasoning and says nothing.
Most people have this backwards
The biggest inference bill is usually not the model tier. It is an uncached preamble - a fixed block of instructions sent ahead of every request - that the system re-bills at full price on every request.
The catch
It holds when a fixed instruction block could stay prefix-stable across requests - the same opening words every time, which is the common case for chat and tool wrappers - because then the same tokens are re-billed at full rate instead of the discounted cached rate, run after run, for as long as a stamp, shuffle, or rotation goes unnoticed. It weakens when the opening block is legitimately different per request, as with unique documents or tenant rules that must lead the prompt, because there is nothing stable to cache and the tier question reclaims first place.
Baseline first - your honest read.
Traffic is flat, output quality is unchanged, and the inference bill doubled. Which pairing most often explains a doubling like that?
Make each call, then see the verdict
Predict first
A harness stamps a fresh run id into the head of the system preamble on every request. The stamp is a dozen tokens. Removing the stamp entirely - what does it do to the bill?
Four workloads, four bills to cut. Pick the lever each one wants.
A support assistant prepends the same policy manual to every request - hundreds of pages of identical text, sent fresh about 600 times a day. Answers are short.
A weekly report generator produces long documents. Users stop reading after the first section, and the second half of every report is nobody's demand.
A pipeline routes incoming tickets by label. The answer is one of a handful of labels, and the strongest tier spends most of its effort not making this decision.
An agent harness stamps a fresh run id into the head of its system preamble on every request. Support wants the bill cut; the traffic doubled and the answers did not change.
How it works
Picture two stacked bars over the same request. The cost bar: one long slab for the prompt prefix, colored cached or full-price, a slim slab for new input tokens, then the output slab drawn wider for every token generated. The latency bar underneath it: a fixed sliver for the network leg, a prefill slab that grows with prompt size, then the decode stretch that grows with the answer. Reading the bars teaches one comparison: when I pull a lever, which slab moves. A small stamp added to the front of the prompt repaints the entire prefix slab from cached to full price - same prompt, several times the bill - which is why the stamp, not the tier, often owns the invoice. A cap on the answer shortens the last slab of both bars at once. The two bars also separate the median request from the p95 one: the tail lives at the far end of the decode stretch, not in the prompt slabs, which is why latency work and cost work keep arriving at the same lever from opposite directions.
The prep station
I cook with this one in mind. In a working kitchen, the prep cook spends the quiet hours before service slicing and measuring everything the line will need, so at seven o clock every order is assembly. That identical mise en place serves every order all night - you do not re-chop the onions per plate. A prompt prefix is mise en place: expensive, identical text, prepared once, then read from cache at assembly time on every request. And I know exactly how the bill gets ruined, because I have worked the restaurant where the head chef rewrites the station list before every service - all the prep turns to waste and every order cooks from scratch again. That is a run id stamped into the front of a preamble.
Where it breaks
The analogy breaks at how the station gets consumed. Mise en place re-reads for free - a scoop of prepped onions leaves the rest of the tray intact - while a cached prefix bills on every request, at the discounted rate but never at zero, so an enormous stable prefix that every request re-reads can still dominate a bill at its own discounted price. Preparation also never expires in a kitchen, where cache entries evict under pressure and go stale the moment the prompt text shifts, which makes the hit rate a measured number rather than a setting.
Why interviewers lean on it
Interviews press here because the cost story collapses under one follow-up: what did you change, and which of the two ledgers moved? A candidate who moved to a cheaper model tier and cut the bill by 40 percent should be able to say what that did to wait time (usually nothing) and to cost per useful answer (unknown until measured), and the candidate who cached a fixed preamble should know the cache hit rate they bought - the share of requests getting the discount - and what small changes to the prompt are doing to it. There is a worst-case half too: users do not feel the average cost, they feel their slowest wait, and the stage with room to move is usually decode, which only shrinks when answers get shorter. The vocabulary divider is quick - hit rate, discount rate, per-stage timing, an output cap are engineering terms; a cheaper tier is a purchasing term - and teams fluent only in the second language re-litigate the same invoice every quarter. A candidate who has done the arithmetic answers from the request log, and one who has not answers from the price list, and that difference is audible within a minute.
The follow-up chain
Interviewers do not stop at the surface answer. Answer the way you would out loud, or skip - skipping reveals the strong answer and ends the chain.
The interviewer's next question
Our inference bill doubled this quarter with traffic basically flat. Where do you look first?
Now answer one you have not seen
A teammate wants to halve the product inference bill and proposes a cheaper tier for every request. You have one measurement day before the decision meeting. What do you measure, and what do you bring to the meeting instead of a tier opinion?
Expires fast - as of 2026-10-05
As of 2026-10, the major hosted model APIs all advertise prefix or contextual caching with a stated discount on cached reads - commonly a 50 to 90 percent reduction against the fresh-token rate - and all bill generated tokens at a premium over the cheapest input tokens, with ratios that shift between releases.
The opening failure reappears in public postmortems at a steady rate: a per-request value such as a run id or timestamp placed early in the prompt silently zeroes the hit rate, discovered when an invoice spikes or a p95 moves, which is why more teams now gate deploys on a hit-rate check.
Latency engineering in 2026 splits its attention between measuring stages (prefill versus decode versus network) and levers that change the decode curve itself, including speculative approaches, batching, and hardware-side acceleration, while answer caps stay the lever an application controls without any platform change.
In the 2026 job postings mined for this track, inference cost and latency surface under reliability-and-cost wording such as rate limits, spend ceilings, and p95 targets, more often than as a dedicated economics round, which is why this concept keeps the money and wait ledgers in one unit.