Metering & Cost Attribution

The fourth serving job is the meter: the model serves traffic, the fleet is paid for, and the dollars land on an invoice that says per 1M, in and out, so the interview question becomes who spent what and on whose behalf. The model is one month of one product. Tokens per request are stated per side - a chat assistant sends 800 prompt tokens and writes 300 out, a coding copilot 2,400 and 600, and a research agent 12,000 and 4,000, because an agentic loop re-sends its context after every tool call and produces tool chatter of its own, so the same "request" moves 15x the tokens. Rates are per 1M tokens, priced separately for prompt and output: the fast tier is $0.80 in and $4.00 out, the flagship $3 and $15, the frontier $15 and $75 - and the out-rate is always the 5x side, because output is generated token by token against a weight read that a prompt prefix caches but an answer never does. Three policy lines decide whether the ledger can be trusted. The prefix cache sells the repeated prompt at a tenth of the fresh rate, so the cached share of prompt tokens is the cheapest dollar on the bill and the saved line meters the discount forgone. Retries re-bill the whole request at full price - both sides - so a 20% retry rate is a 17% tax that product analytics never sees, because from the code's point of view nothing changed. Attribution, tagging the bill by named spender, adds no dollars at all, but a bill with no owner is invisible to the chargeback, the margin check, and the anomaly alert, and the sim lets at most 10% of the bill go untagged before it stops being a ledger. The output numbers compound: the blended rate - bill over every token moved - is the one-line answer to "what does this product cost us per use," and the raw bill against the budget line is the one finance reads. Interviewers test this because the meter is where engineering meets accounting: the xAE-style system-design round asks how you would bill two teams sharing one serving pool, and cost-per-request arithmetic shows up under the scaling follow-ups of nearly every AI design question. The three challenges below work the meter: the dollars one request bills, the bill the cached prefix discounts, and the ghost dollars a half-tagged bill leaks.

The research agent answers better because it re-reads context and streams tool turns, and every one of those tokens bills: 15x the tokens per request turns a rounding-error feature into the budget line. The design answer is not "use less model" but "price the profile first" - tokens per request, the cached share, and the retry rate decide what a feature costs before the tier does.

A high prefix-cache hit share is the cheapest dollar on an agent bill, because agent context is mostly re-sent system prompts and prior turns - exactly what a prefix cache sells at a tenth of the rate. But the discount lands only on the prompt side: a profile that writes long answers bills the output line at full price no matter how perfect the cache is, so the cache strategy follows the in/out ratio, not the hype.

Attribution is infrastructure: a required tag on every metered call costs a code-review rule and a failing test, while retrofitting tags after the fact means untangling a bill that has already taught finance nothing. Ten unowned percent this month is how a chargeback becomes an argument, and the teams that meter leaks as separate lines (retries, untagged share) catch the anomaly the month it starts.

Put Metering & Cost Attribution into practice

The best way to lock in what you've learned is to write code. Solve real interview-style problems right now.