Serving AI Reliably
The AI feature launched ten days ago and is the quietest thing you operate: the dashboard is a strip of green graphs, no complaints, nothing shipped this week. Then at 3am something changes, and running this feature is different because the dependency is a model you neither wrote nor can restart: a provider answers your users, and the provider can change its capacity, its pricing, or the model itself without telling you. So the board grows meters that watch the call rather than your code. p95 latency, the wait that 95 of every 100 requests stay under, catches a capacity squeeze that the median wait, p50, smooths right over. The error share catches a rate limit, a provider ceiling on how many calls fit in a minute that a launch surge can trip with zero code changes. Daily spend catches a prompt edit that quietly broke the prefix cache, the discount a provider grants while your requests resend the same opening text. And an eval set, a small fixed list of prompts with expected answers that your code reruns on a schedule, catches the scariest one: a silent quality drop after the provider swapped the model, the failure users notice and no latency meter ever blinks. What you do about it is the fallback ladder, an ordered list of answers to serve while the model misbehaves: retries with backoff first, then a smaller or older model, then a cached reply, ordered so the user meets slow long before they ever meet gone. Pinning, writing the exact model version into your config instead of floating on the provider default, is what turns a swap back into a rollback. Interviewers ask what you watch and what you do at 3am: a specific ladder is the answer of someone who ran a live AI feature, and we would monitor things is the answer of someone who has not. The three challenges below grade exactly that: the order you triage a crowded alert board in, the p95 and eval math that tells you to roll a model version back, and the share counters that say whether the ladder is holding.
Key Concepts
- 1p50 and p95 are read together: the median describes the typical request, and p95 describes the wait the slowest handful of users get. A flat p50 beside a doubled p95 is the signature of a provider tail problem, and it is a shape nothing in your own code can explain
- 2A rate limit is a ceiling on the minute, not a per-user allowance: when your traffic doubles, the provider refuses the excess with rate-limit wording and everything else about the feature is fine. No rollback of your own code helps, because nothing on your side broke
- 3The fallback ladder is ordered slow before gone: retry with backoff first, because the same call may succeed a minute later; then a smaller or older model, which answers slower but answers; then a cached reply, which is stale but keeps the page up. Each rung trades a little quality for a lot of availability, in that order
- 4An eval set is the quality meter for a model you do not control: rerun the same prompts on any version and compare pass rates, and a slide with no deploys on your side is evidence the provider changed something. The pin, the exact version string your code names, is the one-line rollback that turns that evidence into recovery
- 5Cost without traffic is a diagnosis: spend up 40x with flat traffic and a cached-token invoice line at zero means something fresh near the head of your prompt voided the cache discount for every request. The fix is usually a prompt diff, not a tier change
Run the 3am triage board
The 3am triage walkthrough
Anyone who has kept the on-call phone on the first week after shipping an AI feature knows this one: four alerts light up in the same hour, and your deploys are not even implicated. The model dependency misbehaves on its own schedule - a launch surge trips a rate limit, a regional capacity squeeze doubles the p95, a prompt edit voids the prefix cache, or the provider swaps the model version and only the eval set notices. The board below is one fictional launch night, exactly as a real one arrives.
Four alerts, each with its metric, its context, four possible causes, and four possible first moves. The drill before every commit: predict the cause from the evidence, commit the first action you would actually take, then read the diagnosis and the ladder note the board reveals. Cause options come from the alert text alone, not from experience, so the same board works the first night and the hundredth.
Alert 1 of 4: Model error rate
Call errors went from 0.2% to 8.9% on model calls in the last hour, and every error body carries the provider wording for a rate limit.
Launch traffic doubled after the 8pm push. Your own deploys are two days old, and no feature flag changed at 8pm. The answers that do return are perfect.
Commit the first action
Alert 2 of 4: p95 latency
p95 went from 1,800 ms to 4,100 ms in the same hour. The p50 barely moved. Traffic is flat, and the provider status page is green.
The error rate is back to 0.1%. Every slow request still returns a good answer; the users are just waiting.
Commit the first action
Alert 3 of 4: Model spend
Daily model spend is up 40x since Tuesday. Traffic is flat, the error rate is zero, and the answers are unchanged on the quality board.
A teammate shipped a prompt update on Tuesday; nothing else shipped all week. On the invoice, the line for cached prompt tokens has gone to zero.
Commit the first action
Alert 4 of 4: Eval pass rate
The eval pass rate slid from 0.91 to 0.78 across two days. Latency is flat, errors are zero, spend is steady, and no deploys went out.
The same eval set runs every week. The failing cases cluster on the hardest prompts: multi-step reasoning and long references.
Commit the first action