Tail latency

The median hides what the slowest users live with. Percentiles such as p95 and p99 ask a different question: read the requests from fastest to slowest and name the wait at the 95th or 99th position, so the handful of stragglers - the ones a churned user notices - decide the number. That is why service level agreements quote p95 or p99 and never the median: the tail is where the complaint emails come from.

Two shapes tell different stories. A p95 that rides up with the p50 means everything got slower - a slow model version, a bad cache day. A p50 that stays flat beside a rising p95 means the typical request is fine and a shrinking minority is queueing behind something: a batch of long prompts, a card at saturation, an eviction storm. That second shape is the one that hides until percentiles are measured, and it usually has an infrastructure answer rather than a code one.

Nearest-rank is the honest way to read it from raw samples: sort the waits and take the value at position ceil(0.95 x n), counting from one. With ten samples that is the 9th smallest, so one wild outlier cannot claim the position - which is also why a single 5-second blip in ten requests can raise p99 while leaving p95 untouched, and why both are read together.

1,000 requests in a minute. Nearest-rank percentiles: each band is read by rank, where rank r of n is the r-th smallest sample.

1. The bands: 950 requests land under 400 ms, 40 land between 400 and 900 ms, and 10 land above 5,000 ms during a burst.
2. p95 is rank 950: the top of the fast band, about 400 ms - a number a healthy median story would happily report.
3. p99 is rank 990: the top of the middle band, about 900 ms - and the 10 requests above it are over 5,000 ms.
4. The p50 of the same minute sits well inside the fast band and moved almost none while the tail tripled.

p95 of 400 ms hides it, p99 of 900 ms names it, and 10 users live at 5,000 ms - the wait the median never sees and the users who complain do.

Where this shows up

• Serving AI ReliablyThe AI feature launched ten days ago and is the quietest thing you operate: the dashboard is a strip of green graphs, no complaints, nothing shipped this week. Then at 3am something changes, and running this feature is different because the dependency is a model you neither wrote nor can restart: a provider answers your users, and the provider can change its capacity, its pricing, or the model itself without telling you. So the board grows meters that watch the call rather than your code. p95 latency, the wait that 95 of every 100 requests stay under, catches a capacity squeeze that the median wait, p50, smooths right over. The error share catches a rate limit, a provider ceiling on how many calls fit in a minute that a launch surge can trip with zero code changes. Daily spend catches a prompt edit that quietly broke the prefix cache, the discount a provider grants while your requests resend the same opening text. And an eval set, a small fixed list of prompts with expected answers that your code reruns on a schedule, catches the scariest one: a silent quality drop after the provider swapped the model, the failure users notice and no latency meter ever blinks. What you do about it is the fallback ladder, an ordered list of answers to serve while the model misbehaves: retries with backoff first, then a smaller or older model, then a cached reply, ordered so the user meets slow long before they ever meet gone. Pinning, writing the exact model version into your config instead of floating on the provider default, is what turns a swap back into a rollback. Interviewers ask what you watch and what you do at 3am: a specific ladder is the answer of someone who ran a live AI feature, and we would monitor things is the answer of someone who has not. The three challenges below grade exactly that: the order you triage a crowded alert board in, the p95 and eval math that tells you to roll a model version back, and the share counters that say whether the ladder is holding.
• Design ChatGPT: Tiers & Rate LimitsDesign ChatGPT is the most-cited systems design question at AI companies, and it is really one question: when everyone arrives at once, who gets served, who waits, and who is told no, all without the paying tier ever noticing. The answer interviewers grade against has three parts. First, three pools: a small provisioned pool whose cards exist for one tier only, a larger shared pool where paid requests carry priority over free ones, and the queue itself, which lives in card memory, not in a message broker. Second, a token-bucket rate limiter at the gateway, held in Redis for atomicity under concurrent writers, that admits free arrivals at a chosen share of demand and reconciles the token count after each grant, so a burst of arrivals is answered instantly without consulting a GPU. Third, shedding: a request refused at the gateway with a 429 costs nothing; the same request admitted into a saturated pool holds an expensive seat and lengthens every wait behind it, which is why the tail wait punishes utilization non-linearly, and why an incident that removes 30% of capacity can turn a 97 millisecond paid wait into an 850 millisecond one. Interviewers test this because every knob in the design is a money or trust decision: renting burst capacity for a launch day, tightening the bucket over a jammed pool, or letting the free tier shed so the subscribers do not notice an outage. The three challenges below work the mechanics: the admit count a token bucket grants, the queue wait the utilization formula quotes, and the shed arithmetic that keeps the shared pool from jamming.

Related terms