Backpressure

A server under overload has two things it can give away: refusals and wait time. A design without backpressure gives away wait time, because every arrival is admitted and the queue absorbs the rest, so the overload shows up as a latency breach minutes later rather than a refusal at the door. A design with backpressure gives away refusals: a bounded queue, an admission limit, or a rejection at the gateway keeps the wait inside known limits and points the pain at the request that arrived last, not at every request already in flight.

The mechanics are simple and they stack. A bounded queue caps how much wait can accumulate behind admission. An admission limit caps what enters the pool at all, sized to what the KV cache and batch can actually carry. Load shedding is the visible version - refusing or redirecting arrivals above the limit with a clear status, so the caller can retry, route elsewhere, or degrade on purpose.

In AI serving the stakes are higher than a slow response, because the resource being protected is also the most expensive one. A request admitted into a saturated pool holds a KV cache seat and stretches the wait of every request behind it; the same request refused at the gateway costs nothing. That asymmetry - refuse early, serve calmly - is the whole argument, and it is why the rate limiter, the queue depth meter, and the shed decision are usually one design rather than three.

A pool where each request needs 0.1 s of service sits at rising utilization. Queue wait under the common approximation is service time x utilization / (1 - utilization), which is the arithmetic that makes the refusal cheap.

1. At 80 percent utilization: 0.1 x 0.8 / 0.2 = 0.4 s of wait for the requests that wait.
2. At 95 percent: 0.1 x 0.95 / 0.05 = 1.9 s - almost five times the wait for fifteen more points of utilization.
3. At 98 percent: 0.1 x 0.98 / 0.02 = 4.9 s, and beyond 100 the queue never drains at all.
4. A refusal at the gateway instead of these admissions turns that curve into a flat, known, cheap rejection.

Wait grows super-linearly as a pool saturates - 0.4 s, 1.9 s, 4.9 s at 80, 95 and 98 percent - which is the arithmetic case for refusing at the edge instead of queueing to failure.

Where this shows up

• Design ChatGPT: Tiers & Rate LimitsDesign ChatGPT is the most-cited systems design question at AI companies, and it is really one question: when everyone arrives at once, who gets served, who waits, and who is told no, all without the paying tier ever noticing. The answer interviewers grade against has three parts. First, three pools: a small provisioned pool whose cards exist for one tier only, a larger shared pool where paid requests carry priority over free ones, and the queue itself, which lives in card memory, not in a message broker. Second, a token-bucket rate limiter at the gateway, held in Redis for atomicity under concurrent writers, that admits free arrivals at a chosen share of demand and reconciles the token count after each grant, so a burst of arrivals is answered instantly without consulting a GPU. Third, shedding: a request refused at the gateway with a 429 costs nothing; the same request admitted into a saturated pool holds an expensive seat and lengthens every wait behind it, which is why the tail wait punishes utilization non-linearly, and why an incident that removes 30% of capacity can turn a 97 millisecond paid wait into an 850 millisecond one. Interviewers test this because every knob in the design is a money or trust decision: renting burst capacity for a launch day, tightening the bucket over a jammed pool, or letting the free tier shed so the subscribers do not notice an outage. The three challenges below work the mechanics: the admit count a token bucket grants, the queue wait the utilization formula quotes, and the shed arithmetic that keeps the shared pool from jamming.
• LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
• Serving AI ReliablyThe AI feature launched ten days ago and is the quietest thing you operate: the dashboard is a strip of green graphs, no complaints, nothing shipped this week. Then at 3am something changes, and running this feature is different because the dependency is a model you neither wrote nor can restart: a provider answers your users, and the provider can change its capacity, its pricing, or the model itself without telling you. So the board grows meters that watch the call rather than your code. p95 latency, the wait that 95 of every 100 requests stay under, catches a capacity squeeze that the median wait, p50, smooths right over. The error share catches a rate limit, a provider ceiling on how many calls fit in a minute that a launch surge can trip with zero code changes. Daily spend catches a prompt edit that quietly broke the prefix cache, the discount a provider grants while your requests resend the same opening text. And an eval set, a small fixed list of prompts with expected answers that your code reruns on a schedule, catches the scariest one: a silent quality drop after the provider swapped the model, the failure users notice and no latency meter ever blinks. What you do about it is the fallback ladder, an ordered list of answers to serve while the model misbehaves: retries with backoff first, then a smaller or older model, then a cached reply, ordered so the user meets slow long before they ever meet gone. Pinning, writing the exact model version into your config instead of floating on the provider default, is what turns a swap back into a rollback. Interviewers ask what you watch and what you do at 3am: a specific ladder is the answer of someone who ran a live AI feature, and we would monitor things is the answer of someone who has not. The three challenges below grade exactly that: the order you triage a crowded alert board in, the p95 and eval math that tells you to roll a model version back, and the share counters that say whether the ladder is holding.

Related terms