Design ChatGPT: Tiers & Rate Limits
Design ChatGPT is the most-cited systems design question at AI companies, and it is really one question: when everyone arrives at once, who gets served, who waits, and who is told no, all without the paying tier ever noticing. The answer interviewers grade against has three parts. First, three pools: a small provisioned pool whose cards exist for one tier only, a larger shared pool where paid requests carry priority over free ones, and the queue itself, which lives in card memory, not in a message broker. Second, a token-bucket rate limiter at the gateway, held in Redis for atomicity under concurrent writers, that admits free arrivals at a chosen share of demand and reconciles the token count after each grant, so a burst of arrivals is answered instantly without consulting a GPU. Third, shedding: a request refused at the gateway with a 429 costs nothing; the same request admitted into a saturated pool holds an expensive seat and lengthens every wait behind it, which is why the tail wait punishes utilization non-linearly, and why an incident that removes 30% of capacity can turn a 97 millisecond paid wait into an 850 millisecond one. Interviewers test this because every knob in the design is a money or trust decision: renting burst capacity for a launch day, tightening the bucket over a jammed pool, or letting the free tier shed so the subscribers do not notice an outage. The three challenges below work the mechanics: the admit count a token bucket grants, the queue wait the utilization formula quotes, and the shed arithmetic that keeps the shared pool from jamming.
Key Concepts
- 1The tier plan has three pools, not two: a small provisioned pool that exists for premium traffic alone, a larger shared pool where paid requests carry priority over free, and free arrivals admitted only up to what the gateway token bucket allows. The premium pool is sized for its own tier, so a free-tier surge cannot touch it
- 2A token bucket is reserve-then-reconcile: the reservation is granted instantly if a token exists, and the bucket reconciles its token count afterward at a steady refill rate, capped at a burst size. In Redis this is one atomic script so concurrent writers cannot double-spend, and the decision never consults a GPU
- 3A paid request in a priority queue waits a function of paid load alone, which is the property that makes the tier boundary real: shedding free traffic cannot rescue a saturated paid tier, and a free surge cannot jam the paid tier while priority holds. Verify the approximation in production numbers before repeating it, but the mechanism is what interviewers expect
- 4Queue wait under the M/M/1 approximation is service_time x utilization / (1 - utilization), and it explodes as utilization closes on 1: the tierSim default pool runs at 61% for a 97 ms paid wait, and the same pool at 100% utilization quotes a wait that never drains. A 30% capacity loss on a thin fleet turned 97 ms into 853 ms
- 5Shedding is the cheapest feature in the stack: a 429 at the gateway costs zero card time, while an over-admitted request holds a seat and lengthens every wait behind it. The bucket rate, not the fleet size, is the knob that fixes a launch-day surge
- 6Renting burst capacity is the fleet knob, and it exists because autoscaling lags: a model serving node takes minutes to come up while a spike can triple arrivals in 30 seconds, so the design pre-rents or sheds; the interview follow-up is what your system does in the gap between the two
The trade-offs to name out loud
A token bucket set near 100% of demand keeps the free tier happy until the pool closes on full utilization, where wait time explodes and the queue stops draining; tightening the bucket rejects more at the edge, which costs nothing and buys back a queue that drains. The bucket rate is the admission control interviewers want named, with its refill and burst numbers written down.
A dedicated premium pool isolates the paying tier from surges and incidents, and prices that isolation in idle cards. A single shared pool never idles but hands the paid tier queue risk from traffic it does not control. The compromise is a small provisioned share sized for the premium arrival rate plus priority on the shared pool.
Refusing arrivals at the edge loses users for one request and keeps the pool healthy; admitting everything queues them into dead waits while paid rides above. The design question is where the bucket sits between those, and the honest answer is measured: shed share against paid SLO, recomputed on every incident review.
Run the tier plan yourself
The tier walkthrough
The batch walkthrough beside this one schedules what got in; the KV calculator decides whether the room exists. This one decides who gets to enter at all: a provisioned pool for premium traffic, a shared pool where paid requests cut ahead of free, a token bucket at the gateway that admits free arrivals at a chosen share of demand, and the refusals the pool would have paid for. Dial the four knobs below and watch who sits in which queue, who is served, and who is told no - then load a collapse and fix the tier plan yourself.
The knobs
Demand and admission
Launch surge (free arrivals): 1
The surge is the free tier: launch day multiplies free arrivals while paid traffic rides its usual curve.
Token-bucket admit rate: 60
Reserve-then-reconcile: the reservation is granted instantly at this share of demand and the token count is reconciled after.
Capacity
Fleet cards: 16
One card in four is provisioned to the premium tier; the rest carry paid priority plus the free tier.
Region incident: none
Preemption by infrastructure: one region down removes 30% of every pool, paid tier included.
paid p95 queue wait against a 400 ms SLO
97 ms
free queue behind what paid leaves
423 ms
of free demand refused at the bucket
40%
free req/s served of admitted, 26.0 demanded
15.6 of 15.6
req/s: premium pool plus shared pool
15.0 + 45.0
the fleet card bill, every month
$46,603
The two pools at this config
The tier ledger
The paid tier rides its 97 ms wait inside the 400 ms SLO, and every check against the plan is a knob away. Paid waits are a function of paid load alone: the free surge cannot touch this number, only the cards can.
Free arrivals run at 26 req/s against a calm 26: every arrival above the bucket's rate is refused at the edge, in a serverless call that costs the pool nothing.
The bucket admits 15.6 free req/s instantly (reserve-then-reconcile reconciles the token count after each grant); 10.4 req/s draw a 429 instead, and a shed request costs zero card time.
16 cards split 4 provisioned to the premium tier and 12 to the shared pool: 15.0 req/s of premium capacity against 45.0 req/s shared.
All regions healthy: the premium pool holds 15.0 req/s and the shared pool 45.0 req/s, no preemption anywhere.
Mattered vs noise (which knobs actually moved the tier plan)
Load a famous collapse, then fix the tier plan yourself
See the whole lifecycle as traffic: The Request Highway lab turns every AI request into a vehicle that stops at the edge cache, the guardrails, and the rate limiter before the model pool - and drives it home live, with a surge dial and an outage switch.