All question banks

Scaling and Load Balancing Interview Questions

These are the questions that decide whether a serving fleet holds up on launch day: what metric to autoscale on, how to warm new replicas into readiness, how to route traffic so caches pay, and how to keep the tail latency sane while the fleet grows. Every answer states its assumptions and shows the arithmetic. Terms link to the glossary where they are defined.

Warm-up

1. What should autoscaling watch: CPU, GPU gauges, or queue depth?

Queue depth, or request wait against a target. CPU is borrowed terminology from stateless web serving, and GPU compute gauges lie for decode: the card reports activity while running far under its real bandwidth ceiling. Queue depth is user-visible arithmetic - it grows the moment arrivals outpace service, before any card statistic says so.

Worked answer

  1. Little's law: requests resident = arrival rate x time spent = 10 requests/s x 0.5 s = 5 requests in the system.
  2. Raise arrivals to 12 per second at the same 0.5 s of service: 6 resident, and the queue grows while arrivals stay above service.
  3. A compute gauge at busy decode would have looked the same in both cases - the queue is the signal that moved first.

Queue depth is the leading indicator because the arithmetic that fills it (arrivals times sojourn) runs ahead of every card gauge.

Core

2. Why do new replicas take so long to arrive, and what happens in the gap?

A replica boots before it serves: weights load from storage, the KV cache allocates, kernels warm. Until then the burst of demand lands entirely on the replicas already running, which is exactly when they are busiest, so queues grow and latency SLOs slip. Warm pools and pre-warming close the gap by paying for the readiness in advance.

Worked answer

  1. Boot cost: 70 GB of 8-bit weights at 6 GB/s from NVMe is 70 / 6 = 11.7 s to read the model, plus a stated 20 s of initialization: about 32 s to first token.
  2. A burst triples demand for 20 s: all of it queues behind the running replicas, because nothing new is serving for the first 32 s.
  3. One warm replica absorbs the first arrivals while the pool refills from launches.

About 32 s of boot is the gap between a trigger and relief - which is why the trigger must fire on a forecast, and warm pools exist at all.

Hard

3. How big should the warm pool be?

Big enough to cover the rise of the burst until new launches are ready, and no bigger, because every standby pays rent around the clock. Read the arrival pattern, cost a standby honestly, and prefer one warm replica plus fast autoscaling over a fleet of idlers sized for the once-a-year peak.

Worked answer

  1. Rent per replica: $2.50 per hour x 24 x 365 = $21,900 per year, paid whether or not a request arrives.
  2. Four always-warm replicas: 4 x $21,900 = $87,600 a year of idle rent.
  3. One warm replica: $21,900, and it covers the first seconds of a burst while the rest launch after the 32 s boot.
  4. Saving: $87,600 - $21,900 = $65,700 a year for a slightly deeper queue at the burst edge.

One warm replica costs $21,900 a year against $87,600 for four - a burst-shaped pool, not a constant, is the answer.

Hard

4. How would you load balance when you run a prefix cache?

Route by cache affinity: requests that share a prefix should land on a replica that already holds its cache, typically by hashing the prefix or keying a routing table, rather than spreading everything round-robin. The trade is fairness: strict affinity can overload one replica, so admission has to watch queue depth and overflow politely rather than silently.

Worked answer

  1. The workload: 20,000 requests a day x 1,500 shared prompt tokens = 30 million prompt tokens a day.
  2. Perfect affinity, first request primed: 30 million x $0.30 (cached) = $9.00 per day.
  3. Random spread with small per-replica caches that hit on only 10 percent: 10 percent of tokens at $0.30 costs $0.90, and the other 90 percent at $3.00 costs $81.00, so $81.90 per day.
  4. The same traffic, same replicas: affinity cut the prompt bill from $81.90 to $9.00.

Cache-aware routing is worth an order of magnitude on the prompt bill here - the router, not the serving stack, decided what the cache was worth.

Core

5. When is spot capacity acceptable for serving replicas?

When the workload absorbs eviction: batch and offline jobs first, then fault-tolerant serving where autoscaling replaces evicted replicas and in-flight requests drain. Price it per effective card-hour - the discount divided by the expected share of cards actually alive - and compare that against on-demand before committing anything latency-critical to it.

Worked answer

  1. A 12-card fleet: on-demand at $2.50 per hour is $30 per hour for 12 effective cards.
  2. Spot at a 50 percent discount: $1.25 per hour, and with a stated 5 percent evicted at any moment, effective cards are 12 x 0.95 = 11.4.
  3. Per effective card-hour: $15 / 11.4 = about $1.32, against $2.50 on-demand.
  4. The saving is $15 - $14.25 = $0.75 per hour in the raw bill and far more per effective card - the difference is the eviction work.

Spot at about $1.32 per effective card-hour against $2.50 wins for eviction-tolerant work, and the discount survives its own downtime.

Core

6. What does draining mean when you scale down, and what has to happen first?

A scaled-down replica must finish its in-flight requests before it dies: kill it mid-reply and the user sees a truncated answer or a retry with a fresh prefill. Draining means stop admitting, let decode finish, then exit. The bound on drain time is the slowest in-flight request, which is why long-context or long-reply traffic sets when the replica can actually be released.

Worked answer

  1. Slowest in-flight reply: 500 tokens at 24 tokens per second = 500 / 24 = about 21 s to finish.
  2. A replica killed at the moment of the downscale call truncates a reply the user has been reading for 20 s.
  3. With draining, the replica waits at most the bounded 21 s, then releases a card worth $2.50 an hour.

Drain time is the long tail of the reply distribution - about 21 s here - and skipping it costs a broken reply to the user who has waited longest.

Core

7. One replica runs 3x slower after a deploy. How do you keep the fleet healthy?

Detect, quarantine, roll back. Health checks must read user-visible signals (request latency, tokens per second against the target), not just process liveness, so a slow replica looks sick from the outside. Remove it from rotation so it stops being chosen by the load balancer, migrate its queue, and roll the deploy back on that replica - then treat the cause. The fleet-wide guardrail: canary the deploy to one replica before touching the rest.

Core

8. How do you size a serving fleet for a launch you cannot forecast?

Pick a capacity rule and a safety valve. Size for a forecast you can defend (floor, expected, ceiling), hold headroom above the expected case, and define what gets shed when reality exceeds the ceiling: which workloads wait, which get a polite rejection, and which are protected. Load shedding is a designed feature, not an incident.

Worked answer

  1. Fleet sustains 9,000 tokens per second with 3 replicas plus 1 spare, using the 3,000 per-replica figure.
  2. Hold 25 percent headroom over the forecast: a forecast of 7,200 tokens per second needs 7,200 / 3,000 = 2.4, so 3 replicas carry it inside the SLO.
  3. At 2.4 of 3 usable replicas the fleet is at 80 percent of its headroom: the trigger at that line adds a replica before the queue grows.
  4. The cost of the rule: 4 x $2.50 = $10 per hour versus $9.60 that a bare 3.84-replica forecast would imply.

The headroom rule costs about $0.40 an hour over the bare forecast and buys the trigger firing early - the cheaper kind of insurance.

Hard

9. What is the difference between scaling for throughput and scaling for concurrency?

Both move together, but the bottleneck differs. Adding replicas multiplies everything: aggregate throughput, concurrency, and per-user headroom. Raising the batch size inside one replica multiplies aggregate throughput while per-user speed holds - until it does not, because cache reads and scheduling crowd the step. Knowing which knob is full tells you which to turn: an exhausted batch needs replicas; spare bandwidth inside a small batch does not.

Worked answer

  1. One replica, batch 16, sustains about 766 aggregate tokens per second (the bandwidth-shared ceiling at 48 per user).
  2. Four replicas: 4 x 766 = about 3,064 aggregate tokens per second, and concurrency rises to about 64 users.
  3. One replica alone raising the batch to 64 cannot reach 3,064: per-user speed drops and cache reads bite.

Replicas are the clean multiplier for aggregate load; the batch is the cheap one and pays in per-user speed first.

Hard

10. Requests are spread evenly across replicas but p99 latency is 10x p50. Why?

Even request counts are not even work. A few long prompts, a few cold prefix caches, a few users with huge generation requests - any of these makes one replica or one slot far slower than its neighbor, and percentiles surface exactly that tail. Load balancing by request count smooths the average and leaves the tail untouched.

Worked answer

  1. Per-user reply time: TTFT 0.4 s plus 500 tokens at 24 tokens per second is 0.4 + 500 / 24 = about 21.3 s.
  2. A user whose prompt missed the cache prepills 4,000 extra tokens: add 4,000 x 140 GFLOPs = 5.6e14 FLOPs, about 0.57 s more compute in prefill.
  3. A user queued behind two such slow requests: wait is about 2 x 0.57 s extra, roughly 22.5 s total - the p99 shape without any replica being at fault.

The tail lives in the work inside each request - prompt size, cache misses, generation length - so the balancer must be load-aware, not count-aware.

Need the vocabulary first? The glossary defines each of these terms with a worked example, and the other banks cover the other half of the interview.